注册模型:
model.json
"version": 2,"context_length": 2048,"model_name": "qwen2.5-1.5b-instruct","model_lang": ["en", "zh"],"model_ability": ["chat"],"model_family": "qwen2.5","chat_template": "{% for message in messages %}{{ message['role'] }}: {{ message['content'] }}\n{% endfor %}assistant:","model_specs": [{"model_format": "pytorch","model_file_name_template": "model.safetensors","model_size_in_billions": 2,"quantization": "none","model_hub": "huggingface","model_uri": "file:///data/models/qwen2.5-1.5B/qwen2.5-1.5B-instruct"}],"is_builtin": false
}
注册模型
xinference launch --model-name qwen-2.5-1.5 --model-engine vllm
运行模型
xinference launch --model-name qwen-2.5-1.5b-instruct --model-engine transformers
验证:
curl -X 'POST' 'http://localhost:9997/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-1.5b-instruct",
"messages": [
{
"role": "user",
"content": "hello"
}
]
}'
