openLLM
安装
# or pip3 install openllm
pip install openllm
openllm hello
- 聊天界面
OpenLLM provides a chat UI at the /chat endpoint for the launched LLM server at http://localhost:3000/chat.
- 使用
openllm serve llama3.2:1b
openllm run llama3:8b
openllm model list
openllm repo update
openllm model get llama3.2:1b
- 添加模型
openllm repo add <repo-name> <repo-url>
openllm repo add nightly https://github.com/bentoml/openllm-models@nightly
- test
export VLLM_DEVICE=cpu
openllm serve gemma2:2b