一、前言

通过langchain框架调用本地模型，使得用户可以直接提出问题或发送指令，而无需担心具体的步骤或流程。vLLM可以部署为类似于OpenAI API协议的服务器，允许用户使用OpenAI API进行模型推理。

二、术语

2.1. vLLM

vLLM是一个开源的大模型推理加速框架，通过PagedAttention高效地管理attention中缓存的张量，实现了比HuggingFace Transformers高14-24倍的吞吐量。

2.2. OpenAI-Compatible Server

遵循 OpenAI API 的接口规范，让开发者可以使用OpenAI API相同的方式和方法来调用这些服务，从而利用它们的语言模型功能。

三、前提条件

3.1. 基础环境及前置条件

操作系统：centos7
Tesla V100-SXM2-32GB CUDA Version: 12.2
提前下载好qwen1.5-7b-chat模型
通过以下两个地址进行下载，优先推荐魔搭
hugging face：https://huggingface.co/Qwen/Qwen1.5-7B-Chat/tree/main
modelscope：git clone https://www.modelscope.cn/qwen/Qwen1.5-7B-Chat.git

3.2. 安装虚拟环境

conda create --name langchain python=3.10
conda activate langchain
pip install langchain vllm openai

3.3. 启动OpenAI-Compatible Server

nohup python -m vllm.entrypoints.openai.api_server  --model  /model/qwen1.5-7b-chat  --swap-space 24 --disable-log-requests --trust-remote-code --max-num-seqs 256 --host 0.0.0.0 --port 9000  --dtype float16 --max-parallel-loading-workers 1  --max-model-len 10240 --enforce-eager > output.txt 2>&1 &

PS：注意替换上述命令中的参数，特别是端口（上面配置仅适配v100 32G显存,42G内存）

四、技术实现

4.1.基础调用

from langchain_community.llms import VLLMOpenAI
 
llm = VLLMOpenAI(
    openai_api_key="EMPTY",
    openai_api_base="http://localhost:9000/v1",
    model_name="/model/qwen1.5-7b-chat",
    max_tokens=1024,
    top_p=0.9,
    temperature=0.45,
    streaming=True,
    verbose=True,
)
 
print(llm.invoke("广州有什么特色景点?"))

4.2.设置System Prompt

# -*-  coding = utf-8 -*-
 
from langchain_community.llms import VLLMOpenAI
 
from langchain import LLMChain
from langchain.prompts import SystemMessagePromptTemplate, HumanMessagePromptTemplate, ChatPromptTemplate
 
 
llm = VLLMOpenAI(
    openai_api_key="EMPTY",
    openai_api_base="http://localhost:9000/v1",
    model_name="/model/qwen1.5-7b-chat",
    max_tokens=1024,
    top_p=0.9,
    temperature=0.45,
    streaming=True,
    verbose=True,
)
 
 
system_template = "你是一位旅游向导，擅长给客户推荐全国各地的旅游景点。"
system_message_prompt = SystemMessagePromptTemplate.from_template(system_template)
 
human_template = """Question: {question}
Answer: Let's think step by step."""
human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)
 
prompt_template = ChatPromptTemplate.from_messages([system_message_prompt, human_message_prompt])
# prompt = prompt_template.format_prompt(question="我家在广州，有很多好玩的地方，你能介绍一些我家的特色景点吗？").to_messages()
# print(prompt)
 
llm_chain = LLMChain(prompt=prompt_template, llm=llm)
print(llm_chain.run(question="我家在广州，有很多好玩的地方，你能介绍一些我家的特色景点吗？"))

4.3.实现多轮对话

# -*-  coding = utf-8 -*-
 
from langchain_community.llms import VLLMOpenAI
from langchain.chains.conversation.base import ConversationChain
from langchain.memory import ConversationBufferMemory
 
llm = VLLMOpenAI(
    openai_api_key="EMPTY",
    openai_api_base="http://localhost:9000/v1",
    model_name="/model/qwen1.5-7b-chat",
    max_tokens=1024,
    top_p=0.9,
    temperature=0.45,
    streaming=True,
    verbose=True,
)
 
 
memory = ConversationBufferMemory()
 
conversation = ConversationChain(
    llm=llm,
    memory=memory,
    verbose=True
)
 
memory.save_context({"input": "hi，你好"}, {"output": "你好！有什么我可以帮助你的吗？"})
memory.save_context({"input": "我家在广州，很好玩哦"}, {"output": "广州是一个美丽的城市，有很多有趣的地方可以去。"})
 
print(conversation.predict(input="我家在哪里？"))

五、附带说明

5.1.注意事项

示例代码中“openai_api_base”的IP和端口要修改为OpenAI-Compatible Server的地址

5.2. VLLMOpenAI支持的参数

param allowed_special: 
  Union[Literal['all'], AbstractSet[str]] = {}
  Set of special tokens that are allowed。
 
param batch_size: 
  int = 20
  Batch size to use when passing multiple documents to generate.
 
param best_of: 
  int = 1
  Generates best_of completions server-side and returns the “best”.
 
param cache: 
  Union[BaseCache, bool, None] = None
  Whether to cache the response.
 
  If true, will use the global cache.
 
  If false, will not use a cache
 
  If None, will use the global cache if it’s set, otherwise no cache.
 
  If instance of BaseCache, will use the provided cache.
 
  Caching is not currently supported for streaming methods of models.
 
 
param callbacks: 
  Callbacks = None
  Callbacks to add to the run trace.
 
param custom_get_token_ids: 
  Optional[Callable[[str], List[int]]] = None
  Optional encoder to use for counting tokens.
 
param default_headers: 
  Union[Mapping[str, str], None] = None
 
param default_query: 
  Union[Mapping[str, object], None] = None
 
param disallowed_special: 
  Union[Literal['all'], Collection[str]] = 'all'
  Set of special tokens that are not allowed。
 
param frequency_penalty: 
  float = 0
  Penalizes repeated tokens according to frequency.
 
param http_client: 
  Union[Any, None] = None
  Optional httpx.Client.
 
param logit_bias: 
  Optional[Dict[str, float]] [Optional]
  Adjust the probability of specific tokens being generated.
 
param max_retries: 
  int = 2
  Maximum number of retries to make when generating.
 
param max_tokens: 
  int = 256
  The maximum number of tokens to generate in the completion. -1 returns as many tokens as possible given the prompt and the models maximal context size.
 
param metadata: 
  Optional[Dict[str, Any]] = None
  Metadata to add to the run trace.
 
param model_kwargs: 
  Dict[str, Any] [Optional]
  Holds any model parameters valid for create call not explicitly specified.
 
param model_name: 
  str = 'gpt-3.5-turbo-instruct' (alias 'model')
  Model name to use.
 
param n: 
  int = 1
  How many completions to generate for each prompt.
 
param openai_api_base: 
  Optional[str] = None (alias 'base_url')
  Base URL path for API requests, leave blank if not using a proxy or service emulator.
 
param openai_api_key: 
  Optional[str] = None (alias 'api_key')
  Automatically inferred from env var OPENAI_API_KEY if not provided.
 
param openai_organization: 
  Optional[str] = None (alias 'organization')
  Automatically inferred from env var OPENAI_ORG_ID if not provided.
 
param openai_proxy: 
  Optional[str] = None
 
param presence_penalty: 
  float = 0
  Penalizes repeated tokens.
 
param request_timeout: 
  Union[float, Tuple[float, float], Any, None] = None (alias 'timeout')
  Timeout for requests to OpenAI completion API. Can be float, httpx.Timeout or None.
 
param streaming: 
  bool = False
  Whether to stream the results or not.
 
param tags: 
  Optional[List[str]] = None
  Tags to add to the run trace.
 
param temperature: 
  float = 0.7
  What sampling temperature to use.
 
param tiktoken_model_name: 
  Optional[str] = None
  The model name to pass to tiktoken when using this class. Tiktoken is used to count the number of tokens in documents to constrain them to be under a certain limit. By default, when set to None, this will be the same as the embedding model name. However, there are some cases where you may want to use this Embedding class with a model name not supported by tiktoken. This can include when using Azure embeddings or when using one of the many model providers that expose an OpenAI-like API but with different models. In those cases, in order to avoid erroring when tiktoken is called, you can specify a model name to use here.
 
param top_p: 
  float = 1
  Total probability mass of tokens to consider at each step.
 
param verbose: 
  bool [Optional]
  Whether to print out response text.

注：代码的缩进可能有问题