Conversation roles must alternate user/assistant/user/assistant/...

[silvacarl2]

we are getting this error: {"object":"error","message":"Conversation roles must alternate user/assistant/user/assistant/...","type":"invalid_request_error","param":null,"code":null}

using hte exact same curl example from the documentation:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "mistralai/Mistral-7B-Instruct-v0.2", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ] }'

i am sure it is simple, but any ideas?

[simon-mo]

Mistral instruct and Mixtral instruct doesn't accept system prompt. This is error message directly from their official chat template.

[yananchen1989]

hello, if mistral does not accept messages which contain system role, why there is 'system' in tokenizer.default_chat_template

MODEL = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer = AutoTokenizer.from_pretrained(MODEL, cache_dir='/home/yanan/downloads')

{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% elif false == true and not '<>'
in messages[0]['content'] %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially
unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don\'t know the answer to a question, please don\'t share false information.' %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message
in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<>\n' + system_message + '\n<>\n\n' + message['content'] %}{% else %}{% set content
= message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'system' %}{{ '<>\n' + content.strip() + '\n<>\n\n' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}"

[jeanfredd]

Thanks for the answear ! Is there a simple way to fix the issue then ? I want to use ChatOpenAI langchain object with mistral instruct run from vllm for a RAG ?

[silvacarl2]

whoops! ok, got it!!!!!!!!!!!!!

THANK YOU!!!!! VLLM IS AWESOME!!!

[pandadom]

can you provide the working curl cmd?

[sofianhw]

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "mistralai/Mistral-7B-Instruct-v0.2",
        "messages": [
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'

[timokinyanjui]

If you are interested in python code snippet that does streaming, it would look like this

`from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://your_vllm.endpoint.com/v1"

client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

chat_completion = client.chat.completions.create(
messages=[{
"role": "assistant",
"content": "You are a helpful assistant.",
"role": "user",
"content": "Who won the world series in 2020?"
}],
model=model,
stream=True
)

for chunk in chat_completion:
print(chunk.choices[0].delta.content, end="")`

[tonyaw]

This shall be an issue to be fixed from vllm side.

"system" role is not only used for system prompt, and can be also used to provide some "system" messages.
For example, one autogen use case can use "system" role for multiple times:
https://github.com/microsoft/autogen/blob/main/notebook/agentchat_chess.ipynb
This use case can't be run via vllm without some code hacking of autogen.

GPT(OpenAI official API) can handle system role.
As an OpenAI API compatible server, vllm shall also be able to handle system role. Otherwise, it is not 100% OpenAI API compatible. :-)

[remiallain]

Faced this error on first shot with vllm.

My workaround is to use a custom chat template.

You can pass it to vllm with --chat-template=/chat-templates/mistral.jinja

Where the content of mistral.jinja is :

{%- for message in messages %}
      {%- if message['role'] == 'system' -%}
          {{- message['content'] -}}
      {%- else -%}
          {%- if message['role'] == 'user' -%}
              {{-'[INST] ' + message['content'].rstrip() + ' [/INST]'-}}
          {%- else -%}
              {{-'' + message['content'] + '</s>' -}}
          {%- endif -%}
      {%- endif -%}
  {%- endfor -%}
  {%- if add_generation_prompt -%}
      {{-''-}}
  {%- endif -%}

It now works like a charm 🥳

[jeanfredd]

this prompt is the same as the one of this database isn't it ? Would you recommand to use the other prompts for other LLMs ?
https://github.com/oobabooga/text-generation-webui/tree/main/instruction-templates

[remiallain]

Yes, it's from there. I've only tested Alpaca and LLama-v2 templates and it worked well.

[jeanfredd]

Thank you so much for your answears @remiallain , it helped a lot for my project !

[remiallain]

glad to help 😉