[Frontend] OpenAI API server: Do not add bos token by default when encoding

[bofenghuang]

Hi,

I noticed that the BOS token is always duplicated when running with the OpenAI API server. As shown in the console output below when launching Meta-Llama-3-8B-Instruct, there are two repeated 128000 (bos_token_id) at the beginning of prompt_token_ids.

INFO 05-08 18:24:46 async_llm_engine.py:529] Received request cmpl-1fbfa55a0fc64724b5c383dcaafd7575: prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|
>\n\nHi!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, rep
etition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_
ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8179, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_betwe
en_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 128000, 128006, 882, 128007, 271, 13347, 0, 128009, 128006, 78191, 128007, 271], lora
_request: None.

After investigating, I found that this happens because the chat template already includes the BOS token (AFAIK as is common practice in most cases). However, the encoding step adds it again due to add_special_tokens being enabled by default.

vllm/vllm/entrypoints/openai/serving_engine.py

Line 206 in 0f9a6e3

input_ids = self.tokenizer(prompt, **tokenizer_kwargs).input_ids

https://github.com/huggingface/transformers/blob/5962d62bac850cd01ee830ffba880469338c96fd/src/transformers/tokenization_utils_base.py#L2808

A simple fix is to disable the add_special_tokens, as is done in the apply_chat_template function of Transformers (when tokenize=True).

https://github.com/huggingface/transformers/blob/5962d62bac850cd01ee830ffba880469338c96fd/src/transformers/tokenization_utils_base.py#L1820-L1829

---

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Model] for adding a new model or improving an existing model. Model name should appear in the title.
• [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
• [Kernel] for changes affecting CUDA kernels or other compute kernels.
• [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
• [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• We adhere to Google Python style guide and Google C++ style guide.
• Pass all linter checks. Please use format.sh to format your code.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
• Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

• After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
• After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
• After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
• Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

[simon-mo]

Add a comment in the code for this?

[DarkLight1337]

Have you checked whether this would affect the usage of the example templates in vLLM? I don't think that the examples include BOS tokens.

[bofenghuang]

Hi @DarkLight1337,

You mean this one? I think it's the same llama-3-8b-instruct reuploaded by NousResearch? So they have the same chat template.

Also got repeated 128000 for this model:

INFO 05-09 11:52:28 async_llm_engine.py:524] Received request cmpl-9a55a4ffde904ef8b34a81af927aeea2: prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|
>\n\nHi!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, rep
etition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_
ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8179, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_betwe
en_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 128000, 128006, 882, 128007, 271, 13347, 0, 128009, 128006, 78191, 128007, 271], lora
_request: None.

Here are a few examples from other models:

https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/blob/main/tokenizer_config.json#L42

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/main/tokenizer_config.json#L119

https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer_config.json#L304

[DarkLight1337]

I am referring to the ones that are in vLLM repo, not those on HuggingFace.

[bofenghuang]

Certainly, done @simon-mo

[bofenghuang]

@DarkLight1337 I got your point. I'm not familiar with all these models, but falcon-instruct doesn't use the BOS token, so this PR won't affect it.

Here is a test with falcon-7b-instruct.

# server
python -m vllm.entrypoints.openai.api_server \
    --model tiiuae/falcon-7b-instruct \
    --chat-template examples/template_falcon.jinja \
    --dtype float16

# client
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "tiiuae/falcon-7b-instruct",
        "messages": [
            {"role": "user", "content": "Hi!"}
        ],
        "temperature": 0
    }'

Before this commit:

INFO 05-09 14:31:17 async_llm_engine.py:529] Received request cmpl-fcfdb44a2e0040c49b086b11db91b288: prompt: 'User: Hi!\nAssistant:', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=2041, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [7932, 37, 16588, 12, 193, 44295, 37], lora_request: None.

After this commit:

INFO 05-09 14:32:26 async_llm_engine.py:529] Received request cmpl-f8b11e3a3e7f4a1b983def4a258519f0: prompt: 'User: Hi!\nAssistant:', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=2041, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [7932, 37, 16588, 12, 193, 44295, 37], lora_request: None.

They have the same prompt token IDs, where 7932 represents "User".

I believe the idea of the chat template is to include all special tokens, as explained in the doc. So we should just update these templates if necessary.

If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes handy support for generation prompts, but note that it doesn’t add BOS or EOS tokens! If your model expects those, they won’t be added automatically by apply_chat_template - in other words, the text will be tokenized with add_special_tokens=False. This is to avoid potential conflicts between the template and the add_special_tokens logic. If your model expects special tokens, make sure to add them to the template!

[DarkLight1337]

@DarkLight1337 I got your point. I'm not familiar with all these models, but falcon-instruct doesn't use the BOS token, so this PR won't affect it.

It would be great if you could add some test cases to avoid regression issues.

[simon-mo]

@DarkLight1337 can you help take another look and let me know whether this is mergable?

[DarkLight1337]

The CI tests fail so those have to be addressed first.

[bofenghuang]

Thanks for pointing that out. I tried to fix it but still encountered issues. I don't quite understand the error since "Any" has been imported. It passed on my end, though.

[DarkLight1337]

Thanks for pointing that out. I tried to fix it but still encountered issues. I don't quite understand the error since "Any" has been imported. It passed on my end, though.

Try merging the current main branch into your branch.

[bofenghuang]

Oh, you are right. It was deleted in this PR. Just added it back.

[DarkLight1337]

Seems that the existing tests assume the addition of bos token. Can you update them accordingly?

[DarkLight1337]

Please also check whether this is consistent with the behaviour of the official OpenAI API.

[bofenghuang]

Seems that the existing tests assume the addition of bos token. Can you update them accordingly?

Hey, could you point me to the relevant tests?

Please also check whether this is consistent with the behaviour of the official OpenAI API.

Which behavior would you like to check?

[DarkLight1337]

Hey, could you point me to the relevant tests?

You can check the logs of the failing tests in CI.

Which behavior would you like to check?

Whether openai.types.CompletionUsage.completion_tokens == openai.types.CompletionUsage.prompt_tokens in general, which would be the case if BOS token is not added by default.

[bofenghuang]

Just realized that the modified _validate_prompt_and_tokenize function is used for completion serving, chat serving, and embedding serving.

However, we only need to disable add_special_tokens for chat serving, as these special tokens are already introduced by the chat template. We still need to enable it for the others. Therefore, I added this argument with a default value of true and only disabled it for chat serving. Now, all the CI tests have passed.

[bofenghuang]

Whether openai.types.CompletionUsage.completion_tokens == openai.types.CompletionUsage.prompt_tokens in general, which would be the case if BOS token is not added by default.

Not sure I fully understand. Why should we compare input (prompt_tokens) with output (completion_tokens)?

[DarkLight1337]

Whether openai.types.CompletionUsage.completion_tokens == openai.types.CompletionUsage.prompt_tokens in general, which would be the case if BOS token is not added by default.

Not sure I fully understand. Why should we compare input (prompt_tokens) with output (completion_tokens)?

Originally I wasn't sure whether you should modify the existing tests or update the tests to comply with the new behaviour. If it's the latter than I wanted to make sure that it doesn't violate any invariants in OpenAI's API. Now that the tests pass I guess it can be merged now. @simon-mo

[bofenghuang]

@DarkLight1337 thanks for the discussion!