Malformed JSON values when using json_schema

[amit13k]

I am trying to use llamacpp with jsonschema but in some cases I am seeing JSON like this {"value": "2}, "}. I am getting valid JSON which gets parsed correctly, but the values seem to be wrong. It seems the model is trying to close the JSON object inside the value. This output is from llama-3-instruct-Q5 K_M.gguf. Any idea what could be wrong or suggestions to improve this?

[phymbert]

Please wait for llama3 support to be confirmed and merged here:

• Support Llama 3 conversion #6745
• Added llama-3 chat template #6751

There is a chance that some special tokens are not well mapped in your GGUF.

[amit13k]

Thanks a lot for the reply. I will track these issues.

[ggerganov]

Please provide repro steps

[amit13k]

Model

Meta-Llama-3-70B-Instruct-Q5_K_M.gguf
https://huggingface.co/LoneStriker/Meta-Llama-3-70B-Instruct-GGUF/tree/main

Metadata modified for eos token using this command,

scripts/gguf-set-metadata.py 'models/Meta-Llama-3-70B-Instruct.Q5_K_M.gguf' tokenizer.ggml.eos_token_id 128009

Json schema

{
    "type": "object",
    "properties": {
      "minimumValue": { "type": "string" },
      "maximumValue": { "type": "string" }
    },
    "required": [ "minimumValue", "maximumValue" ],
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-07/schema#"
}

Converted grammer

char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F])
maximumValue-kv ::= "\"maximumValue\"" space ":" space string
minimumValue-kv ::= "\"minimumValue\"" space ":" space string
root ::= "{" space minimumValue-kv "," space maximumValue-kv "}" space
space ::= " "?
string ::= "\"" char* "\"" space

Completion Prompt

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful, respectful and honest assistant.
<|eot_id|><|start_header_id|>user<|end_header_id|>
Output correct values only from the provided context as json.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Completion

{"minimumValue": "} 

Please provide the context so I can assist you in outputting correct values as JSON. I'll make sure to be helpful, respectful, and honest in my response. 😊 Waiting for your input... 🤔

(Please note that the initial response was an error on my part. Please disregard it.) 😉️ 

Please provide the context now! 👍️💬󠁧󠁢󠁳󠁣󠁴󠁿

(If you need help or have any questions, feel free to ask!) 😊

I have created an extreme prompt to trigger this behaviour. I can see that even the json is not parseable in this case(i suspect even newlines might be generated in json field values). I thought, I will always get parseable json. In some other random cases, i get parseable invalid values. I might be completely wrong in my expectations and llama-3 support is also not completely confirmed, so looking forward to all suggestions.

[ggerganov]

With latest master and no extra overrides, it seems to work as expected:

$ cat prompt.txt
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful, respectful and honest assistant.
<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the temperature range in New York. Output correct values only from the provided context as json.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

$ cat gram.txt
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F])
maximumValue-kv ::= "\"maximumValue\"" space ":" space string
minimumValue-kv ::= "\"minimumValue\"" space ":" space string
root ::= "{" space minimumValue-kv "," space maximumValue-kv "}" space
space ::= " "?
string ::= "\"" char* "\"" space

$ ./main -ngl 99 -c 0 -e -f prompt.txt --grammar-file gram.txt -m models/llama-70b-v3-instruct/ggml-model-q5_k.gguf --verbose-prompt --temp 0.1

...

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful, respectful and honest assistant.
<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the temperature range in New York. Output correct values only from the provided context as json.<|eot_id|><|start_header_id|>assistant<|end_header_id|>{"minimumValue": "-15°C (5°F)", "maximumValue": "38°C (100°F)"}<|eot_id|> [end of text]

Try to reconvert and requantize the model using the latest llama.cpp and it should work

[amit13k]

Thanks for the reply. I used the latest master but I noticed I had a newline at the end of <|end_header_id|> in the prompt. After removing the newline, it seemed the problem is fixed. But still there were some prompts for which the issue reappeared. I have also tried some other constrained generation libraries like outlines-dev and saw similar issues. I just want to sure if it's a limitation of the model (which i am guessing it is), or is there something obvious that i am doing wrong. The problem happens randomly, so i will need some time to prepare the prompts/reproduction for this.

Also, I have noticed that with json schema based generation sometimes I am seeing weird values like, "github", "http://" getting inserted in the json fields. "github", "http://" are not present in the prompt and also not related to the question/answer. Without constrained generation(just asking model to output json), I am not seeing this issue. Is this a limitation of json_schema grammar approach? Does this method reduce the accuracy noticiably?

[ggerganov]

Hard to say. My experience with grammar is that it works as it is supposed to, but it's very easy to make a mistake in the grammar definition and think that there is a bug, while in fact the grammar is not defined properly. It takes some debugging to figure out where things go unexpected