Stream only JSON values from function call

[hayescode]

I would like to stream only the JSON values (not keys) to the UI.

For example, I have a function that writes SQL then I execute it and submit the results as output. The final JSON looks like this:

{
  "sql_query":
  "SELECT * FROM some_table"
}

Parsing after it's done is easy and works great, but my users want to see the SQL being generated, especially on much longer queries they stop the run because they think it's hung.

I was thinking to check for ": then start streaming the characters after that until ", or "} is reached, but I wonder if anyone has a better way?

I'm using Assistants API and the AsyncStreamHandler helper that was just released.

[RobertCraigie]

Hey @hayescode, you may be interested in the recently added from_json function in Pydantic: https://docs.pydantic.dev/latest/concepts/json/#partial-json-parsing. I think this would be exactly what you're looking for.

[hayescode]

Thanks @RobertCraigie very much for sharing this new functionality! If I "pre_close" the values to fake valid JSON this works perfect for streaming the value tokens kind of like streaming code interpreter inputs!

For example, GPT4 decides to use a tool and ultimately gives us {'sql_query': 'SELECT * FROM some_table', 'schema': 'default'}. The tokens might look something like this:

import time
from pydantic_core import from_json

stream = ['{', ' ', ' "', 'sql_qu', 'ery":', ' "', 'SELEC', 'T * FR', 'OM ', 'som', 'e_tabl', 'e"', ',"sche','ma": "','default','"}']
buffer = ''
for token in stream:
    buffer += token
    try:
        pre_close = buffer + '"}'
        result = from_json(pre_close, allow_partial=True, )
        print(result)
    except ValueError as e:
        continue
    time.sleep(0.1)

this gives the following result:

{}
{}
{}
{'sql_query': ''}
{'sql_query': 'SELEC'}
{'sql_query': 'SELECT * FR'}
{'sql_query': 'SELECT * FROM '}
{'sql_query': 'SELECT * FROM som'}
{'sql_query': 'SELECT * FROM some_tabl'}
{'sql_query': 'SELECT * FROM some_table'}
{'sql_query': 'SELECT * FROM some_table', 'schema': ''}
{'sql_query': 'SELECT * FROM some_table', 'schema': 'default'}
{'sql_query': 'SELECT * FROM some_table', 'schema': 'default'}

Then I send the value tokens to the UI, it looks like this:

SELEC
T * FR
OM som
e_tabl
e