Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helicone-stream-force-format: true seems to cause JSON::ParserError when using Azure OpenAI #411

Open
rohitpaulk opened this issue Dec 19, 2023 · 1 comment

Comments

@rohitpaulk
Copy link
Contributor

Describe the bug

When working with the Azure OpenAI service, we seem to be running into occasional "JSON::ParserError" errors. These don't happen frequently, but if you run a script that runs 100 prompts, it almost always triggers the error.

We tracked this down to the "helicone-stream-force-format" parameter suggested in the README. When that's present, the bug triggers. When it isn't, the bug doesn't trigger.

The error looks something like this:

unexpected token at '{"finish_reason":null,"index":0,"deltdata: {"id":"chatcmpl-8XckshxV3cEBasKbvvUD4MDSpUgYJ","object":"chat.completion.chunk","created":1703024490,"model":"gpt-35-turbo","choices":[{"finish_reason":null,"index":0,"delta":{"content":"."}}]}' (JSON::ParserError)

To Reproduce

Run this script:

client = OpenAI::Client.new(
  access_token: ENV.fetch("AZURE_OPENAI_API_KEY"),
  uri_base: "https://oai.hconeai.com/openai/deployments/gpt-35-turbo",
  request_timeout: 240,
  extra_headers: {
    "api-key": ENV.fetch("AZURE_OPENAI_API_KEY"),
    "Helicone-Auth": "Bearer #{ENV.fetch("HELICONE_API_KEY")}",
    "Helicone-OpenAI-Api-Base": ENV.fetch("AZURE_OPENAI_ENDPOINT"),
    "Helicone-Property-Prompt": "TestUnexpectedTokenErrorsPrompt",
    "helicone-stream-force-format": "true"
  },
  api_type: :azure,
  api_version: "2023-03-15-preview"
)

100.times.peach(8) do |i|
  puts ""
  puts "Iteration #{i}"
  puts ""

  client.chat(
    parameters: {
      model: "gpt-3.5-turbo",
      messages: [
        {role: "user", content: "Write me 3 paragraphs about the topic 'OpenAI parsing needs to be fixed'"}
      ],
      response_format: {type: "text"},
      seed: i, # Default, can be overridden using parameters
      stream: proc do |chunk, _bytesize|
        if chunk.dig("choices", 0, "finish_reason") == "stop" # last message
          puts ""
          puts "Iteration #{i} finished."
          puts ""

          next
        end

        content = chunk.dig("choices", 0, "delta", "content")
        unless content.nil?
          $stdout.write(".")
        end
      end
    }
  )
end

Expected behavior

JSON::ParserError shouldn't be triggered.

@rohitpaulk
Copy link
Contributor Author

rohitpaulk commented Dec 19, 2023

Looks like the "helicone-stream-force-format" header was added in #251, and the work in #338 possibly negates the need for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant