fix: Gemini text chat - prevent sending broken messageContent and history

[parhammmm]

Gemini was completely broken for text chat and was getting confused when prompted with an empty context.

It is still broken for vision. Currently the mimeType is incorrect and inlineData isn't taken into account. Will submit a separate PR to address.

[changeset-bot]

🦋 Changeset detected

Latest commit: 2992cc0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
llamaindex	Patch
docs	Patch
@llamaindex/experimental	Patch
@llamaindex/cloudflare-worker-agent-test	Patch
@llamaindex/next-agent-test	Patch
@llamaindex/nextjs-edge-runtime-test	Patch
@llamaindex/waku-query-engine-test	Patch
@llamaindex/autotool-01-node-example	Patch
@llamaindex/autotool-02-next-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
llama-index-ts-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 8, 2024 3:38pm

[marcusschiesser]

@parhammmm looks great - do you have a test case that failed with the old implementation? if so, can you add it to 'examples'?

[parhammmm]

@parhammmm looks great - do you have a test case that failed with the old implementation? if so, can you add it to 'examples'?

@marcusschiesser

The test case would be the following, I'm not sure if you want the following in the examples as it requires a fake response from the llm? Let me know if that's ok and I'll add it or not

import { Gemini, GEMINI_MODEL } from "llamaindex";

(async () => {
  if (!process.env.GOOGLE_API_KEY) {
    throw new Error("Please set the GOOGLE_API_KEY environment variable.");
  }
  const gemini = new Gemini({
    model: GEMINI_MODEL.GEMINI_PRO,
  });
  const result = await gemini.chat({
    messages: [
      { content: "You want to talk in rhymes.", role: "system" },
      {
        content: [{ type: "text", text: "Who is woodchuck?" }],
        role: "user",
      },
      {
        content: [{ type: "text", text: "Woodchuck is a Bird" }],
        role: "assistant",
      },
      {
        content:
          "How much wood would a woodchuck chuck if a woodchuck could chuck wood?",
        role: "user",
      },
    ],
  });
  console.log(result);
})();

There were two issues:

1. role = assistant was being mapped to Gemini's user, causing the entire history to be merged into one
2. when MessageContent was of the type MessageContentDetail[] the message sent would be:

[
  {
    text: 'You want to talk in rhymes.\n' +
      '[object Object]\n' +
      '[object Object]\n' +
      'How much wood would a woodchuck chuck if a woodchuck could chuck wood?'
  }
]

[parhammmm]

@marcusschiesser I also have the Gemini Vertex implementation ready for PR after this is merged in

[marcusschiesser]

Thanks a lot @parhammmm - just tested it. Great work!

[marcusschiesser]

Looking forward to your Vertex implementation