Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

.Net: Gemini function calling throws error when a kernel function with Enum parameter is available #5796

Closed
HillPhelmuth opened this issue Apr 7, 2024 · 4 comments · Fixed by #5998
Closed

.Net: Gemini function calling throws error when a kernel function with Enum parameter is available #5796

HillPhelmuth opened this issue Apr 7, 2024 · 4 comments · Fixed by #5998
Assignees
@RogerBarreto
Labels
bug Something isn't working .NET Issue or Pull requests regarding .NET code

Comments

@HillPhelmuth
Copy link

Describe the bug
When using function calling with Gemini ToolCallBehavior set to GeminiToolCallBehavior.AutoInvokeKernelFunctions, and the Kernel instance has a plugin function with an Enum parameter, the Gemini request will return a 400 (bad request) error.

To Reproduce
Steps to reproduce the behavior:

  1. Create Kernel instance with a plugin containing a function with enum parameter (TimePlugin is simple example to test).
  2. Create GeminiPromptExecutionSettings with ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
  3. See error when running InvokePromptAsync that includes the execution settings.

Expected behavior
Similar to current OpenAI function calling, the Gemini connector should handle necessary type changes when converting kernel functions to gemini function definitions to avoid these errors

Platform

  • OS: Windows
  • IDE: Visual Studio 2022
  • Language: C#
  • Source: Version 1.7.1 and 1.7.1-alpha

Additional context
Created example that demonstrates the issue - uses TimePlugin which causes error and TimePlugin with DateMatchingLastDayName function removed, which does not cause the error.
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code triage labels Apr 7, 2024
@github-actions github-actions bot changed the title Gemini function calling throws error when a kernel function with Enum parameter is available .Net: Gemini function calling throws error when a kernel function with Enum parameter is available Apr 7, 2024
@markwallace-microsoft markwallace-microsoft added bug Something isn't working and removed triage labels Apr 8, 2024
@Krzysztof318 Krzysztof318 mentioned this issue Apr 24, 2024
Merged
4 tasks
@cisionmarkwalls
Copy link

I get errors even when no enum parameters are used with AutoInvokeKernelFunctions.

As an example:

...
kernel.ImportPluginFromFunctions("HelperFunctions",
[
    kernel.CreateFunctionFromMethod(() => DateTime.UtcNow.ToString("R"), "GetCurrentUtcTime", "Retrieves the current time in UTC."),            
]);
GeminiPromptExecutionSettings settings = new() {
    ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
};
try
{
    FunctionResult result = await kernel.InvokePromptAsync(
       "What is the current time?", new(settings));

    // Display results
    var geminiMetadata = result.Metadata as GeminiMetadata;
...

Gives me a 500 error from Gemini for gemini-1.0-pro-002 and gemini-1.5-pro-preview-0409

@Krzysztof318
Copy link
Contributor

Krzysztof318 commented Apr 26, 2024 •
edited

You should check at first in google docs do selected versions of models support function calling.

And 500 is server error, so this problem is rather related to google api not connector.

@cisionmarkwalls
Copy link

You should check at first in google docs do selected versions of models support function calling.

Yes, both of those models have function calling. If I change GeminiToolCallBehavior.AutoInvokeKernelFunctions to GeminiToolCallBehavior.EnableKernelFunctions it does the function calling correctly.

@Krzysztof318
Copy link
Contributor

Krzysztof318 commented Apr 26, 2024 •
edited

I am checking google ai documentation link and I don't see models you tried to use.
You can use gemini-1.0-pro-001 and via alias gemini-1.0-pro or gemini-1.5-pro-latest.

Here are aliases for vertex ai link

And in your sample you used googleAi but with models from vertex.
Keep in mind function calling is experimental and not every model will work correctly.

github-merge-queue bot pushed a commit that referenced this issue May 13, 2024
@Krzysztof318 @RogerBarreto
.Net: Fix 5796 function calling enum params (#5998)
8a8cd95 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Fixes #5796

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Type for enum wasn't correctly set in JsonSchemaMapper, it didn't matter
for OpenAI but gemini was throwing exception if type isn't specified.
Fixed that with `string` type.
Added new unit tests for Gemini and OpenAI. Both passed.

@RogerBarreto 
@SergeyMenshykh

DataHelper and BertOnyx was updated automatically by formatter.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Roger Barreto <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RogerBarreto RogerBarreto

Labels
bug Something isn't working .NET Issue or Pull requests regarding .NET code
Projects
Semantic Kernel
Status: Sprint: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

.Net: Fix 5796 function calling enum params Krzysztof318/semantic-kernel
5 participants
@RogerBarreto @HillPhelmuth @Krzysztof318 @cisionmarkwalls @markwallace-microsoft