How can I use images directly in the handlebar template?

[takeo-iw]

I want to use image directly in the handlebar template using gpt-4o. Is it not supported yet?

I tried the code below, but it returned "I'm unable to view images, but if you describe the image to me, I can help provide more information or assist with any questions you might have!".

const string ImageUri = "https://upload.wikimedia.org/wikipedia/commons/d/d5/Half-timbered_mansion%2C_Zirkel%2C_East_view.jpg";

var kernel = Kernel.CreateBuilder()
    .AddAzureOpenAIChatCompletion("gpt-4o", endpoint, key)
    .Build();

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

var chatHistory = new ChatHistory();
chatHistory.AddUserMessage(
[
    new TextContent("What’s in this image?"),
    new ImageContent(new Uri(ImageUri))
]);

string handlebarTemplate = @"
    <message role=""system"">You are a friendly assistant.</message>
    {{#each chatHistory}}
        <message role=""{{role}}"">{{content}}</message>
    {{/each}}";

var hanlebar = kernel.CreateFunctionFromPrompt(
    new()
    {
        Template = handlebarTemplate,
        TemplateFormat = "handlebars",
    },
    new HandlebarsPromptTemplateFactory()
);

var answer = kernel.InvokeStreamingAsync<StreamingChatMessageContent>(
    hanlebar!,
    new() {
        { "chatHistory", chatHistory }
    }
);
string streamingMessage = "";
await foreach (var chunk in answer)
{
    streamingMessage += chunk;
    Console.Write(chunk);
}
Console.WriteLine();

[dmytrostruk]

@takeo-iw If you want to use text and image contents in chat prompts (i.e. <message> tags) you need to explicitly specify the type of content using dedicated tags:

<message role='user'>
   <text>Text content</text>
   <image>https://image-url</image>
</message>

Let me know if that helps. Thank you!

[takeo-iw]

Thanks. I solved it.
Maybe I don't know how to write dedicated tags.
If you know, could you teach me URL of the documentation or sk sample on how to explicitly specify the type of content using dedicated tags?

[dmytrostruk]

@takeo-iw In this syntax we only have message, text and image tags at the moment, so you already know all the tags :) But we will definitely add more examples with text and image tags for visibility.