.Net: Feedback on SK C# Beta 8

[IntegerMan]

I was encouraged by Devis Lucato to post an issue with early feedback on Semantic Kernel and C#.

First, some context on me and my usage. I'm an AI specialist at work and so I'm learning SK as a viable AI orchestration framework for us to implement for our clients. However, I'm also a Microsoft MVP in AI, a retired educator, and a blogger / speaker / author / general content creator. A lot of my interest for these types of things is in learning so I can teach and recommend things for the community. As such, I look at SK in terms of how I explain it to others, how I demo it, etc.

Things I love:

I love the idea of the project and the role it plays in AI orchestration. I also love that so many things revolve around Plan objects. That's nice and workable.

SK is powerful and its planners, frankly, terrify me in a good way.

The familiar Builder pattern works very nicely and the built-in extension methods really help you get started well.

The core plugins are great and I love their inclusion.

Existing documentation is quite good and sometimes even ahead of the code. For example, documentation referenced things on Assistants that were not even available in the most recent NuGet packages as of a half week ago or so.

I love the Polyglot Notebook integration for SK.

Things that are challenging:

There's a lot of inconsistency between the different planners. Their interfaces are all different and not standardized. This is challenging when you're trying to figure out which planner you want to run with. It's more challenging when you want to demo it to people in a conference talk. Some planners let you RunNextStep while others require a RunAsync. Plan vs PlanAsync. Things like that. Planners also use metadata very differently, and the way you get out things from the plans varies by each planner. It generally feels like Planners don't really respect the Liskov Substitution principle, because when you change which planner you use, you tend to have to work with the Kernel slightly differently.

SKException gets thrown a lot when it shouldn't, particularly on Sequential Planner. That planner just struggles in general because it tries to do too much and it just doesn't scale. It's very frustrating to have a small set of steps and know a plan is viable, but get SKExceptions for things that the planner can't figure out without much pointers into how it thought or what you can do to improve things.

There's a constant struggle on tokens and trying to minimize your descriptions while keeping them handy for your planner.

Not all planners trigger the events on the kernel that indicate tokens are being used. I had to do some log parsing with the logger factory in order to get at token usage from StepwisePlanner, for example.

The sequential planner tends to generate invalid results that print out variables in their final strings.

Embeddings with Ada 2 don't seem to perform very well for me. I think I'm likely doing something wrong there, but searching memory usually gives me bad results in my demo app.

Related: I need a version of VolatileMemory that serializes to JSON whenever a change occurs. This is a horrible thing in general, but it works really well for getting people set up for a quick demo at a workshop without having to worry about local database storage.

Any time I add a new plugin method or tweak some description I feel the need to test everything again and there's not a great answer for that yet with PromptFlow being web-based and not supporting C#.

I want to make my functions more modular so I can get good stable modules that I can test extensively and have the kernel coordinate between modules. I believe assistants are the answer to that, but I could be wrong. Ideally the planner would work at the assistant level, so SK would know which assistant or assistants to talk to and those assistants would use a planner (or the shared planner) to resolve requests only at that assistant level. This would limit the complexity of planning and the token usage as well.

Assistants are not yet usable with current architecture as of beta 8. I shared feedback on this on the 21st, but didn't get an acknowledgement on the feedback in Discord. Essentially I wanted to add a plugin to a single assistant and it didn't work and a lot of the workarounds from the documentation weren't syntactically valid in beta 8 (referring to SKPlugin for example, which wasn't a thing). My code snippet for that follows:

AssistantBuilder assistantBuilder = new AssistantBuilder()
    .WithName("Conference Concierge")
    .WithDescription("A virtual assistant to help search conference sessions and speakers")
    .WithInstructions("You can ask me to search for sessions, speakers, or other information about the conference. For example, you can ask me to find sessions about Azure or speakers by their names.");

// Add the Sessionize plugin to the assistant (Doesn't work - no method for this at the moment)
assistantBuilder.WithPlugin(new SessionizePlugin(Memory, settings.SessionizeToken!));

IAssistant assistant = assistantBuilder.BuildAsync().Result; // I'd love a synchronous build method

// This is in example 70 but doesn't work. Of course, Example 70 talks about SKPlugin, but that's not available in beta 8 either
myKernel.ImportPluginFromObject(assistant, assistant.Id);

Some planners can call functions just to trying things out and not commit to the results if they don't like them. That makes me worried if you have a function with a side effect, such as sending an E-Mail, appending to a log, or modifying a database. You almost need some mechanism to preview, commit, or rollback changes in some of these mutation-oriented scenarios.

---

Sorry, this is a lot of random feedback. I hope it has some merit to the team. This is just where I've been and I've put some of this in Discord in the feedback channel without getting much response and thought I'd put it here instead at your suggestion since some of these negatives are the negatives I'll share with audiences when talking about SK.

My general impression is that SK is awesome, but not yet testable or scalable in terms of managing large volumes of functions. It still has some baking to do, but it's a very promising technology. I'm committing to supporting it in the community, but my current support comes with a "you should watch this and wait for a few key things to slide into place".

My sample SK demo project I'm working on is at https://github.com/integerman/batcomputer if you want to see more what I'm talking about with the abstraction layers around planners or the desire to modularize different groups of plugins for added maintainability. I'm also very happy to answer whatever questions the team has about my feedback.