A Visual Planner with Support for Iterative Planning

[iassafc]

In many use cases, it might be beneficial to show a user the intended plan and ask for their approval, especially when planning a long-running or otherwise costly operation. This would also allow the user to optionally iterate on the plan, making corrections and additions until a satisfactory result is reached.

While a textual representation of the plan could also be generated, certain expert users may benefit more from a visual representation of the actual plan.

For these scenarios, a block-based visual programming language might be suitable as a planning DSL:

The above are tests that I've made with Google's Blockly, an open-source library, which forms the foundation for some of the largest block-based coding products, such as Scratch and Code.org.

Blockly can serialize its programs into both XML and JSON. I found that the older XML representation, which has been around for over 10 years, is very familiar to GPT-4 and it is able to generate Blockly XML code without any complex prompt instructions.

To illustrate this concept, I developed a frontend-only HTML/JS demo. This demo showcases live sessions with the 'gpt-4-0125-preview' model, with the playback speed increased for brevity.

The next recording shows use of custom blocks that were fed into the context:

A Blockly-based SK Planner could execute such plans server-side using a Blockly interpreter.
For reference, please see IronBlock: https://github.com/richorama/IronBlock

Looking forward to receiving community feedback.