Compilation to LCEL

[AriMKatz]

I'm not sure if this is feasible or desirable by the outlines team, but here goes:

Langchain has a lot of production conveniences around logging, streaming, async, retriever modules, agents, deployment etc. It's IR (intermediate representation, not information retrieval :) ), LCEL, seems to be rapidly becoming something of a standard.

There's also the new ability to autotune lcel with dspy :https://twitter.com/lateinteraction/status/1745134421219639513

Some info about dspy here: https://twitter.com/lateinteraction/status/1736119027997831210 The docs have been revamped and the author is very responsive on twitter.

My vision for the perfect LLM library is something like this: #528 (backend agonistic high level typed application interface with polymorphism) with the ability to drop down to some kind of kernel language to guide generation, with production conveniences, integration with popular modules and new techniques and then having certain functions marked for autotuning prompts/ finetuning of function specific t5/llm/slm models based on loss/eval. dspy provides the latter. I think this issue and #528 would round out the essential components of this vision.

I'm not sure if all this is at odds and some really fundamental tradeoffs have to be made. To me it makes conceptual sense that dspy use constraints given by programmers as inductive biases, and have these programs wrapped up into typed composable function boundaries for software engineers. Or perhaps initializing dspy programs with an outlines prompt like a bayesian prior.

At the very least, it makes sense for some functions to be autotuned with high level typing and some to just be manually guided.

Anyway, hopefully this is all conceptually cogent and intelligible even if not being technically feasible.

This vision was fleshed out a bit here: https://twitter.com/lateinteraction/status/1739471180938269158 with the lead author of dspy, but minus the constrained generation kernel language part.

[AriMKatz]

Here: stanfordnlp/dspy#264 (comment) Is where @okhat touches on outlines+dspy

[okhat]

Hey @AriMKatz , I think LCEL is great but is probably not the right construct for this conversation.

Outlines is much lower level than that, and that's its power!

I will chat with Remi in 1-2 days (we're scheduling) to get this sorted out. In short, controlled generation is the lowest level above LMs and is one of the most versatile. Above that there's the level of compilers / optimizers, aka DSPy's core. Then there's application dev, where LCEL comes in and other things in langchain.

[AriMKatz]

Awesome @okhat ! Excited to see what comes out of this! Makes sense that LCEL is too high level, but I was thinking that it may be possible to wrap outlines programs into LCEL intrinsics (are these "runnables"). Maybe I'm a bit confused about where LCEL is supposed to sit in that schema. It doesn't seem like a particularly great way to program on the application side, feels like you're manually constructing a DAG, when it should be define by run (or compiled by tracing or AOT). That's what @shroominic and I have discussed with funcchain

Edit:

So something like Normal typed python with pydantic support (funcchain like api) -> LCEL->DSPY-> Guidance programs I guess? With the possibility of seeding the compiler with hints from function typing, tests and maybe guidance, or having the capability to manually denote guidance boundaries like writing your own cuda calls. Anyway just brainstorming here.

[shroominic]

I don't think LCEL is too high level you just need to patch the model part of LCEL to force a desired structure. For example you can bind grammars or a json schema to chat models and still use them normally in runnable pipelines. Maybe there could be a function that takes a langchain LLM/ChatModel and patches it so it is then controlled by Outlines.