An llm Plugin Hack

The llm package from Simon Willison has a plugin architecture. So I decided to try it. Any reasonable person would begin with a new model plugin. I decided instead to create llm rag, an encapsulation of rag queries that otherwise requires a pipeline of invocations of both llm and files-to-prompt.

The model for this plugin is approximately this:

llm similar "PROMPT" \
  | jq -r '[ "<document>", .content, "</document>"] | join("\n")' \
  | { printf '<documents>\n' && cat - && printf '</documents>\n%s'; } \
  | llm -m "$(model)" "PROMPT"

This works for its narrow purpose, but is rather inflexible with respect to the many options that llm offers. How do we enable use of those options in this context?

Creating this plugin makes one thing clear. This problem maps very poorly onto the exiting llm implementation architecture. So the result is very messy. The rough idea is to pull in llm code to parse arguments, then run a similarity prompt, package the outputs, and then run the same prompt with the similarity results as context.

I did this two ways, and the results are here. The first is on the master branch. It processes the requests with code adapted from llm and files-to-prompt. An unfortunately large volume of code is required for this, as it pulls from multiple levels within the llm implementation. Some options from the original commands are not supported, including templates and conversations.

A second approach is on the subprocess branch. This one focuses on parsing and filtering options, and then runs llm in a subprocess for each stage of processing. This is slightly less messy, and handles all options from each stage, modulo some minor disambiguation.

It still borrows far too much code from llm, mostly to deal with the variety of prompt constructions, in particular templates. Adding a template option to llm similar would clean this up some.

Most of the remaining mess pertains to the crude reconstruction of the command-line arguments for the subprocess. Perhaps using -- separators between options sections is a better approach. Maybe there is a feature of click that I overlooked. Even if so, the code does show some simple ways to work with the structures from click. If you are interested in creating your own llm plugin, this is probably a good anti-pattern example.