Each of those is great, and more sophisticated strategies are being developed all the time: incorporating static analysis at inference time to ensure code is more than just syntactically correct [0]; to multi-LLM call agent-based frameworks [1]; decomposing hard tasks into many "easier" LLM tasks [2]; analyzing internal activations to identify potential hallucinations [3]; activation steering to control generations [4]; and "Language Model Arithmetic" to compose biases for style, vocab, etc [5].
But each also either requires tight integration with an LLM inference/serving engine (to access low-level internals of an inference engine); or adds a lot of overhead (many individual LLM calls) if they don't have tight integration.
The AI Controller Interface is creating an abstraction layer that exposes the low-level primitives so that all the above strategies can be implemented without each one needing to dive into the internals of every LLM engine. AICI doesn't support all the necessary primitives for all of these yet (e.g., we don't know what the right way is to represent internal activations); and not everything will end up fitting in a WASM module anyway.
Its a start at thinking about a new layer in the inference stack
But each also either requires tight integration with an LLM inference/serving engine (to access low-level internals of an inference engine); or adds a lot of overhead (many individual LLM calls) if they don't have tight integration.
The AI Controller Interface is creating an abstraction layer that exposes the low-level primitives so that all the above strategies can be implemented without each one needing to dive into the internals of every LLM engine. AICI doesn't support all the necessary primitives for all of these yet (e.g., we don't know what the right way is to represent internal activations); and not everything will end up fitting in a WASM module anyway.
Its a start at thinking about a new layer in the inference stack
[0] https://arxiv.org/abs/2306.10763 [1] https://github.com/microsoft/autogen [2] https://arxiv.org/abs/2305.14292 [3] https://arxiv.org/abs/2312.02073 [4] https://arxiv.org/abs/2308.10248 [5] https://arxiv.org/abs/2311.14479