1. You still have to manually set up and run your own inference server (e.g., with ollama, lmstudio, vllm, etc.).
2. You need to ensure that the model you choose works well with Codex's pre-defined prompt setup and configuration.
3. Prompting patterns for small open-source models (like phi-4-mini) often need to be very different - they don't generalize as well.
4. The function calling format (or structured output) might not even be supported by your local inference provider. - Write the tool specifically to run _locally_ out of the box, no inference API server required.
- Use model directly (currently for phi-4-mini via llama-cpp-python).
- Optimize the prompt and execution logic _per model_ to get the best performance.
1. You still have to manually set up and run your own inference server (e.g., with ollama, lmstudio, vllm, etc.).
2. You need to ensure that the model you choose works well with Codex's pre-defined prompt setup and configuration.
3. Prompting patterns for small open-source models (like phi-4-mini) often need to be very different - they don't generalize as well.
4. The function calling format (or structured output) might not even be supported by your local inference provider.
- Write the tool specifically to run _locally_ out of the box, no inference API server required.
- Use model directly (currently for phi-4-mini via llama-cpp-python).
- Optimize the prompt and execution logic _per model_ to get the best performance.