I have been working on a project that is built off of Cloudflare's pingora project - https://sbproxy.dev. This provides all of the features of a LiteLLM, but can also support your API traffic too. I worked on the edge infrastructure for Walmart, and took those learnings and build a fully open source gateway.
Take a look - should be able to address your needs.
Another reason golang is interesting for the gateway is having clear control of the supply chain at compile time. Tools like LiteLLM the supply chain attacks can have more impact at runtime, where the compiled binary helps.
SBproxy (https://sbproxy.dev) - An API and AI gateway as a single binary. Sits between your apps and LLM providers to give you more control. Think Cloudflare meets LiteLLM.
After the LiteLLM supply chain hack last month I feel pretty good about that choice. Your LLM gateway holds every provider API key you have. That probably shouldn’t be a pip install.
Worked on API gateway for a major retailer and incorporated my learnings into this platform.
Entropy is real. Microsoft has lessened the friction of purchasing vs their competitors. Public Sector this may be their only choice because they set the groundwork decades ago to do business with most organizations. It may not be the best solution, but it offers the least resistance to getting something up and running.