I originally built this to help with self-hosting Hugging Face models, but it evolved into an authenticated pull-through cache.
A big part of this project was an experiment to see how far I could get Cursor + (Opus/Fable) to optimize the hot path. I initially used fasthttp and Go's net/http, but after some deep optimizations, the only way to squeeze out more performance was to eliminate syscalls.
I ended up building two optimization paths that use a custom-built HTTP/1.1 parser: one for macOS using sendfile + sf_hdtr, and one for Linux using io_uring. For anyone interested in the threat model or how the custom parser is tested against Go's standard library, I wrote a detailed security breakdown here: https://pulsys.io/docs/security/
The result is that it can sustain 1.36M req/s at 4 KiB and 90 GB/s at 16 MiB on an EC2 instance (see benchmarks https://pulsys.io/docs/benchmarks/)
It drops right in front of existing clients (just set HF_ENDPOINT).
I’d love to hear your thoughts, feedback, or any questions on the AI-assisted optimization process!
I have 14 YOE and its the opposite. Really brilliant jrs don't have the experience to know the consequences of their designs, they build fast under pressure. They reinvent things that already exist (they don't know because they a Junior).
I worked in the music streaming industry, which involves ingesting 10's of millions of XML files. Absolutely do not use XSLT. Schema changes between different versions of DDEX were a nightmare for all the legacy XSLT we had.
I don't see any reason to use this for AWS over CDK. CDK already has some pretty higher level modules that make it really easy to setup ecs, k8s, vpc, ect.
Every framework have worked on for the last 12 years has bugs, breaks in certain situations. Every system in fact, i have worked on in 12 years has bugs and breaks. That is just the nature of software development.
You should be using frameworks instead of trying to reinvent things. It takes away decision fatigue and makes it easier for most teams to follow conventions that are already established.
auto scaling could also mean scaling to 0. If you are running GPU workloads in k8's, you would typically setup a node pool with gpus that can scale to 0 after the job runs.
I originally built this to help with self-hosting Hugging Face models, but it evolved into an authenticated pull-through cache.
A big part of this project was an experiment to see how far I could get Cursor + (Opus/Fable) to optimize the hot path. I initially used fasthttp and Go's net/http, but after some deep optimizations, the only way to squeeze out more performance was to eliminate syscalls.
I ended up building two optimization paths that use a custom-built HTTP/1.1 parser: one for macOS using sendfile + sf_hdtr, and one for Linux using io_uring. For anyone interested in the threat model or how the custom parser is tested against Go's standard library, I wrote a detailed security breakdown here: https://pulsys.io/docs/security/
The result is that it can sustain 1.36M req/s at 4 KiB and 90 GB/s at 16 MiB on an EC2 instance (see benchmarks https://pulsys.io/docs/benchmarks/)
It drops right in front of existing clients (just set HF_ENDPOINT).
I’d love to hear your thoughts, feedback, or any questions on the AI-assisted optimization process!