what if this wasnt something you add after infra but the checkpoint you start with. right now you spin up a vm or db then wrap vpn or firewall around it. but imagine writing access rules first in way : 'team ml can hit service x' or 'web app can hit this backend' and the system wires infra from that.. infra becomes a side effect of access intent. access isnt something you cant guard always( as things move fast, breaks fast), it's may become seed where you can design with.
lets say if i someone wants to keep using bittorrent dht for peer finding but swap out quic for something else maybe grpc, does the lib support that split clean? asking from a modular embed first tooling pov, where discovery logic needs to outlive or outswap transport depending on deployment
been running llama.cpp and vllm on same 4070, trying to batch more prompts for serving. llama.cpp was lagging bad once I hit batch 8 or so, even though GPU usage looked fine. vllm handled it way better.
later found vllm uses paged kv cache with layout that matches how the GPU wants to read fully coalesced without strided jumps. llama.cpp was using a flat layout that’s fine for single prompt but breaks L2 access patterns when batching.
reshaped kv tensors in llama.cpp to interleave ; made it [head, seq, dim] instead of [seq, head, dim], closer to how vllm feeds data into fused attention kernel. 2x speedup right there w.r.t same ops.
GPU was never the bottleneck. it was memory layout not aligning with SM’s expected access stride. vllm just defaults to layouts that make better use of shared memory and reduce global reads. that’s the real reason it scales better per batch.
this took its own time of say 2+days and had to dig under the nice looking GPU graphs to find real bottlenecks, it was widly trial and error tbf,
> anybody got idea on how to do this kinda experiment in hot reload mode without so much hassle??
haha wait do you actually read long commit messages( more than a line) all the way through? like line-by-line, imo commit msg = tweet, git note = blog post.
i use git notes pretty heavily in my current role. started as an experiment to keep track of internal code reviews without flooding the commit message or making PRs for everything. i tag every commit with context what tickets it maps to, infra constraints, links to incident threads if it's a fix. all lives in the repo. this avoids the need to grep slack or jira just to know why a line changed. nce you start using it at scale, you realise how little you need the platform UI at all. we keep talking about reproducibility in builds, but never in intent. maybe this is where that starts