They were originally on their own datacenters + huge amounts of burst and ancillary stuff in AWS, the internal push to move away from "the competitor's cloud" after the acquisition was huge and entirely stupid. I'm ex-GitHub and was one of the annoying people constantly saying GitHub should only move where it was provably the best option for GitHub - the third best major cloud provider is likely never it on merits alone.
If this story is true, it's good that they finally realised that GitHub's performance and availability mattered more than using Microsoft's products. It would mean someone finally came to their senses rather than forcing a wholesale push to Azure - but I bet they still want to have it both ways even if they concede some AWS now.
I think people spend way too much time trying to say that LLMs are bad / shouldn't be used / etc because the LLM can't get it right the first time and/or makes mistakes. I think this is because we all hope that software/computers work like this in an ideal world, and LLMs are software.
This is the wrong mental model.
The way to think about an LLM is like a human: prone to following bad examples if it sees them, needs guardrails to catch mistakes, needs code review. It also needs access to what "correct" looks like: architectural design documents, skills that explain each type of change, etc. It needs prompting/skills telling it to follow a safe workflow, telling it to consider how a safe rollout would work, what a safe rollback would look like, what the performance implications are - just like a human.
The nice thing is that you now have a very knowledgable assistant that can help write additional guardrails that would have always ended at the bottom of your backlog. Perhaps it used to take many hours to research and understand how to write a custom linter to catch a specific coding pattern. Today, ask Claude to do it and an hour later you'll have a custom linter rule for your language of choice, guaranteeing the same mistake can't happen again because CI will block it.
This. A ~staff software engineer designing big changes at one level above the raw implementation details using Opus 4.7 + superpowers today can genuinely ship multiple times more at the same quality level than pre-AI. The level of what a whole team could ship before.
You have to use something like superpowers, the key is that the humans need to make the important decisions.
You have to review the code - just like you had to review the code humans wrote. There will be iterations.
You have to give the LLM skills and patterns to follow, access to architectural documents, etc, just like humans needed to be onboarded at a company and do the same.
If you get all of these right with today's LLMs, you will never write code at all because it is so obviously not the best use of your time. If you feel that you are still better at writing the code manually, you have not done the above right, fix your workflow and try again.
I used to be a hubber - I really hope that institutional change happens at GitHub and I'm glad there are people there still fighting for it. I left because caring deeply (maybe, like the OP, too deeply) for the platform wasn't enough to shift many layers of leadership to focus on performance and reliability. Watching it crumble and lose its early-days focus on engineering excellence was just awful.
Even when we had a CEO telling people to focus on making GitHub Fast (<3), telling people to do what made GitHub better and not what people thought would make Microsoft better or make their jobs more secure to MS (because Microsoft bought GitHub because GitHub was the important thing, not so it could be a consumer of Azure) - it was still an uphill fight to change mindset and it didn't go anywhere meaningful.
I really hoped that others would be enable to enact that change after I left, but everything coming out of GitHub makes it look like the core focus is still not on making it fast or reliable or just better overall for people. If a blog post about GitHub's availability still includes the world's third-best major cloud provider as name drop on a solution, people are not focusing on making GitHub better. They are focusing on being part of Microsoft. Make GitHub better first, buy more super fast computers with the fastest everything you can buy, or at least use the best or second best public cloud provider, then work out how to leverage Microsoft to do it in-house or cheaply later. A GitHub that nobody wants to use is a bad investment for Microsoft.
Also, rip out 90% of default feature bloat integrations from initial page loads, add them as discoverable drill-downs or optional things to enable if people need it. Adding every feature under the sun (and copilot) on every place you can think of might read nice, but if you then fail people's most basic needs, you're worse off. GitHub still needs to function as the basic tool for all the future stuff to work. Accept the agentic coding paradigm shift and focus on making the tools faster for the hot paths rather than every path - and yes, that means removing stuff from the hot paths. GitHub today is like a maze, layers of settings and features, like nobody is thinking through UX anymore. This seems unrelated to performance, but this sort of "throw everything at the screen" style of UX is also what leads the backing systems to crumble under load.
I can't speak to the last few years since I left, but over the many years I was there the git storage layer was almost never the core issue - it was well designed by infrastructure-minded nerds that leveraged and improved git and replicated it really well across multiple nodes.
What always struggled was the richness of the Rails monolith itself and its backing MySQL databases - the expectation that everything links to everything throughout the product (think: issue cross references across orgs that only appear if you're able to access the remote repo, and other things like that).
Those details appear richly everywhere you look, and the combination of that with a general lack of understanding and/or focus on performance (shipping big features is hard, shipping them with performance at scale is MUCH harder), compounded by Ruby being an easy language to get performance wrong in (object count really hurts, and it makes it very easy to create many) leaves every feature adding to the performance problem, and makes it daunting/impossible to make fast once it's slow.
There was a full on year or more of making GitHub fast while I was there that just couldn't gain enough momentum to make enough of a dent to make it better. I remember finding and fixing a N^3 (or maybe it was N^4? something bad) in the home page activity feed - the worst thing I found but gives an idea. IMO it would need a fresh view of how to keep interfaces simple and how to design the data layers performantly - not adding every bell and whistle to every screen.
I hope someone at GitHub realises they are about to lose everything that was hard earned by early GitHub - it once was a site people (myself included) looked up to for ideal availability, responsible releases, data driven improvement - but no more it seems :(
Thanks for the suggestion! I did come across the `tcpsynbl.bt` script as I was writing up this post, but wanted to add the additional information around namespaces and report additional information, which didn't seem as trivial in `bpftrace` as it was in Python, but that might be my lack of familiarity with the DSL :)
We didn't find any metric that surfaced zombie cgroups, presumably because the kernel mostly tries to hide them from user space since they have been deleted, but haven't been cleaned up. The only way we found at the time to track them was via a BCC script and observing the latency on reading the /sys/fs/cgroup/memory/memory.stat file.
This insert_failed issue described in the video was one of the ones we discovered during investigations as well, but it was already well understood because of this excellent Xing blog post which was extremely useful and referenced internally a lot: https://tech.xing.com/a-reason-for-unexplained-connection-ti...
This is actually interesting, we chose to use an overlay network because Kubernetes was a new, complex system that we initially didn't have experience with internally, and so we wanted to isolate problems it could create as much as possible, as well as ensure that teams working on Kubernetes deployment weren't blocked by needing network engineering time.
This meant that we felt an overlay network was the most pragmatic (works out of the box) solution, even though IPIP has some drawbacks in adding complexity (within the Kubernetes cluster) and hashes across router ECMP / NIC RX queues poorly (neither look into the inner IP packet as part of hashing). It was definitely a concern of ours, though, considering we'd very intentionally avoided IPIP in our load balancer (https://github.blog/2018-08-08-glb-director-open-source-load... has a write-up of why).
We've considered the alternative of announcing a /24 or /25 or similar via BGP on each Kubernetes node, which is supported by systems like Calico, but so far the limitations of IPIP haven't caused an issue because of our Kubernetes clusters being large enough to hash evenly regardless of the addition of an overlay network, so we haven't needed to migrate. It's definitely an interesting trade-off on complexity overall, though - simple routed networks are much easier to understand than another layer of encapsulation.
Initially a few teams around the org had folks investigating poor performance from different perspectives of the applications that were observing issues. Once it was clear that it wasn’t the applications themselves or their configuration at fault, the team that runs our Kubernetes infrastructure started collating information together (in github issues) and getting to the point of having a clear repro (the Vegeta test) and what to look out for. This was the slowest part of the process because we needed to understand that something non-application-level was going on (and because “random network latency” is a very difficult thing to narrow down) - it probably took on the order of months from the first sign of an issue to fixing all the other issues that were contributing to small amounts of latency and being sure we still had an underlying problem to find.
At that point it became clear that something more low level was going on, we put together a focus team from a selection of teams to investigate the underlying cause - that was a group of about 5 engineers actively working on it, with another 5-10 interested engineers following along and helping out. Folks were typically working in pairs or solo to dive in to different potential leads, looping in everyone else in Slack as they go. Most of the work here was finding signal in the noise, we found a lot of other smaller system-level issues along the way that got ruled out and/or low priority to fix. There were other DNS related issues at play, fixing those also improved things, but not the specific underlying issue in the post here. Going down the specific path in the post took just a few days once the first few steps showed something was wrong at the packet level. The remediation from there was also just a few days, because we already had infrastructure in place to detect a known issue and mitigate in a safe/graceful way. The focus team was working on this as a primary task for around a few weeks overall.
Fastly's MAC-based solution to this was actually one of the existing implementations we read about back when designing the original implementation of GLB in 2015/16, along with Facebook's IPVS-based solution. We loved the ideas behind Fastly's model, but didn't want to mess with Layer 2 to do it. GLB Director took some inspiration from both designs in the creation of L4 second chance and the L4/L7 split design.
We found that we could achieve 10G line rate with just the queues available to the VF, the NIC didn't seem to be a bottleneck providing DPDK was processing packets faster than line rate. It's worth noting that other traffic on the PF was/is minimal in our setup.
We tested this using DPDK pktgen on a identically-configured node (GLB Director and pktgen both using DPDK on a VF with flow bifurcation, on 2 separate machines on the same rack/switch), with GLB Director essentially acting as a reflector back to the pktgen node. pktgen was able to generate enough 40 byte TCP packets to saturate 10G with 2 TX cores/queues, and GLB Director was able to process those packets and encapsulate them with a sizeable set of binds/tables with 3 cores doing work (encapsulation) and 1 core doing RX/distribution/TX.
During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in these attacks.
Since SYN packets only contain a limited set of information, if the SYN packets have spoofed source addresses then it is very difficult for a device in the destination network to filter/mitigate a SYN flood, since they look like legitimate SYN packets from many different clients. That said, if it's a non-spoofed attack, then you can definitely filter them at the edge. For spoofed attacks, if you filter from multiple global POPs (as do many DDoS scrubbing services), you may be able to guess that a packet is arriving at an inappropriate POP given the source address, but even that will only let you filter a certain amount of the traffic. Because of that, you still need some level of protection at the destination server.
In general in the order of a hundred thousand PPS per core is easily possible, but it really depends how busy the server is, since it primarily comes down to how much CPU is being used to generate the syncookies instead of being allocated to the application.
Great question! A large portion of our infrastructure still runs on Ubuntu Precise, and so the default "supported" options for the kernel on those machines are 3.2 or 3.13. Once you're on newer releases you get 4.x support, at the very least in backports, and at that point I would definitely agree the upgrade is worthwhile.
If this story is true, it's good that they finally realised that GitHub's performance and availability mattered more than using Microsoft's products. It would mean someone finally came to their senses rather than forcing a wholesale push to Azure - but I bet they still want to have it both ways even if they concede some AWS now.