Incident with Git Operations, Issues and Pull Requests [resolved](githubstatus.com)
githubstatus.com
Incident with Git Operations, Issues and Pull Requests [resolved]
https://www.githubstatus.com/incidents/k1qts16jgtcp
56 comments
How stupid are we all to rely so heavily on this single point of failure which is so unstable?
I cannot count the times it's been down or had temporary instability. It's easy to draw the conclusion that it happened after MS took over, but I honestly no longer remember how it was before.
I cannot count the times it's been down or had temporary instability. It's easy to draw the conclusion that it happened after MS took over, but I honestly no longer remember how it was before.
We're a primarily GitLab shop so I have a biased view, but I haven't noticed that much disruption from GitHub's instability. We do get a lot of packages from GitHub too and I haven't noticed a ton of CI jobs failing or anything.
> We do get a lot of packages from GitHub too
Do you get them directly from GH or from a local mirror of the GH repo? The latter makes you immune to GH hiccups at the cost of some disk space (assuming you run GL yourself).
Do you get them directly from GH or from a local mirror of the GH repo? The latter makes you immune to GH hiccups at the cost of some disk space (assuming you run GL yourself).
Should be directly from GitHub except with some caching involved (like our Docker build cache).
(We don't run GitLab ourselves.)
(We don't run GitLab ourselves.)
It is every month, something in Github (especially GitHub Actions) goes down. The last time this happened was less than a month ago. [0] [1]
At this point, It is so unreliable that if I was to bet anyone that something in GitHub would go down next month, I will win each time. Guaranteed.
[0] https://news.ycombinator.com/item?id=35359212
[1] https://news.ycombinator.com/item?id=35325709
At this point, It is so unreliable that if I was to bet anyone that something in GitHub would go down next month, I will win each time. Guaranteed.
[0] https://news.ycombinator.com/item?id=35359212
[1] https://news.ycombinator.com/item?id=35325709
It was worse before MS took over. But their status page displayed live metrics, it was more transparent, so people were less frustrated. As opposed to now, where the status dashboard is updated by a human.
IMHO, the issue is not the unstability. I'm really concerned that so much of our software infrastructure relies on one single service owned by Microsoft.
IMHO, the issue is not the unstability. I'm really concerned that so much of our software infrastructure relies on one single service owned by Microsoft.
I remember it very differently. I remember remarkably good reliability, to the point that HN would regularly laud GitHub for “doing it right” by investing heavily in infrastructure and reliability and being “boring”. I remember some GitHub leader or founder affirming that sentiment, that they’ve always felt their value proposition depends greatly on reliability. I remember it didn’t change as soon as MS took over, but a year or so later, and that for the first few years of diminishing reliability people would always point fingers at the MS takeover.
There are many github alternatives.
Are you saying you'd prefer your ci/cd pipeline to be scattered across services, or that you'd like a github backup if it goes down? You could probably mirror your code on gitlab and host a Jenkins server on aws instead of relying a single point of failure. Personally I'll take the single point of failure downtime in exchange for lower maintenence
There are many github alternatives, sure. But sofware is still centralized on github.
I double dare you to "pip install", "cargo build", or "go get" without hitting github.
Your project can be hosted on repo.or.cz, gitlab or codeberg, you will still need github to build it locally, or to build it on your self hosted CI.
The solution is P2P git hosting. Dare I say magnet-links git repo urls?
I double dare you to "pip install", "cargo build", or "go get" without hitting github.
Your project can be hosted on repo.or.cz, gitlab or codeberg, you will still need github to build it locally, or to build it on your self hosted CI.
The solution is P2P git hosting. Dare I say magnet-links git repo urls?
Ok this makes sense, thanks for the clarification.
You seem to have had a different pre ms github experience than I. Github didn't really embrace the Suck til it foisted Actions on the world. Gitlab had a lot of downtime for a few years (shockingly mostly to do with CI related mechs).
Pull request eventually succeeded. Just "retry until the load balancer gives you a good node"
Well behaved services will enter exponential backoff when seeing errors, but humans will just keep spamming F5 as much as they can :)
Earlier in the day GitHub actions went down as soon as I had pushed some changes and wanted a deployment to run. Talk about bad timing.
It's a bit strange seeing GitHub having problems again in such short time. But to be fair, GitHub's scale is insanely huge. I'm surprised they can keep everything running fine 99% of the time.
It's a bit strange seeing GitHub having problems again in such short time. But to be fair, GitHub's scale is insanely huge. I'm surprised they can keep everything running fine 99% of the time.
Use git-bug. It never goes down, is about as private as possible, and the issue tracker UI is actually really pleasant.
https://github.com/MichaelMure/git-bug
Or https://fossil-scm.org/home/doc/trunk/www/index.wiki if you don't mind learning some new commands.
Most professional software engineers need to communicate with managers, PMs, support, sales, etc - and "download this obscure software and learn how to use git" is not an acceptable answer for how to access the issue tracker.
You're right, it's definitely not for corpo-clone engineers and more for those of us that like to be in control.
I guess only us "corpo-clone engineers" care about users being able to report bugs without learning how to use the command line and git.
There's a webui which is quite good at reporting bugs, even when GitHub is down. I guess you didn't even look at the software before commenting.
I did look at the software - the web UI requires running it locally, they specifically call out hosting it publicly being impossible.
We picked Bitbucket instead of Github many years ago, basically because the person setting up the repos already knew Jira
I’ve been very happy about that past decision lately, Bitbucket is just…boring. It works, rarely gets in the way, and the company doesn’t really partake in politics in the tech scene
I’ve been very happy about that past decision lately, Bitbucket is just…boring. It works, rarely gets in the way, and the company doesn’t really partake in politics in the tech scene
God I hate bitbucket.
I can go into details about why if you want, but it's not relevant here and it's a bit off the topic in my opinion.
So, just commenting so that; so then you're aware that bitbucket love is super far from universal.
I can go into details about why if you want, but it's not relevant here and it's a bit off the topic in my opinion.
So, just commenting so that; so then you're aware that bitbucket love is super far from universal.
I didn't really mind it either way, but I knew someone who loved the integration of Confluence, Bitbucket, HipChat etc. I guess it's what you know and what features you want.
the useful integrations are even better in gitlab from my perspective.
Gitlab, however, has priced itself out of being competitive in my opinion
Gitlab, however, has priced itself out of being competitive in my opinion
Yes, that is an issue. But I wonder if we need to get slightly more comfortable with paying for foundational tools. Docker is another one I'm happy to pay for, and I wish they'd got people paying 5 years ago.
The Bitbucket I've used have all been self-hosted and it's pretty good. There might be some cases where it doesn't have the features you want, but for 99% of all usage it's absolutely fine. The Jira integration is really neat and something I've missed in other setups.
Your right it is kind of boring, and that's really all I want. GitHub is okay as well, but it's also pretty weird how it tries to be a social network to the point where I can't have the first page I see be my own repos.
Your right it is kind of boring, and that's really all I want. GitHub is okay as well, but it's also pretty weird how it tries to be a social network to the point where I can't have the first page I see be my own repos.
This is true, until bitbucket starts to go down.. The fact that it's better is just circumstantial.
I've missed this, I guess. What Github politics?
Its distant memories now, but there was a time that they did a lot to make left leaning ideology look good, likely setting back the movement.
For example they mentioned publicly that “its hard to even interview a white male” and that “white women are the biggest barriers to [progressive policies] in tech”.
There was also the whole kerfuffle with badgering repo maintainers to remove words such as blacklist, and (very minorly) the push for main instead of master (and the understandable frustration to everyone not bothering with identity politics).
Honestly, I was following it at the time and I was greatly irritated at the time, now I’m just tired of it all and refuse to engage with the topic in any way. I can understand why others feel the same.
Github has not done anything irritating in quite some time though, so its moot in my opinion.
For example they mentioned publicly that “its hard to even interview a white male” and that “white women are the biggest barriers to [progressive policies] in tech”.
There was also the whole kerfuffle with badgering repo maintainers to remove words such as blacklist, and (very minorly) the push for main instead of master (and the understandable frustration to everyone not bothering with identity politics).
Honestly, I was following it at the time and I was greatly irritated at the time, now I’m just tired of it all and refuse to engage with the topic in any way. I can understand why others feel the same.
Github has not done anything irritating in quite some time though, so its moot in my opinion.
I think I've seen main instead of master, but I don't even recall that being a political issue. Although, I'm biased by my empiricism which means I generally only keep track of actual policy rather than "culture war" issues.
the most neutral source I could find: https://www.zdnet.com/article/github-to-replace-master-with-...
Our experience with BitBucket in the Netherlands has been that it's slow, almost as bad as Jira. I also found the ways things like the PR interface and Markdown support etc to be lackluster with third party support for things like semantic-release lacking and painful. We haven't been affected by Github outages in the last 5 or so years that we've switched to that, so I'm happy we did.
TL;DR: Perhaps the levels of service vary per region?
TL;DR: Perhaps the levels of service vary per region?
As someone who used Jira and BitBucket quite a bit: The mistake was using the cloud version.
It's really a shame that Atlassian have price almost everyone out of their self-hosted solutions, because those where always much faster. I'm sure that Atlassian understands the business case better than I do, but I just don't see the point of their offerings when I can't afford on-prem. The main selling point for us was always: "We do not want this data on any servers we do not control, and Atlassian products allows us to do that". Now you need to be able to afford a data center licens, and I can't imagine those sell to well, other that to existing customers who now have no choice.
It's really a shame that Atlassian have price almost everyone out of their self-hosted solutions, because those where always much faster. I'm sure that Atlassian understands the business case better than I do, but I just don't see the point of their offerings when I can't afford on-prem. The main selling point for us was always: "We do not want this data on any servers we do not control, and Atlassian products allows us to do that". Now you need to be able to afford a data center licens, and I can't imagine those sell to well, other that to existing customers who now have no choice.
Based on their status page it seems to affect only issues and pull requests … for now.
The main page throws a 500. Status page must be wrong/delayed
I cannot see my notifications nor deploy an app on heroku.
For me even the repository view doesn't work.
My (and all) profile homepages give a 500 server error, but I can access all repos themselves directly if I already know the URL. A bit odd.
I can't access any of my repos in the web UI by direct URL.
Noticed as well. It is up again now (Netherlands).
Edit: Oops. Not up. Repo's aren't shown (error 500) sometimes. Refresh helps (maybe due to choice of load balancer as other comments refer to).
Edit: Oops. Not up. Repo's aren't shown (error 500) sometimes. Refresh helps (maybe due to choice of load balancer as other comments refer to).
How much of GitHub stack is still Ruby? Is it an appropriate language at this scale?
I don't think it really comes down to language choice.
Now the incident page states that the Git Operations are degraded as well.
Repo view just came back in this moment. Let's see how that evolves.
Man, they ought to put CoPilot in charge of their ops
All green lights now. :)
gh cli interface still works fine
I remember seeing tweets that suggested that a lot of the people from the Azure DevOps organisation have been moved to GitHub, whilst a lot of former GitHub employees have been sacked or left and that things like GitHub Actions have been migrated to .NET. Is there any truth to this?
Also is GitHub being slowly pushed to use Azure as the underlying infrastructure?
A lot of people say that outages have increased since the takeover and others say they have become less. I don't know but it certainly feels like a lot recently.