GitHub Is Down Again(githubstatus.com)
githubstatus.com
GitHub Is Down Again
https://www.githubstatus.com/incidents/zhtplv7zd052
70 comments
How do you prevent this from impacting your systems? What kind of features can GitHub invest in to make it easy for you to insulate from downtime?
Don't be afraid to use another code forge, there are many alternatives. People will come over if your project is interesting.
[deleted]
Good thing I spent time refactoring my Github workflow files to put all of the meaningful logic into separate shell scripts, so that I can still deploy without relying on Github Actions to be available.
After a few outages and bugs from multi AAA cloud providers, I do the same. Rely most on bash scripts that are portable and only use the cloud platform for offloading builds and secrets management. Every built in feature always comes to bite you back in the ass when least expected.
#HugOps to GitHub Engineers! Such things are usually stressful, especially before the weekend. In addition, in some European countries today is a bridge day (day between a holiday and the weekend, many people take off on such a day).
In the US this coming Monday is Memorial Day. Memorial Day weekend is a very popular travel time and many people will be taking today off to extend those trips an additional day. Hopefully no GitHub engineers will be forced to sacrifice such plans.
I would imagine this is mostly the domain of on-call engineers/SREs (wasn't this historically a problem that was fixed by a reboot?) and you have to assume that you're going to be called at 4AM on a Saturday morning. So even booking vacation when you're on shift is risky, and unfortunately that means someone has to work public holidays. Good companies will give you a day off in lieu.
Having done on-call work, the most annoying bugs are the ones that you feel could be automated (especially when the pager goes off when you're asleep). Not to say that I hope these guys get called out, but it's the nature of the job.
Having done on-call work, the most annoying bugs are the ones that you feel could be automated (especially when the pager goes off when you're asleep). Not to say that I hope these guys get called out, but it's the nature of the job.
I have managed many on-call people during my career. I hated whenever one of my guys actually had to answer a call. It meant we had failed somewhere and needed to get better. I certainly didn't accept it as just the nature of the job.
>It meant we had failed somewhere and needed to get better.
That is the work-ethic i would love to see from all manager.
Excellent attitude to life/work.
That is the work-ethic i would love to see from all manager.
Excellent attitude to life/work.
No problem.
It's almost Friday's 5 PM in Japan right now.
It's time to stop working and go out for some beer.
It's time to stop working and go out for some beer.
what is it like programming in japan? do language look like japanese ?
Here are some additional fun facts. Fair warning - I'm neither making fun nor any form of derogatory remarks but rather the observations I learned working with both sides of the protagonist in my story below.
I'm usually the guy who will translate the English spoken by Indians to the Japanese and vice versa. We, Indians, have our Ing/Hin-lish, and they have the Japlish.
I once led the Indian team and interacted with the Japanese team at Dentsu to launch a new eye-care product for Johnson & Johnson. The content was, of course, Japanese, and we do have translators on our end but do not speak Japanese. So, English and here is a dramatized reproduction from my point-of-view.
Japanese team, "Now, we do propaganda to make people buy the products."
Me, in super-slow, enunciating beyond what my mouth can stretch, "No, no, we won't spread propaganda. Yes, we will do Marketing to help launch and make people know."
Me to my team waiting to stitch the Japlish, "Yes, Confirmed. Marketing team involving."
Me to the super nice Japanese team, "All Good. We, in India, Do everything."
Oh! Btw, commenting in Japanese with English terms thrown in is very common in programming code.
I'm usually the guy who will translate the English spoken by Indians to the Japanese and vice versa. We, Indians, have our Ing/Hin-lish, and they have the Japlish.
I once led the Indian team and interacted with the Japanese team at Dentsu to launch a new eye-care product for Johnson & Johnson. The content was, of course, Japanese, and we do have translators on our end but do not speak Japanese. So, English and here is a dramatized reproduction from my point-of-view.
Japanese team, "Now, we do propaganda to make people buy the products."
Me, in super-slow, enunciating beyond what my mouth can stretch, "No, no, we won't spread propaganda. Yes, we will do Marketing to help launch and make people know."
Me to my team waiting to stitch the Japlish, "Yes, Confirmed. Marketing team involving."
Me to the super nice Japanese team, "All Good. We, in India, Do everything."
Oh! Btw, commenting in Japanese with English terms thrown in is very common in programming code.
Even though some languages allow to use UTF-8 characters in the source code, we don't use it other than commenting.
But sometimes, it's really wired.
If you work in a team with members who don't care about naming, you see some wild variables like this (a little bit exaggerated);
const CONSUTANTO_1 = 100;
if (github.isNotWork()) salaryMan.drinkBiilu()
> biilu
That’s really funny
That’s really funny
A lot of katakana is. Katakana words are words or names that are borrowed from other languages typically, and written quasi-phonetically in Katakana, one of three writing systems (the other two being Hiragana and Kanji).
Since the individual sounds in Japanese are often consonant-vowel (with a few exceptions, like "n", vowel-only sounds, or the silent "u" in "su"), then it's often the case that foreign words that end in a consonant (like "bill") will have an attached vowel at the end - from my limited experience with Japanese, it's often "u" or "o". "L" in Japanese doesn't exist and is typically substituted for a rolled "r" (less roll, more single 'flick').
Many of the Katakana words are English-derived and often sound as though someone is using a fake Japanese accent to speak English. Learning a lot of katakana definitely earned a few laughs.
"Pinku" (ピンク, phonetically "pi-n-ku", pronounced more like "PEE-n-koo") meaning "pink", etc. always got chuckles out of me.
I'm just a beginner though, I'm sure someone else has more [accurate] information :)
Since the individual sounds in Japanese are often consonant-vowel (with a few exceptions, like "n", vowel-only sounds, or the silent "u" in "su"), then it's often the case that foreign words that end in a consonant (like "bill") will have an attached vowel at the end - from my limited experience with Japanese, it's often "u" or "o". "L" in Japanese doesn't exist and is typically substituted for a rolled "r" (less roll, more single 'flick').
Many of the Katakana words are English-derived and often sound as though someone is using a fake Japanese accent to speak English. Learning a lot of katakana definitely earned a few laughs.
"Pinku" (ピンク, phonetically "pi-n-ku", pronounced more like "PEE-n-koo") meaning "pink", etc. always got chuckles out of me.
I'm just a beginner though, I'm sure someone else has more [accurate] information :)
Honestly this is so cringe I can’t tell if you’re being sarcastic
Valid question IMHO. For instance in German software companies it seems to be fairly common to write code comments and documentation in English (even mostly communicate in English so that employees from other countries don't feel lost), while when I spent a few weeks working in China I noticed that comments and documentation are written in Chinese and communication in English isn't easy (spoken communication, written works quite well). Now I do indeed wonder if this is common for other Far East countries like South Korea and Japan too.
In a lot of the modern startups that's definitely the case, as a contractor I've come across some questionable "Denglisch" codebases a few times though.
I feel it's a valid question. When programming in Australia you have to either learn to read upside down or stand on your head otherwise all the electrons fall out!
That is why you always put down a foil-mat underneath laptop !
You'd be surprised what people around the world do : https://en.wikipedia.org/wiki/Non-English-based_programming_...
I don't think any of those are used for anything serious though (at least I hope not)
I don't think any of those are used for anything serious though (at least I hope not)
1С (odin-es) as language is very popular in Russia. Quite a lot of companies uses 1С (as company) products, most of which are ERP systems. Their english branding seems to be Yellow ERP yellow-erp.com
Market share of 1С in Russia was about 40%, it may grow soon due to foreign vendors leaving russian market.
Market share of 1С in Russia was about 40%, it may grow soon due to foreign vendors leaving russian market.
It's "cringey" not "cringe". You need an adjective. And I'm amazed at how many people on HN are oblivious to obvious sarcasm.
Thanks for the correction :) Sometimes sarcasm is difficult to pick up when it appears without context you’re near the center of it, if I must make up an excuse.
Damn, I'm starting now
Very jealous when is Japan opening its borders to tourists?!
is Github ops team mainly in Japan? I would think it's heavily US based
do you think it's worth it to go there to work as a SDE?
do you think it's worth it to go there to work as a SDE?
I'm not sure I'd want to go there still, I was unfortunately born into a class of people that are "hard to even interview"[0].
I'm sure if you fit into one of the many preferred classes there is much to learn there, and a fair amount of prestige in working with high scale systems.
(relavant topic from 6 years ago) https://news.ycombinator.com/item?id=11049067
[0] https://www.businessinsider.com/github-the-full-inside-story...
EDIT: I'm not trolling, I have legitimately not entertained the idea of interviewing here because of this, which is probably preferred for them, but it's worth pointing out to others who may not be informed.
I'm sure if you fit into one of the many preferred classes there is much to learn there, and a fair amount of prestige in working with high scale systems.
(relavant topic from 6 years ago) https://news.ycombinator.com/item?id=11049067
[0] https://www.businessinsider.com/github-the-full-inside-story...
EDIT: I'm not trolling, I have legitimately not entertained the idea of interviewing here because of this, which is probably preferred for them, but it's worth pointing out to others who may not be informed.
[deleted]
Does anyone have long-term stats for GitHub status incidents over the last few years or so?
They used to have graphs showing incident frequency right on the main status page, and a detailed uptime page. Guess those got retired after things got real embarrassing.
https://web.archive.org/web/20200101000934/https://www.githu...
https://web.archive.org/web/20200101000934/https://www.githu...
That's it! I had a feeling I'd seen it before.
do you think this is incorrect https://www.githubstatus.com/history?
Not at all. I'm looking for something that shows incident frequency over time, without scrolling through a bunch of pages.
Seems to be up again - but there are still some performance issues. The main page was down for approximately 15 minutes
And that is why you host your own mirror of all your repositories. I have Gitea mirror hosted on Raspberry PI connected to 7200RPM HDD at my father's house. I also host other cool apps like paperless, jellyfish, etc,. It's awesome.
Self hosting is cheap people, you just need to invest some time in it.
Self hosting is cheap people, you just need to invest some time in it.
Where time might be the more expensive item.
I don't think I've ever seen a GitHub outage that has actually impacted my ability to deliver software that week. I rarely need to do something time critical with a window of only a few hours and can't shuffle what I'm doing around.
[deleted]
[deleted]
Honestly anyone can have issue, that's not a point. The point is the amplitude of that issue, or why do you FLOSS devs want GH instead of develop more modern Email clients to effectively develop over mailing-lists and why not usenet? We have some very modern from 50+ years ago, perhaps it's time to adjust their aesthetic and rediscover their power?
I don't think mailing lists can do CI/CD.
Why not? SMTP is nothing if not a robust queuing system and surely procmail or your MDA of choice should be able to rig something up.
I've learnt enough to hate modern DevOps, where the Ops is not a contraction of "operation" but a literal "ops!" where a casual dev mistakes have created a devastating cascading effect that no one know what to do.
Oh, BTW I've switched time ago from procmail to MailDrop, it's more flexible and more maintained :D
Oh, BTW I've switched time ago from procmail to MailDrop, it's more flexible and more maintained :D
Current mailing list software can't. Of course you could control a CI/CD system with a Nokia 3310 if you wanted, but why spend the effort when CI/CD systems already have all the right features? Unless OP is actually advocating for self-hosting.
> Of course you could control a CI/CD system with a Nokia 3310
I have a feeling that such a system actually exists somewhere in this vast world, and probably even operates to this day...
I have a feeling that such a system actually exists somewhere in this vast world, and probably even operates to this day...
They absolutely can. With a simple shell script and a bit of configuration you can set up a box that will check some email and act after detecting some kind of commands directed at it. Securing that sounds like a lot of fun that I wouldn't want to experience.
That being said, I'm 90% sure the GP is sarcastic, trying pull reductio ad absurdum on people who will almost assuredly come here commenting that "the community" should "abandon centralized services" in favor of this or that pet-project.
That being said, I'm 90% sure the GP is sarcastic, trying pull reductio ad absurdum on people who will almost assuredly come here commenting that "the community" should "abandon centralized services" in favor of this or that pet-project.
> With a simple shell script and a bit of configuration you can set up a box that will check some email and act after detecting some kind of commands directed at it.
Yes but I don't see how using a mailing list would improve anything. You'd just end up re-writing all GitLab features on top of a different database and interface, and the scaling issues would remain. This seems like more of a centralisation-vs-self-hosting issue.
> That being said, I'm 90% sure the GP is sarcastic, trying pull reductio ad absurdum on people who will almost assuredly come here commenting that "the community" should "abandon centralized services" in favor of this or that pet-project.
Makes sense but I didn't interpret it that way.
Yes but I don't see how using a mailing list would improve anything. You'd just end up re-writing all GitLab features on top of a different database and interface, and the scaling issues would remain. This seems like more of a centralisation-vs-self-hosting issue.
> That being said, I'm 90% sure the GP is sarcastic, trying pull reductio ad absurdum on people who will almost assuredly come here commenting that "the community" should "abandon centralized services" in favor of this or that pet-project.
Makes sense but I didn't interpret it that way.
A ML is personal. You can move it from a hoster, potentially yourself included, to another. You can't move GitHub because it's not an implementation of something standard but a proprietary service...
That's the issue. You ML can be down, other 99% of FLOSS ML projects will not even notice the downtime, if all are on a service and the service goes down all are affected. Your ML is yours, a political issue between two government will not block you, GH can be blocked and actually was blocked and is still for various countries and reasons NOT directly related to individual projects on it. And so on.
Peoples these days tend to think that mega-IT-giants will always be up and running, so being on their shoulder is a guarantee of stability, actually is the contrary is a threat.
That's the issue. You ML can be down, other 99% of FLOSS ML projects will not even notice the downtime, if all are on a service and the service goes down all are affected. Your ML is yours, a political issue between two government will not block you, GH can be blocked and actually was blocked and is still for various countries and reasons NOT directly related to individual projects on it. And so on.
Peoples these days tend to think that mega-IT-giants will always be up and running, so being on their shoulder is a guarantee of stability, actually is the contrary is a threat.
Oh boy, I was wrong. You were serious? I mean, I myself am often telling other programmers to study history of the field, pointing out that a lot of "ground-breaking" ideas were invented in the '70s, but trying to bring those ideas as realized back then to today's world is not going to work. Both USENET and self-hosted mailing lists died for a reason: it's too cheap to spam them and with increasing number of users, the probability of someone wanting to fuck them up approaches 1.
Self-hosting is a good thing, but you can get halfway there by simply having a backup hosting from an unrelated entity. Actually, if you look at GNU Emacs development that I'm familiar with, it's conducted over a mailing list and a self-hosted git repo, but that repo is mirrored and backed up on GitHub...
Self-hosting is a good thing, but you can get halfway there by simply having a backup hosting from an unrelated entity. Actually, if you look at GNU Emacs development that I'm familiar with, it's conducted over a mailing list and a self-hosted git repo, but that repo is mirrored and backed up on GitHub...
That's an argument for self-hosting, not for mailing lists. What you're describing is GitLab.
ML can be both self-hosted or hosted by third party, being a standard thing you are not tied to any specific player. Your "public upstream repo" is on your domain, no matter who bound it right now.
It took me days to set up my team’s CI/CD pipeline with GitHub’s existing features. If I started from a simple mailing list, it would take months of full-time work to add automated build steps with dependencies between steps, caching, secure secret management, proper authentication, a practical issue database, and link that to my git repo.
At that point, I would have a completely new product that takes a full-time maintainer and has the same complexity as the big players. It would be more efficient to fork GitLab. I think you don’t realize how wasteful that is.
And what’s the point? There’s already a well-maintained FOSS self-hosted solution that benefits from not being based on antiquated technologies.
At that point, I would have a completely new product that takes a full-time maintainer and has the same complexity as the big players. It would be more efficient to fork GitLab. I think you don’t realize how wasteful that is.
And what’s the point? There’s already a well-maintained FOSS self-hosted solution that benefits from not being based on antiquated technologies.
At that point you might have discovered not needing at all 99% of DevOps concepts, and that probably they exists because they help some making profits not because are good ideas.
In that terms, it's OT and very long to discuss here, my opinion is that's about time to ERASE 30+ years of IT crappy history to start back at pioneering time Xerox PARC for instance where ALL we use today was invented and implemented far better than now, where their antiquated technologies are actually far more advanced than most we use everyday calling them new and advanced.
An OS must came back to a unique fully-integrated environment, with a single programming language, no matter if Lisp, Smalltalk or something else, applications simply do not exists, are just built-in functionality we develop inside the system, deploy is just pushing code to available instances etc. Such VERY ancient model is actually far more advanced than modern one, demand far less efforts and resources giving back far more. It's not there because at a certain point in time IBM decide that compartmentalization and crap made profit and modern IT giants understand that very well going far beyond.
In that terms, it's OT and very long to discuss here, my opinion is that's about time to ERASE 30+ years of IT crappy history to start back at pioneering time Xerox PARC for instance where ALL we use today was invented and implemented far better than now, where their antiquated technologies are actually far more advanced than most we use everyday calling them new and advanced.
An OS must came back to a unique fully-integrated environment, with a single programming language, no matter if Lisp, Smalltalk or something else, applications simply do not exists, are just built-in functionality we develop inside the system, deploy is just pushing code to available instances etc. Such VERY ancient model is actually far more advanced than modern one, demand far less efforts and resources giving back far more. It's not there because at a certain point in time IBM decide that compartmentalization and crap made profit and modern IT giants understand that very well going far beyond.
This is why centralized services are a bad idea.
Resolved after 1 hour and 21 minutes, that's quick.
When I go to the link the page keeps refreshing, and I get duplicate messages at the top of the page that says "Hang on there partner, You're subscribing too fast (give it a quick second and try again)."
There is also a native browser notification that says "Oops, something went wrong while subscribing you! Let's reload the page and try again."
Is anyone else getting that? I haven't "subscribed" to any thing.
I'm using the latest version of Firefox with the uBlock Origin addon.
There is also a native browser notification that says "Oops, something went wrong while subscribing you! Let's reload the page and try again."
Is anyone else getting that? I haven't "subscribed" to any thing.
I'm using the latest version of Firefox with the uBlock Origin addon.
There is zero new knowledge is these threads. I understand the HN crowd relies heavily on GitHub but it feels like Twitter. Anyway, I'm not adding anything too so...
Self host gitlab
Duplicate your efforts and do not rely on a single git hosting provider! Gitlab.org is there. Use it. There are multiple ways to sync workflows and git repositories