Gitlab.com was experiencing elevated error rates for Git, Web, and API(status.gitlab.com)
status.gitlab.com
Gitlab.com was experiencing elevated error rates for Git, Web, and API
https://status.gitlab.com/
93 comments
s/sensible/sensitive/
Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
> s/sensible/sensitive/
Thanks :)
> Would this be a place where a plan hint would help?
It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons.
Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.
Thanks :)
> Would this be a place where a plan hint would help?
It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons.
Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.
Gotcha, that makes sense to me - refactoring queries which wanna flip would be my choice as well, just thinking that if the analyze fixes plans its (in my experience) "well we could have just used this plan anyway" - both of them are short term fixes for sure.
/r
took me a lot of effort to convince my CTO to move from Bitbucket to Gitlab.
Since we have started using it ( 3 + months ), we have experienced more than 4 instances of downtime. If some one from Gitlab is reading this message, please consider a code freeze and spend some time on stabilizing the system.
time to add a new backup remote to all our repos.
time to add a new backup remote to all our repos.
Run a private instance? We haven't had trouble with that approach.
Agreed. I use gitlab for administering all of my courses. I started when they had more features compared to github (free private repos, built in test runners), but over time github has regained some but not all of that ground. I will still be using gitlab in the future, but am going to convince our systems admin to set up a local instance.
We are using bitbucket atm and I can't fathom a reason to move, care to elaborate?
I use GitHub and Bitbucket fairly extensively and I find Bitbucket painfully slow at times (eg. 7.38s to load a page just now with a whopping two open pull requests on it, whilst somehow transferring 8.8MB of crap; the equivalent page on GitHub is about 6 times better on both fronts) and the UI only ever seems to get more clunky. I fear the tight integration with JIRA is slowly infecting it with JIRA-esque problems.
Atlassian is Australian company.
Australia has very weird and wide reaching laws regarding software backdoors.
Australia has very weird and wide reaching laws regarding software backdoors.
Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request?
Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian.
Still curious about what sort of backdoor laws they have.
Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian.
Still curious about what sort of backdoor laws they have.
I think it's in reference to this from a few years ago: https://www.wired.com/story/australia-encryption-law-global-...
EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...
EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...
mainly for DevOps related features.
https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ ( marketing page, but mostly valid points )
https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ ( marketing page, but mostly valid points )
Most of those are either supported by other Atlassian products or not important at all. Why would I care about a web ide?
and better code review experience as compared to Gitlab
I love Gitlab because I love open source and competition but Gitlab either on-prem or gitlab.com has been having growing pains ever since I started using it around 2015.
And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing.
Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it.
They will sort this out eventually.
And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing.
Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it.
They will sort this out eventually.
I have nothing but praise for our enterprise Gitlab implementation. It hosts 2000+ projects and for many teams is mission critical. Our team that maintains it is wonderful and I've only seen it down without planning once for about an hour. Using Gitlab at work actually swayed me to move to it for my personal projects as well.
+1 for on-prem. I tried gitlab.com first but it was very slow (this was years ago, YMMV) but on-prem has been rock-solid and fast for us hosting ~100 projects. One thing I've been surprised by is how easy updates have been, I don't think we've ever had to even reconfigure anything, including runners. I know it seems like that is how it should work, but I've had a lot of opposite experiences with the open source tier of tools.
My company does on-prem gitlab as well, and it's pretty fantastic. We're in the beginning stages of migrating off of Azure DevOps but so far we're wowed by the features.
Hi, GitLab team member here. Heads-up: GitLab.com appears to be stable again. Our engineers are monitoring and investigating the root cause of the incident.
https://twitter.com/gitlabstatus/status/1371443832865222658
https://twitter.com/gitlabstatus/status/1371443832865222658
Out of curiosity, why does your "about" page vouch for a suspended Twitter account? Is @GitLab -> @GitLabs a typo?
https://about.gitlab.com/handbook/marketing/community-relati...
https://about.gitlab.com/handbook/marketing/community-relati...
<td><a href="https://twitter.com/GitLabs">@GitLab</a></td>Thanks! It is a typo and I submitted a fix https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
Thanks Sid, merged :) We've also fixed the formatting at https://about.gitlab.com/handbook/marketing/community-relati... in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
Cheers for that.
I merely wanted to verify the @ was authentic, and that page was the top "site:gitlab.com" result.
I merely wanted to verify the @ was authentic, and that page was the top "site:gitlab.com" result.
Thanks for raising it.
Thanks for flagging, that's a typo. Here's an MR to fix: https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
The page has formatting problems, I'll rework this in a separate MR.
The page has formatting problems, I'll rework this in a separate MR.
Gitlab is a great service and a great alternative to github and bitbucket. But it has been quite flaky over many years and it's quite surprising they haven't prioritised system stability over everything else. https://twitter.com/gitlabstatus
Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.
Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.
My guess is it has to do with their prioritization of only hiring people with "significant ruby/rails" experience. Kills a lot of the talent pool, which doesn't make sense as easy is it is is to pick up a new high level language and framework.
Fully remotes has nothing to do with it.
My company is full remote and we don't have these issue.
My company is full remote and we don't have these issue.
Agreed. Blaming a remote work culture is a complete non sequitur.
We switched from self-hosted to gitlab.com a couple months back and the migration has been extremely disappointing.
Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive.
Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).
Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive.
Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).
Any chance you could drop me some details about pushes hanging? I'm an engineer in the source code team doing a lot of performance work and I might be able to correlate it with known issues or investigate it further. I know it's a bit awkward to get data after-the-fact but just some links to the problem repos would be useful. I might not be very useful if it's related to network/hardware issues but I do work in that aspect of the software at least.
You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at [email protected] if not :)
You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at [email protected] if not :)
We don't use Gitlab CI, but only their hosted git. Just thought I'd throw in our sterling experience over the past three years. We've really not noticed whatever downtime they've had.
What prompted you to move away from self-hosted? What was the maintenance overhead like?
I can't even get to the issue where they're tracking this any more. At least the status page is on different infra though, hopefully they'll keep it updated.
GitLab employee here.
That is correct - we also will be posting updates here: https://twitter.com/gitlabstatus/.
That is correct - we also will be posting updates here: https://twitter.com/gitlabstatus/.
[deleted]
I can't be the only one that feels that GitLab has extended downtime at least weekly, sometimes bi-weekly.
Not downtime, but some kind of issue, yes [1]. However github is not much better [2] and gives much less detail and fewer updates. Bitbucket is more solid recently [3] (but wasn't really better than the others in 2020)
1: https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
2: https://www.githubstatus.com/history (don't get fooled by them collapsing the incident list after three per month)
2: https://bitbucket.status.atlassian.com/history (also collapses after three per month)
1: https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
2: https://www.githubstatus.com/history (don't get fooled by them collapsing the incident list after three per month)
2: https://bitbucket.status.atlassian.com/history (also collapses after three per month)
I use gitlab daily and have not experienced any outage in last 6 months probably.
It is usually their shared runners. I experience outages with them a couple times a month. It isn't a big deal since we maintain our own private pool, so I just move jobs to them. (They make you pay for CI minutes so we use them for certain jobs.)
I use GitLab for my side project (source control only; don't have any kind of CI set up just at the moment) and it's been years since there was any really noticeable slowness for me. I can't say I use it daily, but fairly frequently.
Same - I've had zero issues.
Our self-hosted GitLab instance is running fine!
Yup; I just updated ours to 13.9.4 and it was seamless as always (EE version).
Pitty its licensing cost will go up ten fold next year.
Github seems to be doing worse
Yes. They (GitHub) have been doing a lot worse. Last time I checked, they where completely down 2 days ago. [0] Before that, GitHub Actions was down 14 days ago: [1].
At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear.
Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
[0] https://news.ycombinator.com/item?id=26439075
[1] https://news.ycombinator.com/item?id=26301659
[2] https://news.ycombinator.com/item?id=26439213
At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear.
Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
[0] https://news.ycombinator.com/item?id=26439075
[1] https://news.ycombinator.com/item?id=26301659
[2] https://news.ycombinator.com/item?id=26439213
> Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover...
Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI.
I suppose this is why people just end up self hosting gitlab instead.
I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover...
Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI.
I suppose this is why people just end up self hosting gitlab instead.
If you want transparent failover you need to use your own domain for the repo URL. In case of a failure you (or some automatic job) would have to change the DNS record.
Gitlab has a repo mirroring feature [1]. But of course you'd also need to sync users public keys.
Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
You can instead set up a read-only mirror so people can at least still pull and browse the code.. Gitea [2] might be a better choice than Gitlab, since it's much more lightweight and easier to host.
[1] https://docs.gitlab.com/ce/user/project/repository/repositor...
[2] https://gitea.io
Gitlab has a repo mirroring feature [1]. But of course you'd also need to sync users public keys.
Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
You can instead set up a read-only mirror so people can at least still pull and browse the code.. Gitea [2] might be a better choice than Gitlab, since it's much more lightweight and easier to host.
[1] https://docs.gitlab.com/ce/user/project/repository/repositor...
[2] https://gitea.io
Thanks this is useful to know
> Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
That's essentially the same conclusion I keep coming too, occasionally it has hit me when I go to push something but rarely has it blocked me or anyone else from continuing to work.
> You can instead set up a read-only mirror so people can at least still pull and browse the code
Yeah, this I need to do eventually just for peace of mind as a more automated backup solution. At least I don't have to care about failover.
> Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
That's essentially the same conclusion I keep coming too, occasionally it has hit me when I go to push something but rarely has it blocked me or anyone else from continuing to work.
> You can instead set up a read-only mirror so people can at least still pull and browse the code
Yeah, this I need to do eventually just for peace of mind as a more automated backup solution. At least I don't have to care about failover.
Maybe some kind of pre/post-push hook to automatically push to a secondary repo? Not entirely sure if this is possible, I’ve never written a Git hook before.
https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
Gitlab is completely down until just now, similar happened not that long ago. Over the last three weeks continuously had issues with the Gitlab runners (even not shared ones) were it wouldn't jobs or only after a second or third retry attempt.
I think the flakiness of jobs succeeding is also related whether you use their package registry. I can't say I had similar issues with Github.
I think the flakiness of jobs succeeding is also related whether you use their package registry. I can't say I had similar issues with Github.
Well, they are stretching and extending gitlab with more and more functions.. With a... Not so great code base..
Is there any code base in this world that developers would not complain about? ;)
The only perfect code is the one you write, all others are crap
The one that will never be deployed on production. Real world unfortunately corrupts beautiful code with it's pesky complications.
Xmonad
Hi, what could be improved in GitLab's code base in your opinion? :)
In the past people have suggested we rewrite GitLab in Scala, Haskell, Node.js, and IIRC also regular Java.
I think it mostly comes down to "I didn't write it, so it's bad" :)
I think it mostly comes down to "I didn't write it, so it's bad" :)
Nah, i don't care which language. Like, at all. I just think that gitlab outgrow ruby.
Suprised no rust or elixir there.
[deleted]
Do you have info on how different providers stack up?
*any non self-hosted git provider
Obligatory "is bi-weekly 2 or 0.5 times a week" confusion comment ;)
Though from the context it's 2.
Though from the context it's 2.
This is why we have "fortnightly" in the English language.
TIL:
"A fortnight is a unit of time equal to 14 days (2 weeks). The word derives from the Old English term fēowertyne niht, meaning "fourteen nights".
https://en.wikipedia.org/wiki/Fortnight
"A fortnight is a unit of time equal to 14 days (2 weeks). The word derives from the Old English term fēowertyne niht, meaning "fourteen nights".
https://en.wikipedia.org/wiki/Fortnight
Or just say "twice a week" and "once every two weeks" and not leave everyone in a meeting thinking you're talking about popular video game.
ok
Like it or not, fortnightly and “once every two weeks” are your only unambiguous choices.
We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
The word fortnight is in common parlance. Four score is not. But for the record the term bi-weekly is still confusing.
This status page is weird, it says that there is an active incident and that there is a databate latency issues, but everything is green:
https://i.imgur.com/oXSiyTG.png
I was just pushing some new code to production when it stopped working. So frustrating.
whalesalad(1)
Oh gitLAB - for a moment there I misread “GitHub” and was worried. (J/k you rock guys!)
There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency).
That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic).
Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet.
At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...