Gitlab Critical Security Release: 16.3.4 and 16.2.7(about.gitlab.com)
about.gitlab.com
Gitlab Critical Security Release: 16.3.4 and 16.2.7
https://about.gitlab.com/releases/2023/09/18/security-release-gitlab-16-3-4-released/
44 comments
This happened to us using Flowdock (a Slack competitor), they had a main DB failure, and when a new one was promoted the state was bad, and so "new" employees from another company started appearing in our organisation replacing our own employees, they could then read our private messages and access our rooms.
https://web.archive.org/web/20220704223244/https://flowdock-...
It was really hard for me to find a responsible from Flowdock/Broadcom/Rally Software/CA Technologies to report the leak, but in the end they killed the servers until they got the issues under control.
Because of this I mostly share encrypted files over chat, using OpenPGP, but the user experience is terrible for most :(
https://web.archive.org/web/20220704223244/https://flowdock-...
It was really hard for me to find a responsible from Flowdock/Broadcom/Rally Software/CA Technologies to report the leak, but in the end they killed the servers until they got the issues under control.
Because of this I mostly share encrypted files over chat, using OpenPGP, but the user experience is terrible for most :(
Wow, that's really disturbing.
> Because of this I mostly share encrypted files over chat, using OpenPGP, but the user experience is terrible for most :(
Agreed. Even with builtin OpenPGP support in Thunderbird the UX is still bad for regular users (well, for technical ones too).
> Because of this I mostly share encrypted files over chat, using OpenPGP, but the user experience is terrible for most :(
Agreed. Even with builtin OpenPGP support in Thunderbird the UX is still bad for regular users (well, for technical ones too).
session tokens are a fun one, that's how Linus Tech Tips (tech Youtuber) and other Youtubers got their accounts yoinked.
Mind sharing links on what's happened?
Someone from the staff opened a pdf with malware sich stole their Session tokens and bam.
Wasn’t a PDF, straight up just regular exe. Their business unit aren’t “techies” per se like the rest of them.
In fairness, it was file extension spoofed[1] exe made to look like a PDF file. When sent to someone who frequently downloads and opens PDFs as part of their job, it would be a very easy mistake to make, especially for the unaware to this attack vector.
[1] https://www.malwarebytes.com/blog/news/2016/09/lesser-known-...
[1] https://www.malwarebytes.com/blog/news/2016/09/lesser-known-...
Yup, just clarifying that it wasn’t some crazy PDF reader exploit (which would have been kinda amazing).
> some crazy PDF reader exploit
One of those: https://6point6.co.uk/insights/abusing-pdf-files/
One of those: https://6point6.co.uk/insights/abusing-pdf-files/
[deleted]
This is the original issue thread from two months ago:
https://gitlab.com/gitlab-org/gitlab/-/issues/417594
And the root cause seems to be...
> Change your git config and include username of the your victim that is,
> git config --global user.name "victim"
Oops.
https://gitlab.com/gitlab-org/gitlab/-/issues/417594
And the root cause seems to be...
> Change your git config and include username of the your victim that is,
> git config --global user.name "victim"
Oops.
This is not the root cause but the way to exploit it imho. The root cause was that the pipeline process would reuse CI tokens that gives access to other repos/projects just by being added as member on someone else's repo and being formally marked as user in a commit.
Wow. That kind of decision does not inspire confidence in the product.
It's specifically exploiting a bad choice made in the SAST analyzer part of the project, which is kind of interesting...as you would think that the security-minded people work on that tool.
That pattern seems kind of predictable too. You have the main product, with the main team, that probably understands things like how they use tokens for CI pretty well. Then, as the product grows, you get satellite teams to work on features outside the core. Things like this SAST analyzer, GitLab pages, Gitlab Wiki, etc. They don't have the same deep understanding of the core product. So they probably do a lot of trial and error until something works.
So I guess one lesson for the main product team is that they need to expose apis and data to these peripheral product teams as if those teams were a third party...don't give them automatic access to everything.
That pattern seems kind of predictable too. You have the main product, with the main team, that probably understands things like how they use tokens for CI pretty well. Then, as the product grows, you get satellite teams to work on features outside the core. Things like this SAST analyzer, GitLab pages, Gitlab Wiki, etc. They don't have the same deep understanding of the core product. So they probably do a lot of trial and error until something works.
So I guess one lesson for the main product team is that they need to expose apis and data to these peripheral product teams as if those teams were a third party...don't give them automatic access to everything.
I’ve gotten a lot of value from GitLab but I have to say that the frequent security issues significantly undercut the messaging about their product being a security solution.
I’ve gotten the impression that they’re trying to expand rapidly to have more points where they can offer a feature which GitHub doesn’t have, which is certainly understandable but also gives the impression that their teams are overstretched and probably burning out. It’s pretty common to encounter an issue which has been open for years without progress despite a fair amount of customer interest.
I’ve gotten the impression that they’re trying to expand rapidly to have more points where they can offer a feature which GitHub doesn’t have, which is certainly understandable but also gives the impression that their teams are overstretched and probably burning out. It’s pretty common to encounter an issue which has been open for years without progress despite a fair amount of customer interest.
I think a lot of that is that they are security minded, open source, and Gitlab is loud about their vulnerabilities, as part of that ethos.
GitHub enterprises, the on-prem version of their product, has plenty of vulnerabilities [1]. There's probably just as many in their hosted product, but we don't hear about them because they're fixed quietly.
[1] https://www.cvedetails.com/vulnerability-list/vendor_id-1191...
GitHub enterprises, the on-prem version of their product, has plenty of vulnerabilities [1]. There's probably just as many in their hosted product, but we don't hear about them because they're fixed quietly.
[1] https://www.cvedetails.com/vulnerability-list/vendor_id-1191...
It’s not just that but, yes, I don’t want to over-weight it due to visibility. It just feels like they’re over-stretched, especially since non-security issues are exactly the same.
> which is kind of interesting...as you would think that the security-minded people work on that tool.
I work in security at a company with similar tools and unfortunately I think this is an incorrect assumption.
Security people aren’t necessarily good developers, and good developers aren’t necessarily good security people. And being “security minded” really isn’t good enough.
At the end of the day, the team building the product has the first and foremost priority of launching the product. This always means cutting corners, or not going as deep as might be necessary for certain issues like this, even if they are “security minded”. You really have to have a checks-and-balances on this by having a dedicated security team that has a priority of first and foremost not allowing security vulnerabilities to make it into launched products.
I’m not sure if GitLab has that or not, I’m just saying that wouldn’t make any assumptions about the security posture of a feature just because it’s a “security feature”. If anything I’d be more wary of them, because in my experience there actually can be an attitude of “my shit doesn’t stink” among people developing security products.
I work in security at a company with similar tools and unfortunately I think this is an incorrect assumption.
Security people aren’t necessarily good developers, and good developers aren’t necessarily good security people. And being “security minded” really isn’t good enough.
At the end of the day, the team building the product has the first and foremost priority of launching the product. This always means cutting corners, or not going as deep as might be necessary for certain issues like this, even if they are “security minded”. You really have to have a checks-and-balances on this by having a dedicated security team that has a priority of first and foremost not allowing security vulnerabilities to make it into launched products.
I’m not sure if GitLab has that or not, I’m just saying that wouldn’t make any assumptions about the security posture of a feature just because it’s a “security feature”. If anything I’d be more wary of them, because in my experience there actually can be an attitude of “my shit doesn’t stink” among people developing security products.
I guess, though they work on an SAST scanner that identifies problems pretty similar to this one, untrusted input, "confused deputy", etc.
I think that lesson is a great takeaway. I don't think it should have been possible to generate access levels equal to the user without their direct input (for example, on a specific "generate access token" page). Then a satellite team requesting that ability hopefully would have raised red flags with the main team, if that's how they're operating.
Would someone share some details on why SAST mentioned in this thread, for those who are not familar with the Gitlab architecture design and product decisions. Thanks.
Their product includes a static code analysis tool you can run in ci/cd pipelines, triggered by the existence of some files in your repo.
It's the entry point for this exploit. Something the attacker abuses to get rights they shouldn't have. The victim doesn't have to use it at all.
It's the entry point for this exploit. Something the attacker abuses to get rights they shouldn't have. The victim doesn't have to use it at all.
What is the vulnerability here? Being an asshole to a co-member on a project? What I'm saying is, this seems like a place of mutual trust, no apparent reason to be especially hardened.
No, it works cross project, with no specific need for the victim to do anything dumb. And the attacker needs no rights to the victim target repo. You end up with an active CI_JOB_TOKEN that impersonates the victim. It's pretty bad.
No, the issue is that a random attacker can create a group, and add the victim to the group themselves, and that gives them access to the victim's private repos, albeit through a pretty convoluted execution path.
Not sure if anyone here is hanging out from Gitlab. One request, and one question:
Request: The new UI has "Admin" link hidden which is hard to find. I liked the old UI better (explicit is better than implicit).
Question: Is the open source code the same as the Enterprise versions of the code? (how do you implement feature differences?)
Request: The new UI has "Admin" link hidden which is hard to find. I liked the old UI better (explicit is better than implicit).
Question: Is the open source code the same as the Enterprise versions of the code? (how do you implement feature differences?)
There's now a single codebase containing both enterprise and FOSS. Enterprise features live in the `ee/` subfolder of the repository. This allows them to share code. (EE used to be a (visible-source) fork of CE, but the repos got merged like a year or so ago).
GitLab team member here. Thanks for your feedback.
> Request: The new UI has "Admin" link hidden which is hard to find. I liked the old UI better (explicit is better than implicit).
This was reported in the new navigation feedback issue [0] and is being worked on in [1]. Suggest adding your comment/upvote there too.
> Question: Is the open source code the same as the Enterprise versions of the code? (how do you implement feature differences?)
The source code for GitLab is located in a single repository. The change was proposed and introduced in 2019 for efficiency reasons [2] [3] [4]
You can learn more about the stewardship of GitLab, and the promises in [5] - point 11) links to the enterprise code location, and the free open source location.
This workflow also ensures that everyone can contribute to GitLab [6] - free users, and customers [7], using the same code base.
[0] https://gitlab.com/gitlab-org/gitlab/-/issues/409005#what-we...
[1] https://gitlab.com/gitlab-org/gitlab/-/issues/415854
[2] https://about.gitlab.com/blog/2019/02/21/merging-ce-and-ee-c...
[3] https://about.gitlab.com/blog/2019/08/23/a-single-codebase-f...
[4] https://about.gitlab.com/blog/2019/10/02/contributor-after-s...
[5] https://handbook.gitlab.com/handbook/company/stewardship/#pr...
[6] https://about.gitlab.com/community/contribute/
[7] https://docs.gitlab.com/ee/development/contributing/#contrib...
> Request: The new UI has "Admin" link hidden which is hard to find. I liked the old UI better (explicit is better than implicit).
This was reported in the new navigation feedback issue [0] and is being worked on in [1]. Suggest adding your comment/upvote there too.
> Question: Is the open source code the same as the Enterprise versions of the code? (how do you implement feature differences?)
The source code for GitLab is located in a single repository. The change was proposed and introduced in 2019 for efficiency reasons [2] [3] [4]
You can learn more about the stewardship of GitLab, and the promises in [5] - point 11) links to the enterprise code location, and the free open source location.
This workflow also ensures that everyone can contribute to GitLab [6] - free users, and customers [7], using the same code base.
[0] https://gitlab.com/gitlab-org/gitlab/-/issues/409005#what-we...
[1] https://gitlab.com/gitlab-org/gitlab/-/issues/415854
[2] https://about.gitlab.com/blog/2019/02/21/merging-ce-and-ee-c...
[3] https://about.gitlab.com/blog/2019/08/23/a-single-codebase-f...
[4] https://about.gitlab.com/blog/2019/10/02/contributor-after-s...
[5] https://handbook.gitlab.com/handbook/company/stewardship/#pr...
[6] https://about.gitlab.com/community/contribute/
[7] https://docs.gitlab.com/ee/development/contributing/#contrib...
It boggles my mind that you direct customers to your issue tracker. With over 50,000 open issue, it is where requests go to die. It is not a place of honor.
Thanks for your feedback.
Requests are triaged using automation and AI, and responsible teams include the reviews in their planning. [0]
Issues (and related MRs) are targeting milestones. Since we at GitLab work in iterations and smaller changes to release features faster, epics help plan and group the issues. Roadmaps and boards are used by product managers, too. [1] [2] Example roadmap filtered by the CI section [3]
Every stage, group, feature has labels applied, which allows the creation of specific filters. For example, "Category:Code Suggestions" as label filter [4]
Additional helpers are available in the handbook, for example the "Features by group" overview [5] which shows the feature maintainers in the section, stage, group, product and engineering managers, etc.
Customers can open support tickets, too. [6] Opening a public issue is encouraged so other community members can engage and collaborate. Often, a workaround or fast solution is unveiled because of collaboration and transparency.
When in doubt whether it is a bug or feature, or asking for (troubleshooting) help, you are welcome to join our community forum or Discord. [7] I'm mostly active on the forum myself. [8]
[0] https://about.gitlab.com/handbook/engineering/quality/issue-...
[1] https://about.gitlab.com/direction/#how-we-plan-releases
[2] https://about.gitlab.com/handbook/product/product-processes/...
[3] https://gitlab.com/groups/gitlab-org/-/roadmap?state=opened&...
[4] https://gitlab.com/gitlab-org/gitlab/-/issues/?label_name%5B...
[5] https://about.gitlab.com/handbook/product/categories/feature...
[6] https://about.gitlab.com/support/
[7] https://about.gitlab.com/community/
[8] https://forum.gitlab.com/u/dnsmichi/summary
Requests are triaged using automation and AI, and responsible teams include the reviews in their planning. [0]
Issues (and related MRs) are targeting milestones. Since we at GitLab work in iterations and smaller changes to release features faster, epics help plan and group the issues. Roadmaps and boards are used by product managers, too. [1] [2] Example roadmap filtered by the CI section [3]
Every stage, group, feature has labels applied, which allows the creation of specific filters. For example, "Category:Code Suggestions" as label filter [4]
Additional helpers are available in the handbook, for example the "Features by group" overview [5] which shows the feature maintainers in the section, stage, group, product and engineering managers, etc.
Customers can open support tickets, too. [6] Opening a public issue is encouraged so other community members can engage and collaborate. Often, a workaround or fast solution is unveiled because of collaboration and transparency.
When in doubt whether it is a bug or feature, or asking for (troubleshooting) help, you are welcome to join our community forum or Discord. [7] I'm mostly active on the forum myself. [8]
[0] https://about.gitlab.com/handbook/engineering/quality/issue-...
[1] https://about.gitlab.com/direction/#how-we-plan-releases
[2] https://about.gitlab.com/handbook/product/product-processes/...
[3] https://gitlab.com/groups/gitlab-org/-/roadmap?state=opened&...
[4] https://gitlab.com/gitlab-org/gitlab/-/issues/?label_name%5B...
[5] https://about.gitlab.com/handbook/product/categories/feature...
[6] https://about.gitlab.com/support/
[7] https://about.gitlab.com/community/
[8] https://forum.gitlab.com/u/dnsmichi/summary
What would you prefer?
In practice, customers can also make requests via private support tickets. I’ve seen GitLab determine priority of an issue based partially on demand from paying customers.
GitLab team member here.
> In practice, customers can also make requests via private support tickets.
Yep. Sometimes, it can also be helpful to ask publicly in a forum topic or issue and link that in the support ticket, asking for prioritized support.
> I’ve seen GitLab determine priority of an issue based partially on demand from paying customers.
Correct. You might have seen team members commenting with customer requests into issues and epics. While we at GitLab cannot share customer names and other confidential data, the feature request itself is public and increases transparency.
The process is documented in the product handbook: https://about.gitlab.com/handbook/product/product-processes/...
> In practice, customers can also make requests via private support tickets.
Yep. Sometimes, it can also be helpful to ask publicly in a forum topic or issue and link that in the support ticket, asking for prioritized support.
> I’ve seen GitLab determine priority of an issue based partially on demand from paying customers.
Correct. You might have seen team members commenting with customer requests into issues and epics. While we at GitLab cannot share customer names and other confidential data, the feature request itself is public and increases transparency.
The process is documented in the product handbook: https://about.gitlab.com/handbook/product/product-processes/...
This comment boggles my mind. There are hundreds of developers just employed by gitlab, and likely many thousands of contributors and a multiple of active users voicing their (probably often contradictory) wishes. You want to track this in a system, and their core product includes exactly such a tool that allows people to have precisely this kind of input and transparency. You can add an issue, add a me-too thumbs up (vote), emoticons, comments etc. You will get updated via mails on the progress of the issue, etc.
The customer got a nice reply saying that the exact concern is being worked on, how and where to signal interest, track progress, etc. What did you expect, what more can you do? What would honor mean?
Have you ever worked in software? Do you know have many issues production code has? 50,000 open issues is nothing for a project of this size. Do you have any idea how much issues are closed daily? Gitlab is very actively using their issuetracker and sharing their progress with the world. Try finding a company with as much transparacy as Gitlab.
The customer got a nice reply saying that the exact concern is being worked on, how and where to signal interest, track progress, etc. What did you expect, what more can you do? What would honor mean?
Have you ever worked in software? Do you know have many issues production code has? 50,000 open issues is nothing for a project of this size. Do you have any idea how much issues are closed daily? Gitlab is very actively using their issuetracker and sharing their progress with the world. Try finding a company with as much transparacy as Gitlab.
I wonder why is this on HN. Gitlab has like 10 Security releases/year, this is the first i see on HN. If you self host gitlab you should be subscribed to the security newsletter.
It would save a lot of time on my end, if gitlab had a :stable release tag for docker.
It would save a lot of time on my end, if gitlab had a :stable release tag for docker.
Direct link: https://about.gitlab.com/company/contact/
Scroll down to "Sign up for security notices"
Scroll down to "Sign up for security notices"
[deleted]
Oh, critical issue again. Looks like Gitlab needs security officer to help to prevent this.
The reason you see so many critical Gitlab security fixes is because they take security so seriously.
They pay huge bounties for security vulnerabilities in their products, so they get the best researchers responsibly disclosing bugs.
They pay huge bounties for security vulnerabilities in their products, so they get the best researchers responsibly disclosing bugs.
This sounds like bias. Replace "Gitlab" with "Microsoft" or "Oracle" in your comment and I'd wager you'd feel differently.
I don’t think it is.
Microsoft has a track record for delaying fixes and marking important issues as “not a bug”, so I’m less impressed with their security.
As terrible a corporation as Oracle is, their security response team has been one of the most effective and fast-paced I’ve ever reported to. With that said, they pay nothing to researchers, so Gitlab certainly shows they care more about security.
Microsoft has a track record for delaying fixes and marking important issues as “not a bug”, so I’m less impressed with their security.
As terrible a corporation as Oracle is, their security response team has been one of the most effective and fast-paced I’ve ever reported to. With that said, they pay nothing to researchers, so Gitlab certainly shows they care more about security.
Nice hypothesis, but far away from the reality
GitLab also releases very frequently, minimum once a month with a whole new release. Between releases usually two or three updates but occasionally more.
In this particular class of impersonation problem (the Gitlab issue is about running a pipeline as another user), sometimes I find it unnerving how much an attacker could accomplish in a software company by being able to impersonate an arbitrary Slack user. We are taught to be cautious around incoming email, but I've yet to see a security training around "how do you know that your instant messages are genuine."