Mintlify GitHub read/write token leak(mintlify.com)
mintlify.com
Mintlify GitHub read/write token leak
https://mintlify.com/blog/incident-march-13
44 comments
This is pretty typical of GitHub, unfortunately. They have terrible scoping and don't give developers the option to even give users that flexibility. It's been like this for over a decade, I don't think GitHub will ever learn.
Wrong. “GitHub Apps” always (?) had fine-grained repo-scoped permissions, it’s the “OAuth apps” that are stuck with coarse-grained permissions. So developers do have the option, it’s even the preferred option.[1]
> In general, GitHub Apps are preferred over OAuth apps.
[1] https://docs.github.com/en/apps/oauth-apps/building-oauth-ap...
> In general, GitHub Apps are preferred over OAuth apps.
[1] https://docs.github.com/en/apps/oauth-apps/building-oauth-ap...
Even with apps, you still do not have the fine grained permissions you'd think you'd get. Have you tried?
You can for sure. I am using it with Cloudflare pages for example. It's a matter of a click.
With apps, you can certainly grant access to individual repos instead of all repos. Yes I’ve tried.
Here’s how this thread went:
brycelarkin: Mintlify’s GitHub app asked for permissions to access all my repos.
You: Developers don’t have the option to not access all repos.
Me: False. GitHub apps do have fine grained permissions, as in they don’t have to have to access all repos.
You: Even with apps you don’t have fine grained permissions “you’d think you’d get”.
So what are you talking about? What’s the thing I’d think I’d get but not getting and how is that related to “permissions to access all repos” as discussed here?
Here’s how this thread went:
brycelarkin: Mintlify’s GitHub app asked for permissions to access all my repos.
You: Developers don’t have the option to not access all repos.
Me: False. GitHub apps do have fine grained permissions, as in they don’t have to have to access all repos.
You: Even with apps you don’t have fine grained permissions “you’d think you’d get”.
So what are you talking about? What’s the thing I’d think I’d get but not getting and how is that related to “permissions to access all repos” as discussed here?
Org access. You seem to be very hostile in your tone, by the way.
"Always" is doing a lot of work here. Github Apps are relatively new.
If half a decade is new, sure. Except I was replying to someone claiming “I don't think GitHub will ever learn”, commenting on an article from a company created in 2021.
Exactly. You'd need to implement standard authentication mechanisms, preventing these vectors of attack from existing in the first place.
They are "in the process" and "looking" to do that. As of now if I am not mistaken the main issue still persists and any other new security vulnerability will possibly give full access to tokens again.
Correct me if I am wrong please but I don't see anything in the comment or Blog post saying otherwise.
> and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
https://www.reddit.com/r/cscareerquestions/comments/1bh22bq/...
They are "in the process" and "looking" to do that. As of now if I am not mistaken the main issue still persists and any other new security vulnerability will possibly give full access to tokens again.
Correct me if I am wrong please but I don't see anything in the comment or Blog post saying otherwise.
> and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
https://www.reddit.com/r/cscareerquestions/comments/1bh22bq/...
[deleted]
Mintlify sponsors one of my open source projects, express-rate-limit, and we use their service to host the docs for it. So I'm probably biased, but I've honestly been pretty happy with it overall - I'm thinking about using it for some of my other projects.
I think the best thing I can say about it is this: Mintlify helps us write better docs. Before Mintlify, we had a humongous readme plus a few one-off articles in the wiki. After switching the Mintlify, we started organizing things better, beginning with a less overwhelming readme.
I found out about this hack when they sent me an email yesterday. I looked through the git history (edit: and github activity), but nothing seems awry.
It sucks, but they're a pretty small business, and I think they're handling it reasonably well.
I think the best thing I can say about it is this: Mintlify helps us write better docs. Before Mintlify, we had a humongous readme plus a few one-off articles in the wiki. After switching the Mintlify, we started organizing things better, beginning with a less overwhelming readme.
I found out about this hack when they sent me an email yesterday. I looked through the git history (edit: and github activity), but nothing seems awry.
It sucks, but they're a pretty small business, and I think they're handling it reasonably well.
> I looked through the git history, but nothing seems awry.
Git history or GitHub event history? You can easily push a malicious commit forged to have the same author and short hash.
Git history or GitHub event history? You can easily push a malicious commit forged to have the same author and short hash.
Good point. I only looked through the git history initially, but I've now looked through both. Same conclusion.
I reported security issues to Mintlify back when they used Discord for support. Gave recreation steps. Never got any feedback that looked to me like they were looking into it.
I reported similar issues to what's described in May 2023.
I reported similar issues to what's described in May 2023.
Hey Ethan! Han from Mintlify here.
It's been awhile I hope you're doing well!
The vulnerability issue that you reported, if I recall correctly, was relating to server-forwarding attacks via our API playground. We've addressed that issue around 8 months ago, and unfortunately, it was not the root of this incident.
Please correct me if I am wrong though!
It's been awhile I hope you're doing well!
The vulnerability issue that you reported, if I recall correctly, was relating to server-forwarding attacks via our API playground. We've addressed that issue around 8 months ago, and unfortunately, it was not the root of this incident.
Please correct me if I am wrong though!
Oof not a fun incident, this is my nightmare as someone who works on this type of stuff.
As an aside, GitHub’s security model for apps/integrations is extremely puzzling to reason about and enables a lot of foot guns. Add the fact that it’s very obtuse to audit integrations (especially within an organization) makes them pretty scary to use sometimes.
As an aside, GitHub’s security model for apps/integrations is extremely puzzling to reason about and enables a lot of foot guns. Add the fact that it’s very obtuse to audit integrations (especially within an organization) makes them pretty scary to use sometimes.
Did I miss it or does the announcement completely evade explaining HOW the tokens actually leaked?
This is Han, I'm one of the co-founders from Mintlify.
For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.
The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information.
Our initial patch upon discovering the incident fixed the response of the vulnerable endpoint, but we have since also implemented a sweeping number of security provisions and are also in the process of completely deprecating the admin tokens and GitHub OAuth entirely to prevent an incident like this from ever happening again.
For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.
The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information.
Our initial patch upon discovering the incident fixed the response of the vulnerable endpoint, but we have since also implemented a sweeping number of security provisions and are also in the process of completely deprecating the admin tokens and GitHub OAuth entirely to prevent an incident like this from ever happening again.
>"we've decided to leave out the technical details of the breach in the blog post"
>"Our dedication to transparency, security, and the trust you place in us remains unwavering."
You are contradicting yourself here.
>"The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information."
Why would you leave that out? Seems like it is vital information.
>"Our dedication to transparency, security, and the trust you place in us remains unwavering."
You are contradicting yourself here.
>"The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information."
Why would you leave that out? Seems like it is vital information.
> For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.
This doesn't exactly inspire confidence that your service is now secure.
This doesn't exactly inspire confidence that your service is now secure.
It's most likely not. Not sure if this was intentional but they pretty much confirmed it in a reddit thread:
> ... and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
https://www.reddit.com/r/cscareerquestions/comments/1bh22bq/...
> ... and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
https://www.reddit.com/r/cscareerquestions/comments/1bh22bq/...
Yea it does. It most likely is related to MDX documents beeing susceptible to XSS attacks. Having worked at mintlify last year I can tell you that this is not surprising at all and I've been warning them extensively of sth like this happening.
I really like mintlify and I found this writeup to be straightforward and responsible. Shit happens.
"we cannot confirm that no other such instances occurred". I'd like to understand this more. Do they 1. not have a fingerprint of what the exploit looked like because it was indistinguishable from customer behavior? 2. Not have logging necessary to audit this? Regardless, with the added monitoring to detect suspicious activity, do they also have the logging to see everything a bad actor did if there is a new exploit?
So a user (or white hat?) alerts them to security issue on their endpoint. Email comes in at 4:55 PDT on a Friday.
[first red flag] No internal monitoring to check for unauthorized access
Poor engineer(s) tasked with issue on a Friday at end of work day manually rakes logs in their app. After ~1 hr of searching, discovers unauthorized device using leaked credentials
[second red flag] allows any device to hook into their critical infrastructure and access apis that have potential to expose PII and possibly move laterally within the org or customers.
Then poor engineers tasked with painstaking task to rotate all of the tokens throughout the night. Bye bye family time, any planned events (game tickets?). I hate this company already and I have no clue wtf they do.
[third red flag] why the fuck are they storing user tokens in a database. This is apparently a “SOC 2” certified application/company. Shows how much that is worth here.
[edit: fourth red flag] no indication of how long the unauthorized credentials have been used and what customers have been impacted. Very very piss poor logging or purposely omitting due to potential litigation issues
Probably missed a few but this is bad.
The “S” in Mintify clearly stands for security.
[first red flag] No internal monitoring to check for unauthorized access
Poor engineer(s) tasked with issue on a Friday at end of work day manually rakes logs in their app. After ~1 hr of searching, discovers unauthorized device using leaked credentials
[second red flag] allows any device to hook into their critical infrastructure and access apis that have potential to expose PII and possibly move laterally within the org or customers.
Then poor engineers tasked with painstaking task to rotate all of the tokens throughout the night. Bye bye family time, any planned events (game tickets?). I hate this company already and I have no clue wtf they do.
[third red flag] why the fuck are they storing user tokens in a database. This is apparently a “SOC 2” certified application/company. Shows how much that is worth here.
[edit: fourth red flag] no indication of how long the unauthorized credentials have been used and what customers have been impacted. Very very piss poor logging or purposely omitting due to potential litigation issues
Probably missed a few but this is bad.
The “S” in Mintify clearly stands for security.
Unfortunate, but impressed by how the Mintlify team is handling it.
They are claming that they resolved the vulnerability that caused the token leak but don't mention it. Doesn't exactly seem transparent to me or like handling it well.
I was contracting for them last year and tried, among other things to build an actual engineering culture that prevents and fixes issues that accumulate to catastrophic incidents like this.
They generally prefer to "ship fast".
I informed them very thoroughly again on January 13th (3+ months after they terminated me for "cultural differences"), because I was worried of this exact nightmare scenario happening very soon.
The reason for this was that they open sourced a package that let's an attacker easily practice and test locally in like a minute.
MDX exposes to Cross site Scripting easily. I assume this is the "fixed vulnerability" they are talking about, just to be transparent.
I was contracting for them last year and tried, among other things to build an actual engineering culture that prevents and fixes issues that accumulate to catastrophic incidents like this.
They generally prefer to "ship fast".
I informed them very thoroughly again on January 13th (3+ months after they terminated me for "cultural differences"), because I was worried of this exact nightmare scenario happening very soon.
The reason for this was that they open sourced a package that let's an attacker easily practice and test locally in like a minute.
MDX exposes to Cross site Scripting easily. I assume this is the "fixed vulnerability" they are talking about, just to be transparent.
Even something as simple as bolding the message about customer repositories being accessed is nice. Not trying to bury the lead.
Two weeks to notify customers with many first finding out on social media is impressive?
I'd been interested in Mintlify and their documentation platform for a while.
I saw this pop up based on this Reddit thread and on Twitter as well:
https://www.reddit.com/r/ExperiencedDevs/comments/1bf7eqa/ni...
This seems serious? Is this really serious?
Why would they need to save these tokens in the first place?
I saw this pop up based on this Reddit thread and on Twitter as well:
https://www.reddit.com/r/ExperiencedDevs/comments/1bf7eqa/ni...
This seems serious? Is this really serious?
Why would they need to save these tokens in the first place?
If they're a GitHub App, they receive a token to authenticate into your account/org when you grant them access/enable the app.
Everyone should audit their GitHub Apps periodically/avoid using them if at all possible IMO. Most of these integrations are just a convenience for adding webhooks, which you can do yourself without compromising security. Always prefer "outbound" integrations.
Everyone should audit their GitHub Apps periodically/avoid using them if at all possible IMO. Most of these integrations are just a convenience for adding webhooks, which you can do yourself without compromising security. Always prefer "outbound" integrations.
It’s super serious …
Out of curiosity, what happened to the customer whose private repo was breached?
This opens the door for supply chain attack. Removing mintlify and auditing other integrations ASAP should be considered as #1 priority.
SOC 2 compliant. Tells you exactly how useful SOC compliance is.
All SOC tells you is that there is a process being (mostly) followed. Always review the reports for your vendors and follow up on any findings or gaps that show up.
It's surprisingly common for places to be SOC compliant, yet their latest report has half a dozen or more gaps/findings.
It's surprisingly common for places to be SOC compliant, yet their latest report has half a dozen or more gaps/findings.
[deleted]
Auditors don't get paid to fail their customers ;) I think many people would be surprised just how.. Accommodating the process can be.
Broadly speaking, being SOC 2 "compliant" isn't really a thing. They just had a SOC 2 audit done, you have to read the report to discover how "compliant" they are.
SOC 2 is useful as setting a baseline for how a business and IT org should be ran, assuming it's followed...
SOC 2 is useful as setting a baseline for how a business and IT org should be ran, assuming it's followed...
An artificial barrier to entry. What a racket.
For customer OAuth tokens, I believe you should NEVER store the access token in the database. Only store the refresh token in the database. When you need api access, get a new access token with the refresh token and client secret.
This prevents incidents like the above as the attacker would also need your client secret, which is ideally not in your database.