Security alerts on GitHub(github.com)
github.com
Security alerts on GitHub
https://github.com/blog/2470-introducing-security-alerts-on-github
102 comments
In the meanwhile you can use a combination of pre-commit (http://pre-commit.com/) and safety-db (https://github.com/pyupio/safety-db) in form of https://github.com/Lucas-C/pre-commit-hooks-safety
That allows you to run a check for vulnerable packages before each commit and via CI.
That allows you to run a check for vulnerable packages before each commit and via CI.
Also for python `pipenv check`[0] to do security checks (obviously for known vulnerabilities) if you use pipenv[1] to manage dependencies.
[0] https://docs.pipenv.org/advanced.html#detection-of-security-...
[1] https://github.com/kennethreitz/pipenv
[0] https://docs.pipenv.org/advanced.html#detection-of-security-...
[1] https://github.com/kennethreitz/pipenv
Another cost of every language wanting at least one of its own dependency managers, often tightly integrated with the build system. The least-bad solution at this point would be to spin up a clean image, install the package in question, and figure out what else was also installed.
> with Python support coming in 2018.
Yeah, but for proper python support you need to handle setup.py too, which being python code is more complex.
https://python-packaging.readthedocs.io/en/latest/dependenci...
https://python-packaging.readthedocs.io/en/latest/dependenci...
And setup.cfg!
Example: https://github.com/HearthSim/python-hslog/blob/master/setup....
Example: https://github.com/HearthSim/python-hslog/blob/master/setup....
and Pipfile - which is what i've switched to in recent projects, example[0]
Altho you can write to one of the other formats or read from it - you'd have to tell the scanner what the true build source is
[0] https://github.com/kennethreitz/pipenv/blob/master/Pipfile
Altho you can write to one of the other formats or read from it - you'd have to tell the scanner what the true build source is
[0] https://github.com/kennethreitz/pipenv/blob/master/Pipfile
I mentioned this elsewhere here, but check out `pipenv check`[0] and just run it in CI or as a precommit hook.
[0] https://docs.pipenv.org/advanced.html#detection-of-security-...
[0] https://docs.pipenv.org/advanced.html#detection-of-security-...
I imagine they are just going to pick one.
> The lack of Python requirements.txt support is a bit odd
I hope they go with Pipfile/Pipfile.lock [0] and pipenv [1] rigth away!
[0] https://github.com/pypa/pipfile
[1] https://github.com/kennethreitz/pipenv
I hope they go with Pipfile/Pipfile.lock [0] and pipenv [1] rigth away!
[0] https://github.com/pypa/pipfile
[1] https://github.com/kennethreitz/pipenv
How would you effectively do this in a way that is cross-build-platform friendly? CMAKE? autoconf? submodules?
I don't see that happening.
I don't see that happening.
Text files?
If people get value from it, editing a simple text file once or twice a year wouldn't be difficult.
If people get value from it, editing a simple text file once or twice a year wouldn't be difficult.
Adding a second source of truth sounds like a bad idea to me. Now you have to update it in lockstep? No thanks.
You could make the build tool generate this standard file.
It's still a many to many problem in both cases.
Option 1: Adding feature to npm/composer/gem/pip ad infinitum
Option 2: add per-language parser support to the alerting tool instead.
Option 2 doesn't necessitate a new (information-duplicating, still potentially error prone) standard, and can likely leverage available, tested libraries ;-)
Option 1: Adding feature to npm/composer/gem/pip ad infinitum
Option 2: add per-language parser support to the alerting tool instead.
Option 2 doesn't necessitate a new (information-duplicating, still potentially error prone) standard, and can likely leverage available, tested libraries ;-)
On the other hand, option #1 requires neither effort nor consent from GitHub to onboard new languages/dependency managers.
A few things things:
* any non Javascript / ruby / go languages don't have standard packages, package names, etc.
* any non Javascript / ruby / go languages may be using one of many build systems. It's just too hard to troll through random build systems to see what dependencies are used
* therefore, a simple text file is what will work, and is what will be trivial for everyone to use
* if you find it too hard to update one line in a text file when you add (for example) a new dependency on libldap... you shouldn't be programming
* any non Javascript / ruby / go languages don't have standard packages, package names, etc.
* any non Javascript / ruby / go languages may be using one of many build systems. It's just too hard to troll through random build systems to see what dependencies are used
* therefore, a simple text file is what will work, and is what will be trivial for everyone to use
* if you find it too hard to update one line in a text file when you add (for example) a new dependency on libldap... you shouldn't be programming
" any non Javascript / ruby / go"
Would have thought Maven popularized standard package names and build dependency (POM) files.
Would have thought Maven popularized standard package names and build dependency (POM) files.
I see.
A text file that just declares that "this project uses lib 2.1". It isn't a part of the build system in any way.
That would be awesome.
A text file that just declares that "this project uses lib 2.1". It isn't a part of the build system in any way.
That would be awesome.
Support for POM files would be nice, too.
Especially since adding pom and/or gradle files would knock out support for a half-dozen languages.
https://gemnasium.com just launched maven support, you can give it a try!
Point probably for lacking support for other than npm&gem packages is that for those langs/formats, there is a good sources for getting that sort of data externally.
Actually, Pipfiles are more akin to the two supported mechanisms, requirements.txt is not always used with version pinning
> With your dependency graph enabled, we’ll now notify you when we detect a vulnerability in one of your dependencies and suggest known fixes from the GitHub community.
Why not go one step further and automatically open a pull request fixing the issue? If you can build a model of the dependency graph, then changing the version to an up-to-date version should be trivial.
Of course, there's no guarantee that the commit in the pull request will successfully build, and you'd still need a project maintainer to fix issues in the build caused by the version change. But the data you get is invaluable. If the security alert is unfixable (from the project maintainer's perspective, since upstream hasn't released a fix yet), then why are you alerting a maintainer about something unfixable? If the proposed fix's build is green, then the alert should have a "higher volume" (so to speak), and if the proposed fix's build is red, then the "volume" of the alert should be turned down so that the team can focus on fixing the build so that the security patch should be applied.
Why not go one step further and automatically open a pull request fixing the issue? If you can build a model of the dependency graph, then changing the version to an up-to-date version should be trivial.
Of course, there's no guarantee that the commit in the pull request will successfully build, and you'd still need a project maintainer to fix issues in the build caused by the version change. But the data you get is invaluable. If the security alert is unfixable (from the project maintainer's perspective, since upstream hasn't released a fix yet), then why are you alerting a maintainer about something unfixable? If the proposed fix's build is green, then the alert should have a "higher volume" (so to speak), and if the proposed fix's build is red, then the "volume" of the alert should be turned down so that the team can focus on fixing the build so that the security patch should be applied.
We do exactly that at Depfu (for Ruby): https://depfu.com/blog/2017/09/28/depfu-now-flags-security-u...
But I'm also pretty sure Github is going to do that soon, it's an obvious next step.
But I'm also pretty sure Github is going to do that soon, it's an obvious next step.
> If the security alert is unfixable (from the project maintainer's perspective, since upstream hasn't released a fix yet), then why are you alerting a maintainer about something unfixable?
Sometimes a vulnerability in a dependency doesn't affect you, because you're not using that feature, in other cases the issue does affect you but it's possible to work around it. For the latter case there isn't necessarily a patch that can be generated automatically.
Sometimes a vulnerability in a dependency doesn't affect you, because you're not using that feature, in other cases the issue does affect you but it's possible to work around it. For the latter case there isn't necessarily a patch that can be generated automatically.
This is both brilliant and frustrating. The brilliant part is much discussed here already, so...
The frustration is that my projects, and I suspect the bulk of projects the get these alerts, get them because of transitive dependency. At least from my first look I did not see the transitive dependency path mentioned, so although I didn't need to know anything to read the alert, I had to dig through NPM features to figure it out.
As the maintainer of "end-user" projects that depend on various big pieces of tooling, it turns out that I'm not really much of a position to do anything about the transitive dependencies anyway, other than manually track down the dependency path and then go open issues in each thing along the way.
Something like the following seems like it would help focus attention toward the points where things have to be fixed: a kind of leaderboard, a (gentle and friendly) "wall of shame". Projects would earn their way to the top of this wall by depending on vulnerable things themselves, and then being depended on by other projects directly or indirectly.
The frustration is that my projects, and I suspect the bulk of projects the get these alerts, get them because of transitive dependency. At least from my first look I did not see the transitive dependency path mentioned, so although I didn't need to know anything to read the alert, I had to dig through NPM features to figure it out.
As the maintainer of "end-user" projects that depend on various big pieces of tooling, it turns out that I'm not really much of a position to do anything about the transitive dependencies anyway, other than manually track down the dependency path and then go open issues in each thing along the way.
Something like the following seems like it would help focus attention toward the points where things have to be fixed: a kind of leaderboard, a (gentle and friendly) "wall of shame". Projects would earn their way to the top of this wall by depending on vulnerable things themselves, and then being depended on by other projects directly or indirectly.
While its not hard to setup, Github doesn't cover how to do so for private repositories. I put the steps for setting this up on a per repositor basis here:
http://fuzzyblog.io/blog/github/2017/11/17/enabling-github-s...
http://fuzzyblog.io/blog/github/2017/11/17/enabling-github-s...
(Hint: Gemnasium founder here)
If you like this feature, you may want to try https://gemnasium.com then. We have a lot more advisories in db, for Java, Python, Ruby, PHP and JavaScript. Please feel free if you have any question, I’ll be glad to help!
I wonder how this interacts with submodule-style dependencies.
node.js security is hard to get correctly.
Even if you keep your dependencies up to date, your application might be pretty vulnerable unless you know what you are doing.
This is because:
1) frameworks are minimalistic and they give you some large room for error.
2) javascript is very dynamic, and it is time consuming to validate types... unless you are using something like typescript.
3) people tend to use node for orchestration layers/api gateways... and focus their security on the underlying API. But exfiltrating at the orchestration layer is as severe.
This is because:
1) frameworks are minimalistic and they give you some large room for error.
2) javascript is very dynamic, and it is time consuming to validate types... unless you are using something like typescript.
3) people tend to use node for orchestration layers/api gateways... and focus their security on the underlying API. But exfiltrating at the orchestration layer is as severe.
I hope github moves into static analysis!
This is great! Looking for to using it.
No python support :*(
The post mentions it will be coming in 2018
This is something that lgtm.com supports. Right now, I can't find a Python project with a dependency on a vulnerable package to show you, but here's the page that shows Django's dependencies (and this is where a known vulnerable version would be highlighted).
https://lgtm.com/projects/g/django/django/dependencies
https://lgtm.com/projects/g/django/django/dependencies
You can try https://gemnasium.com, it has Python support.
You can use requires.io or pyup.io right now for Python.
One can use sourceclear.com for Python support.
But hey!
Python support coming in 2018.When I ran the languages team at Heroku this was something we'd in some ad-hoc basis do for customers when there were major vulnerabilities. To see this fully productized is absolutely awesome! Well done to the GitHub team.
[deleted]
Retire.js does this for JS locally, we have a nightly check for our apps. https://retirejs.github.io/
iOS projects with Fastlane automatically looks for vulnerabilities with the gems installed.
Clicked through the link to the security apps on the Github Marketplace and had a look at Snyk.io's pricing: 119$ through Github, 99$ if you get it straight from Snyk.io.
And looking deeper into Github Marketplace pricing, I can see that Github takes a 25% cut (https://developer.github.com/apps/adding-integrations/managi...).
What is the benefit of getting anything via the Github marketplace that can be subscribed to outside of the marketplace ? What justifies that 25% cut appart from having a listing of apps ?
And looking deeper into Github Marketplace pricing, I can see that Github takes a 25% cut (https://developer.github.com/apps/adding-integrations/managi...).
What is the benefit of getting anything via the Github marketplace that can be subscribed to outside of the marketplace ? What justifies that 25% cut appart from having a listing of apps ?
Hey, I'm one of the cofounders of ZenHub - we're another popular integration on the marketplace and also sell our product separately through our own website.
From a company/integration on the marketplace, the marketplace has been great in terms of building awareness and exposure. Can confidently say that we've been able to reach new users and customers that we wouldn't have otherwise had if we weren't on the marketplace.
From the perspective of a GitHub user/team, lots of teams prefer to consolidate everything together on a single invoice. If you're using GitHub for software development, you're almost certainly going to need a CI/CD tool and PM tool, so why not bring everything together on a single bill?
From a company/integration on the marketplace, the marketplace has been great in terms of building awareness and exposure. Can confidently say that we've been able to reach new users and customers that we wouldn't have otherwise had if we weren't on the marketplace.
From the perspective of a GitHub user/team, lots of teams prefer to consolidate everything together on a single invoice. If you're using GitHub for software development, you're almost certainly going to need a CI/CD tool and PM tool, so why not bring everything together on a single bill?
Single bill is not a reason to pay 25 percent more for everything.
If you're a large enterprise, paying an extra $20/month could definitely be worth not having the pain & cost of having your employees having to manage paying an extra bill each month.
[deleted]
Pricing for the marketplace was completely left up to the discretion of the integrators. There's a number of integrations (ZenHub included) that are the same price whether you buy through the marketplace or through company websites :)
So you lose 25% of your revenue on github marketplace sales?
If you see it that way, yes. But getting people to sign up on your website without the 25% cut is also not for free, think about costly advertising and so on. This is probably what Github and other marketplaces bet on. You always pay for any channel, whether it's direct cuts or upfront costs to get a customer.
What is the benefit of getting anything via the Github marketplace that can be subscribed to outside of the marketplace?
Potentially, your company might have a GitHub account you're allowed to add services to, but the finance department is less keen on new accounts. It also provides a well known company to complain to instead of yet another small vendor. Things enterprisey type customers tend to like. Manifold - https://www.manifold.co/ - is another player in this space.
Potentially, your company might have a GitHub account you're allowed to add services to, but the finance department is less keen on new accounts. It also provides a well known company to complain to instead of yet another small vendor. Things enterprisey type customers tend to like. Manifold - https://www.manifold.co/ - is another player in this space.
Exactly. Heroku Add-ons are also fantastic in that regard.
Perhaps GitHub is taking a cut from the purchase when purchased through their marketplace (20% is reasonably close to other marketplaces). Some services may be building that into the price.
How long before this makes github enterprise I wonder.
I think the answer will be: Once it has been on github.com long enough for people to find any issues that GitHub themselves missed in their internal testing.
(Meta: That GIF should have been two small screenshots. No animation, no looping, no racing to understand before it switches frames. Stop making things GIFs that don't need to be GIFs. It hurts accessibility and adds nothing.)
Or at least a WebM. But not a fucking 2.8 MiB large GIF.
Just looked at the "dependencies" for one of my projects and it interprets "react" as this 5 year old, defunct repo: https://github.com/wballard/react.
I am thinking that it's because the React package.json does not have `repository` listed.
https://github.com/facebook/react/blob/master/package.json
See Redux: https://github.com/reactjs/redux/blob/master/package.json#L3...
https://github.com/facebook/react/blob/master/package.json
See Redux: https://github.com/reactjs/redux/blob/master/package.json#L3...
Let's be honest, unless you use a private NPM registry, "react" means https://www.npmjs.com/package/react. GitHub should use the standard resolution algorithm as employed by npm/yarn (the CLI tools).
That's not the package.json that gets published in the npm react package. This one is what you're looking for:
https://github.com/facebook/react/blob/master/packages/react...
https://github.com/facebook/react/blob/master/packages/react...
How are you depending on react?
I just checked 3 of my projects that use it, and all 3 are pointing to the correct "facebook react" repo.
I just checked 3 of my projects that use it, and all 3 are pointing to the correct "facebook react" repo.
Just a standard dependency on version 15. It figured out react-dom just fine though.
{
"dependencies": {
"react": "^15.6.1",
"react-dom": "^15.6.1"
}
}Get in touch with support, they can point it to the correct place!
Not sure about this. I can see this as an "added stress" if we can't update a dependency for whatever reason and alerts keep popping in.
With this kind of reasoning you don't need any logging on your servers.
I do see how knowing that your app has vulnerabilties that you don't plan to fix could be stressful. :)
Whether github is doing it or not, the baddies will be using similar tools to scan I'm sure.
Whether github is doing it or not, the baddies will be using similar tools to scan I'm sure.
Well this is frustrating. There is an issue with one of my projects apparently (actually saw it before I saw this on HN) and it is because of one of my indirect dependencies.
I presume in this scenario I need to either wait for a patch from the direct dependency or fork and submit a PR myself.
It's a great idea. I like it quite a bit. I just feel like the floodgates just got opened.
Would be great to see PHP and Python in there.
I presume in this scenario I need to either wait for a patch from the direct dependency or fork and submit a PR myself.
It's a great idea. I like it quite a bit. I just feel like the floodgates just got opened.
Would be great to see PHP and Python in there.
There is now an incentive for that vulnerability to be addressed that didn't exist before. Seems like a security win overall.
This is nice but I really hope people don't become complacent with this and rely on GitHub to notify them of such things.
I don't know of any way to keep up with vulnerabilities _except_ an automated system. I use `bundle audit` in ruby-land. I don't know why github's automated system will be worse than anyone elses.
What alternative is there? Hoping you notice something on a listserv and realize its' one of your (possibly indirect) dependencies? That does not seem better. Automated monitoring and alert is the way to go.
And _everyone_ should _always_ be filing CVE's for their vulnerabilities, to make automated detection so much easier.
What alternative is there? Hoping you notice something on a listserv and realize its' one of your (possibly indirect) dependencies? That does not seem better. Automated monitoring and alert is the way to go.
And _everyone_ should _always_ be filing CVE's for their vulnerabilities, to make automated detection so much easier.
Automated tooling is a must, yes. The riskiest part about relying on ONLY GH's solution (IMO) is the NVD/CVE limitation.
I agree, CVE would be _awesome_ in theory. In reality, very few file for CVE's and so the coverage is iffy (~11% of npm package vulns and about ~67% of rubygem vulns https://snyk.io/stateofossecurity/).
But it goes beyond that. There was a great paper earlier this year (https://arxiv.org/abs/1705.05347) that highlighted many other issues: lag between CVE and NVD (which is where all the useful info comes from), mismatched CPE's, nonexistent CPE's, etc.
I would love to see us get to a point where the CVE/NVD was enough, but we're far from it right now.
I agree, CVE would be _awesome_ in theory. In reality, very few file for CVE's and so the coverage is iffy (~11% of npm package vulns and about ~67% of rubygem vulns https://snyk.io/stateofossecurity/).
But it goes beyond that. There was a great paper earlier this year (https://arxiv.org/abs/1705.05347) that highlighted many other issues: lag between CVE and NVD (which is where all the useful info comes from), mismatched CPE's, nonexistent CPE's, etc.
I would love to see us get to a point where the CVE/NVD was enough, but we're far from it right now.
github's announcement made clear they don't plan to only rely on CVE/NVD database, but yeah, you've got to pick a tool you think is good.
I think a great many people at non-large companies are using free tools that I think are unlikely to be better than github's. Or no tool at all.
I think a great many people at non-large companies are using free tools that I think are unlikely to be better than github's. Or no tool at all.
> The riskiest part about relying on ONLY GH's solution (IMO) is the NVD/CVE limitation.
Yup, that is the point I was trying to make.
Yup, that is the point I was trying to make.
Hopefully PHP (composer.json) support makes it in soon! :)
https://gemnasium.com has composer support, you can give it a try!
I'd like it if github shows which projects had to be reminded of vulnerabilities this way instead of the developers knowing/addressing things themselves.
It seems like GitHub is slowly moving towards building their own static code analysis and security scanning. CI is probably coming up. This will bury many 3rd party services, but will be more convenient for end users.
I've said this in the past, but you really have to focus on the more difficult things, if you want to stay competitive in Git hosting. If it is easy to implement and is useful, one of your competitors, is going to copy you.
Right now GitHub, GitLab, Bitbucket, Microsoft, Gogs/Gitea, etc. all have something unique to them, but none of them, have the lock in power, like it was in the past with Perforce, ClearCase and other SCMs.
GitLab and other open source solutions, has turned core Git hosting functionality, into commodity features, that people expect to be good and cheap. So it only makes sense to start focusing on the not so hard things to do, which can't be easily duplicated.
Right now GitHub, GitLab, Bitbucket, Microsoft, Gogs/Gitea, etc. all have something unique to them, but none of them, have the lock in power, like it was in the past with Perforce, ClearCase and other SCMs.
GitLab and other open source solutions, has turned core Git hosting functionality, into commodity features, that people expect to be good and cheap. So it only makes sense to start focusing on the not so hard things to do, which can't be easily duplicated.
GitLab is giving them some competition so they've got to step it up. Nice to see, honestly.
> Social Network for Children
Lego forums (I don't know if they still exist though). Apparently they went to great lengths to make them safe and accessible for children.
That said though, if anyone can crack this problem, it will be awesome. Children are already social, and are increasingly on the internet. And there are very few, if any, kid-friendly (not kid-condescending, or kids-as-afterthought) resources.
Most of the other problems listed look like first-world problems.
Lego forums (I don't know if they still exist though). Apparently they went to great lengths to make them safe and accessible for children.
That said though, if anyone can crack this problem, it will be awesome. Children are already social, and are increasingly on the internet. And there are very few, if any, kid-friendly (not kid-condescending, or kids-as-afterthought) resources.
Most of the other problems listed look like first-world problems.
Wrong thread.
I noticed this yesterday. Too bad there's no Composer support (PHP)
https://gemnasium.com has composer support, you can give it a try!
One can use sourceclear.com for Composer.
I've been using https://hakiri.io/ for this on my ruby github projects[1].
[1] https://github.com/rickhull?tab=repositories&type=source&lan...
[1] https://github.com/rickhull?tab=repositories&type=source&lan...
I love whoever's responsible for this.
This is awesome! I found a security issue in an Elixir/Erlang project that was a dependency of many other projects and it was manual and complicated to coordinate amongst them. IIRC I suggested this to one of the other parties, although it seems trivial and obvious to other people, but it's good that GH also came upon this independently and shipped it.
Have had this for years.
mvn org.owasp:dependency-check-maven:check
Java FTW again :)
mvn org.owasp:dependency-check-maven:check
Java FTW again :)
Just like Clair (https://github.com/coreos/clair), only, for GitHub
Think carefully about the consequences of this automated scan info leaking before enabling it in your private repositories.
Are the consequences significantly different than having private repos on Github to begin with?
I realize this is one more Github service with access to metadata about your code, so the attack surface is technically larger, but is the probability of leak that much greater from this service than it already is by having your code on their servers (which, I assume, are protected by the same security team)?
Maybe it's an oversimplification but my expectation is that any project with such sensitivity wouldn't be hosted on an external service at all.
I realize this is one more Github service with access to metadata about your code, so the attack surface is technically larger, but is the probability of leak that much greater from this service than it already is by having your code on their servers (which, I assume, are protected by the same security team)?
Maybe it's an oversimplification but my expectation is that any project with such sensitivity wouldn't be hosted on an external service at all.
You don't need to pay anyone for this service. GitHub and Snyk use free public vulnerability lists to check your dependencies.
There are plenty of open source alternatives such as https://github.com/RetireJS/retire.js for JavaScript.
It's absurd that companies are charging $100/mo just to run your dependency list against another public list of vulnerabilities. This service should be offered for free by GitHub.
There are plenty of open source alternatives such as https://github.com/RetireJS/retire.js for JavaScript.
It's absurd that companies are charging $100/mo just to run your dependency list against another public list of vulnerabilities. This service should be offered for free by GitHub.
This _is_ free from GitHub. At least there's no mention of price in the blog post. I seem to have this feature.
The lack of Python requirements.txt support is a bit odd, since it's conceptually quite similar to the two supported mechanisms.