The HDFS Juggernaut(blog.shodan.io)
blog.shodan.io
The HDFS Juggernaut
https://blog.shodan.io/the-hdfs-juggernaut/
62 comments
It is changing. Disclosure- I work on Hops Hadoop.
In Hops Hadoop, we have replaced Kerberos with TLS/SSL certificates everywhere. It massively reduces the operational complexity (Certificates are managed by a new service called Hopsworks that also provides UI to Spark/Tensorflow/Hadoop) with about 7-10% overhead for I/O (maximal b/w when reading/writing a file in HDFS is reduced).
We will be launching a full release the end of August on www.hops.io.
Yes, Kerberos seems complex but in practice it's not the technology being a problem but usually organizational problems. But one big benefit that Kerberos has over certificates is that almost every big company has a central Active Directory and almost every user already has Kerberos tickets.
The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues.
I wonder what you mean by "operational complexity" considering that once you have the principals created there's basically no overhead....
Or to put it differently: Can you elaborate on which problem you're trying to solve?
The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues.
I wonder what you mean by "operational complexity" considering that once you have the principals created there's basically no overhead....
Or to put it differently: Can you elaborate on which problem you're trying to solve?
Actually, we use LDAP. If you have an LDAP account, you're authenticated. The use of the certs is transparent to clients - it's wrapped in library calls.
The problem we're addressing is multi-tenancy for users on Hadoop. If you have sensitive data and want to restrict access to it, you need at a minimum dynamic roles or attribute-based access controls. With Kerberos, you have a single ID, a single role. You can't revoke privileges from users, give him/her access to a dataset, then restore privileges. We introduced dynamic roles by introducing a new abstraction - a project. You get a project-specific username - and a new cert for every project-specific username. Unlike Kerberos with the centralized KDC, cert generation can be scaled out with intermediate servers. This way, you can give a project-specific user access to sensitive data in a project - and the user can't export the data, as that user only has privileges for resources within the project. You can still share datasets across projects using either ACLs or group privileges in HDFS.
The problem we're addressing is multi-tenancy for users on Hadoop. If you have sensitive data and want to restrict access to it, you need at a minimum dynamic roles or attribute-based access controls. With Kerberos, you have a single ID, a single role. You can't revoke privileges from users, give him/her access to a dataset, then restore privileges. We introduced dynamic roles by introducing a new abstraction - a project. You get a project-specific username - and a new cert for every project-specific username. Unlike Kerberos with the centralized KDC, cert generation can be scaled out with intermediate servers. This way, you can give a project-specific user access to sensitive data in a project - and the user can't export the data, as that user only has privileges for resources within the project. You can still share datasets across projects using either ACLs or group privileges in HDFS.
Thank you for the extra explanations. But I'm afraid I still don't understand.
With Kerberos you have a single Id, yes but that is augmented by groups coming from LDAP and RBAC systems like Sentry & Ranger.
Also having another system issuing certificates automatically is a no-go in lots of companies and especially security conscious industries.
I'm not saying that there's not a market for what you're developing - I'm merely trying to understand and I have a hard time. Your homepage, I'm afraid though, is useless at conveying any information about what you're actually doing.
With Kerberos you have a single Id, yes but that is augmented by groups coming from LDAP and RBAC systems like Sentry & Ranger.
Also having another system issuing certificates automatically is a no-go in lots of companies and especially security conscious industries.
I'm not saying that there's not a market for what you're developing - I'm merely trying to understand and I have a hard time. Your homepage, I'm afraid though, is useless at conveying any information about what you're actually doing.
Sentry and Ranger offer RBAC, but not dynamic roles or attribute-based access control. Dynamic roles for us are: if you are a data scientist working on a dataset, when working on that dataset, we give you the minimum roles needed to work on that dataset. If you normally have privileges to write to some other directory outside that subtree in HDFS, we take it away while you are active in the project. When you change projects, your access rights change.
Sentry and Ranger are horrendous. Best practice for them in HDFS is to change permissions for subtrees in Hive/Impala to '000' so that sysadmins know that an external system is doing access control here. They do not offer unified access control, rather fragmented access control. In our case, you see a different username in hdfs for each of your projects.
I agree the homepage sucks - it's on our todo list :)
I agree the homepage sucks - it's on our todo list :)
Btw, I gave the talk after you at the Dataworks Summit in Munich. Nice talk!
Rigidity when it comes to tenancy and lack of groups were two of the reasons why Google went with its own LOAS instead of Kerberos for the production environment. The even bigger one was performance.
Yes, the Kerberos KDC is a big bottleneck. In contrast, you can scale out TLS/SSL cert generation with intermediate CAs.
> The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues.
Yeah, getting a new client certificate issued for a system account is like trying to squeeze blood out of stone.
Yeah, getting a new client certificate issued for a system account is like trying to squeeze blood out of stone.
We ship the certificates out transparently to users using Yarn. Yarn keeps them as private local resources and cleans them up for us.
> Sometimes it is OK if you run a cluster insecure.
Sure, but put it behind a VPN or something. This article is literally about clusters accessible via the public Internet. If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them.
Sure, but put it behind a VPN or something. This article is literally about clusters accessible via the public Internet. If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them.
> If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them.
Honeypot? :P
Honeypot? :P
I dont think this article is about clusters accessible via public internet. I think this article is about clusters accessible after breaking through an easily defeated ssh barrier and gaining full access.
Linked from the article:
https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT
That's a list of hadoop nodes you can access via the public Internet.
https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT
That's a list of hadoop nodes you can access via the public Internet.
[deleted]
Why do all these products have "insecure by default" configurations, anyways?
Didn't we learn anything from register_globals?
Didn't we learn anything from register_globals?
The alternative is "asks you lots of questions at install, and/or makes you generate keys or something like that", which makes people say it's too complicated to set up.
Most people want the simple insecure one for their lab to see if it will work for them before they deploy the locked down version on the internet. Of course the part where they make sure to secure the production one is where people forget.
This isn't crazy. If someone asks me to do some benchmarks on an app to see if it would fit our problem, the last thing I want is to spend a week working up a federation of keyservers and a fake domain and everything else that the deployed product will require. If the product doesn't fit our need that's just wasted time.
Most people want the simple insecure one for their lab to see if it will work for them before they deploy the locked down version on the internet. Of course the part where they make sure to secure the production one is where people forget.
This isn't crazy. If someone asks me to do some benchmarks on an app to see if it would fit our problem, the last thing I want is to spend a week working up a federation of keyservers and a fake domain and everything else that the deployed product will require. If the product doesn't fit our need that's just wasted time.
Or there is middle ground
"Allow easy access with no password or username, but only from localhost"
Not foolproof, but as far as I know both this and mongo did the "bind to all ports AND allow access with no password". It was the combo that really killed it.
"Allow easy access with no password or username, but only from localhost"
Not foolproof, but as far as I know both this and mongo did the "bind to all ports AND allow access with no password". It was the combo that really killed it.
It's easier than that - just allow two setup options that generate different config files: "development (insecure)" and "production". First one is quick and easy and relatively more unsafe, second one is fully secured based on whatever best practices. "People will find and set all the configuration options correctly" is what got us into this mess.
I believe this is how Hashicorp's Vault does it.
That only works on a single server, so it's not an option for hdfs in anything but a toy configuration. The nodes in a hdfs cluster must be able to communicate securely, and the way you do that is to set up TLS between the nodes, with Kerberos as authentication, which is quite bothersome to get running.
Or if you feel you have to allow less secure defaults for initial setup (a distributed system needs to bind to something other than localhost so in those cases using that as the key won't work) make sure that a warning is displayed everywhere that an insecure default is still in operation, where "everywhere" includes to stdout when starting/refreshing services, or perhaps stderr, to log files, to the UI if you have one (on all displays not just those exposed to admin users).
The warnings don't stop the initial trial of a service, but are in-your-face enough that any self respecting admin will review the matter before making the service available on a public network (at least with real data and/or on machines/accounts with access to other securables).
Your users may request (in fact, some of them will demand) a setting that turns such warnings off without addressing the issues covered/ You might chose to include such a setting (at least then they make a conscious decision to not address the issues instead of later complaining your product was insecure without warning) or take the high ground and tell people who say "unless you let me get away with an insecure setup I'll use something else" that you;d rather they go use something else instead of potentially tarnishing your product by association when something goes wrong.
Of course like providing good documentation, this is extra work that you may find hard to get by your paymasters at crunch time...
The warnings don't stop the initial trial of a service, but are in-your-face enough that any self respecting admin will review the matter before making the service available on a public network (at least with real data and/or on machines/accounts with access to other securables).
Your users may request (in fact, some of them will demand) a setting that turns such warnings off without addressing the issues covered/ You might chose to include such a setting (at least then they make a conscious decision to not address the issues instead of later complaining your product was insecure without warning) or take the high ground and tell people who say "unless you let me get away with an insecure setup I'll use something else" that you;d rather they go use something else instead of potentially tarnishing your product by association when something goes wrong.
Of course like providing good documentation, this is extra work that you may find hard to get by your paymasters at crunch time...
I rather like the way elasticsearch handles this. If the server is only bound to 127.0.0.1 then anything goes. If the server is bound to a network interface then various setting must be configured to ensure it works well in a non-testing environment.
This is annoying when your development environment involves a VM or Docker container, though. You have to figure out how to "run production in development" just to start playing around.
But the idea is that you put the very same docker image to production. I guess most products come with an automated way to setup docker, including a decently secure setup (in this case).
Yeah, but I'm not trying to do development of a product; I'm trying to learn the stack and/or comparatively evaluate the software for fit for a use-case. When it insists on being configured to run in "JVM-eats-all-your-RAM production mode", even though I want to operate it in "stay out of my way because I'm running you as a background service on the puny Pentium-M Macbook I develop on" mode, I get a bit miffed.
And this is the same software that behaves perfectly reasonably when installed as a package directly on a Linux box; it's just when it thinks it's being used over the network because I'm running macOS and it needs to be in a Linux sandbox, that it starts acting all aggressive.
At least give me a flag to let me explicitly tell the software to relax!
And this is the same software that behaves perfectly reasonably when installed as a package directly on a Linux box; it's just when it thinks it's being used over the network because I'm running macOS and it needs to be in a Linux sandbox, that it starts acting all aggressive.
At least give me a flag to let me explicitly tell the software to relax!
I'm sorry that you're frustrated. Maybe the following will help?
There is such a flag for your use case[1]. Start the node in the Docker container with
I hope that helps!
Disclaimer: I work at Elastic; I welcome any and all feedback
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/...
[2]: https://github.com/elastic/elasticsearch/pull/23595
[3]: https://github.com/elastic/elasticsearch/pull/23598
There is such a flag for your use case[1]. Start the node in the Docker container with
discovery.type: single-node
I hope that helps!
Disclaimer: I work at Elastic; I welcome any and all feedback
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/...
[2]: https://github.com/elastic/elasticsearch/pull/23595
[3]: https://github.com/elastic/elasticsearch/pull/23598
This isn't an IoT home security camera bought by Joe Sixpack though, where that argument makes more sense. This is supposed to be installed and maintained by developers and/or IT professionals. Are there really so many who are that helpless that they can't use a config wizard? Eek.
There are apparently plenty of IT professionals who only know how to Google and copy/paste commands into a shell. The second that the process deviates from the script they're stumped.
Just look at the comments posted to any "how to install foo" blog post or browse around StackExchange, et al., for a few minutes. As an example, there are still tons of people who do things like "chmod 0777 ..." to "fix" permissions issues.
Just look at the comments posted to any "how to install foo" blog post or browse around StackExchange, et al., for a few minutes. As an example, there are still tons of people who do things like "chmod 0777 ..." to "fix" permissions issues.
What else are you expected to do with a product that you don't fully understand (since you're just evaluating it here), and the thing doesn't work and doesn't provide any sort of useful diagnostics?
Oh great, it's saying permission denied, but doesn't tell me where the permission was denied. Now I have to check all permissions everywhere to figure out what the problem is, once I figure out how the product works and where it might have permissions.
Oh great, it's saying permission denied, but doesn't tell me where the permission was denied. Now I have to check all permissions everywhere to figure out what the problem is, once I figure out how the product works and where it might have permissions.
That argument is fine for initial investigation into a product, when you just want it to work so you can poke the features to see if they are adequate and suitable.
But the problem is people doing it that way and putting the result into production without any sort of security review.
But the problem is people doing it that way and putting the result into production without any sort of security review.
[deleted]
I do sysadmin work almost exclusively. The scenario you're describing is, basically, the norm in my life.
I'd take at look at OS and application service logs, then break out the various flavors of diagnostic tools-- strace, ProcMon, etc. It's not ideal, but it's the sorry state of affairs in the industry.
Insecure-by-default isn't the answer to bad error messages.
I'd take at look at OS and application service logs, then break out the various flavors of diagnostic tools-- strace, ProcMon, etc. It's not ideal, but it's the sorry state of affairs in the industry.
Insecure-by-default isn't the answer to bad error messages.
You troubleshoot, investigate, and research -- what systems guys have done since the beginning of computing.
If you're a developer (not you, per se) running into these types of issues, well, now you understand why -- until recently -- there has traditionally been a separate operations group that was responsible for this type of stuff.
If you're a developer (not you, per se) running into these types of issues, well, now you understand why -- until recently -- there has traditionally been a separate operations group that was responsible for this type of stuff.
Wizard step 1: Enter your Kerberos server information...
Me: Fuck, now I've got to set up Kerberos and get all of the certs lined up before I can even see if this thing will work for us...
Step 2: Enter your CORBA...
Me: How much do we really want this solution...
Me: Fuck, now I've got to set up Kerberos and get all of the certs lined up before I can even see if this thing will work for us...
Step 2: Enter your CORBA...
Me: How much do we really want this solution...
Why not have a system that requires a shared secret? It's not difficult to set up and while not secure it's so much better than nothing.
My rule of thumb for adoption UX is: every step roughly halves the number of people who will complete the process.
So a 10-step install means that if a million people discover the app only about 900 will get it running.
As complexity and general noise increases this effect will become more pronounced. People just do not have time for time-consuming installs. The information age is the age of distraction and so if something takes a lot of time "plonk" into the trash it goes.
There are ways to secure things with decent UX. At the very least the defaults could allow direct access only from private IP ranges and require explicit configuration to set otherwise. This would prevent people from shooting themselves in the foot this way.
So a 10-step install means that if a million people discover the app only about 900 will get it running.
As complexity and general noise increases this effect will become more pronounced. People just do not have time for time-consuming installs. The information age is the age of distraction and so if something takes a lot of time "plonk" into the trash it goes.
There are ways to secure things with decent UX. At the very least the defaults could allow direct access only from private IP ranges and require explicit configuration to set otherwise. This would prevent people from shooting themselves in the foot this way.
> People just do not have time for time-consuming installs.
I have two quotes printed on the back of my business cards. One of those is: "If you don't the time to do it right, when will you have the time to do it over?"
I remember hearing something to the effect of "temporary setups never are" and, in all but the most disciplined environments, I've generally found that to hold true.
I have two quotes printed on the back of my business cards. One of those is: "If you don't the time to do it right, when will you have the time to do it over?"
I remember hearing something to the effect of "temporary setups never are" and, in all but the most disciplined environments, I've generally found that to hold true.
People dealing with masses of data should find it hard to make it accessible, security isn't easy, you should have to learn what you're doing.
But also there should be no way to get it working insecurely. Should just refuse to run until everything is secure, data limits, timeout limits, retry limits, ip limits, account limits, password complexity, etc.
But also there should be no way to get it working insecurely. Should just refuse to run until everything is secure, data limits, timeout limits, retry limits, ip limits, account limits, password complexity, etc.
The lesson of history is that people don't learn from history.
Lower the barrier of entry for use.
I don't necessarily agree with it, but having to screw around with authentication/certs/ssl when just playing/trying out a product is annoying.
Not everyone has the patience to deal with this.
I don't necessarily agree with it, but having to screw around with authentication/certs/ssl when just playing/trying out a product is annoying.
Not everyone has the patience to deal with this.
Securing an database at the software level is not necessarily ideal for many use cases.
Typically, I secure private resources via firewalls in the networking layer. A properly configured firewall with intelligent routing rules makes up for a lot of sins in both operating system and application security. If you have servers in a private subnet with a well defined gateway, application level security is just needless complication and overhead. Though admittedly, I am not a security expert.
Typically, I secure private resources via firewalls in the networking layer. A properly configured firewall with intelligent routing rules makes up for a lot of sins in both operating system and application security. If you have servers in a private subnet with a well defined gateway, application level security is just needless complication and overhead. Though admittedly, I am not a security expert.
Security is like an onion - you should have plenty of layers. You are describing perimeter security, which is great to have. If an intruder bypasses perimeter security, intrusion detection systems are the next line of defense. They are expensive/tough to setup. So, internal security (SSL everywhere or Kerberos with TLS/SSL for comms between
services) is a straightforward way to provide an additional level of security.
In many cases these systems are installed behind firewalls in corporate locations, where this is less of an issue. It becomes a bigger issue when you then just deploy it into the cloud without knowing what or how it works.
If your problem justifies a Hadoop cluster, you probably have the team that can properly set it up.
Insecure by default is not a big issue if it has its own dedicated network with restricted access in or out.
Insecure by default is not a big issue if it has its own dedicated network with restricted access in or out.
> If your problem justifies a Hadoop cluster, you probably have the team that can properly set it up.
95% of the installations of a Hadoop cluster do not have a problem that justifies a Hadoop cluster.
95% of the installations of a Hadoop cluster do not have a problem that justifies a Hadoop cluster.
problem is lots of people think they have a problem that justifies a Hadoop cluster, but they don't really need it; the whole "you don't have big data thing".
Hadoop is just another application that needs to talk to its peers to be useful. It shouldn't have to worry about any outside authentication, something else should handle that...
Maybe to reduce startup effort. For people putting together PoCs, security configuration fiddling is going to be a barrier to their using some software.
even if node-to-node communication in a cluster (hadoop or otherwise) itself is not secured, is it not reasonable to secure external access to the cluster itself (i.e. with a firewall)?
from an outsider perspective (I've never used/run hadoop) I cannot see much reason for exposing the cluster to the outside world - either a web-app acts as an intermediary or access can be provided via VPN/ssh-tunnel/etc
... just curious why a fully/publically exposed cluster would be a "requirement"? or does it come down to the fact that firewalling an AWS environment is as painful (if not more) than "kerberizing" a [hadoop] cluster? (I kind of assumed AWS has firewalling functionality that is fairly plug'n'play ... a quick search does really back that up though)
from an outsider perspective (I've never used/run hadoop) I cannot see much reason for exposing the cluster to the outside world - either a web-app acts as an intermediary or access can be provided via VPN/ssh-tunnel/etc
... just curious why a fully/publically exposed cluster would be a "requirement"? or does it come down to the fact that firewalling an AWS environment is as painful (if not more) than "kerberizing" a [hadoop] cluster? (I kind of assumed AWS has firewalling functionality that is fairly plug'n'play ... a quick search does really back that up though)
I used to work at a big data consulting company and dealt with hadoop clusters at a bunch of different companies. What you described was absolutely the norm. The entire cluster closed to the outside world, except for one gateway machine that allows ssh access, and anything within the cluster is totally open. Sometimes some web services were open to the company VPN.
Kerberizing is a pain but not usually needed. You're correct that AWS firewall rules are very easy.
What you're seeing in this article is the exception, people doing it totally wrong.
Kerberizing is a pain but not usually needed. You're correct that AWS firewall rules are very easy.
What you're seeing in this article is the exception, people doing it totally wrong.
In my experience these instances usually are either test/play clusters or just set up by people who don't know their way around Hadoop and its security features.
So: Yes, a Hadoop cluster should always be firewalled (secured or not) and have well defined access points via edge nodes as well as holes in the firewall for specific services that need to be exposed.
So: Yes, a Hadoop cluster should always be firewalled (secured or not) and have well defined access points via edge nodes as well as holes in the firewall for specific services that need to be exposed.
I knew it was a bad idea to post 'getting started' tutorials that skipped all the security steps and replace them with a 'probably don't wanna do it this way in production' (and usually no documentation on how one should do it)...
Not levelling this comment at HDFS solely but it's about time people stopped with the 'hello world' style examples.
Not levelling this comment at HDFS solely but it's about time people stopped with the 'hello world' style examples.
> it's about time people stopped with the 'hello world' style examples
Unfortunately not having the initial "this is the core, look how simple it can be" example a lot of people will turn off.
> (and usually no documentation on how one should do it)
That is the issue. Good documentation is often the issue for many other reasons too. For commercial projects time is money and so often not available in sufficient quantity (at least for externally facing documentation that may be held to standards that internal materials are not so needs extra resource for review+rework). For unfunded work (for instance things that start out as personal projects and/or quick proofs of concept, or many community-only driven works) documentation often gets left because it isn't the interesting/sexy part.
Unfortunately not having the initial "this is the core, look how simple it can be" example a lot of people will turn off.
> (and usually no documentation on how one should do it)
That is the issue. Good documentation is often the issue for many other reasons too. For commercial projects time is money and so often not available in sufficient quantity (at least for externally facing documentation that may be held to standards that internal materials are not so needs extra resource for review+rework). For unfunded work (for instance things that start out as personal projects and/or quick proofs of concept, or many community-only driven works) documentation often gets left because it isn't the interesting/sexy part.
[deleted]
Given the distributed nature, HDFS runs on multiple machines. In linux distributed service security fits well with kerberos. Normally if you want a "secure" HDFS you must "kerberize" the services such that any hadoop operation requires a valid/authorized TGT.
To most people kerberizing a hadoop cluster is a major barrier to getting hadoop running. I dont see this changing but certain vendor hadoop distros break down some of the barriers.
Sometimes it is OK if you run a cluster insecure. Please dont do it if youre handling my financial or medical records though. As Mr.T once said 'dont write checks that yo ass cant cash'