Ask HN: Companies using MongoDB, suggestions to deploy at scale?
64 comments
I really don't understand why RethinkDB, a seemingly superior tool, is not getting more traction. MongoDB was and still is a hack, a simple design that's prone to blow up.
What sorta turned me off about RethinkDB is that it totally relies on having a very complete driver to build up their AST on the client-side. There's no official .NET client, for instance, and the "community" one doesn't support arbitrary documents (you must define classes for every document type, it looks like).
The last time I checked using it in producing wasn't even recommended by the authors http://rethinkdb.com/stability/
Hm. They should call it webscale and be done with it.
1M records? Bro you are seriously scaling.! Good job!
Change your data model so that you don't need to aggregate. Duplicate if necessary, handle the resulting potential inconsistency.
Have the right indexes; especially with complex aggregation you might be surprised to find they aren't being used due to a quirk of your query.
A single mongo instance can handle a beefy load, but not if every read is an aggregation over a million by million by who knows how many docs.
Have the right indexes; especially with complex aggregation you might be surprised to find they aren't being used due to a quirk of your query.
A single mongo instance can handle a beefy load, but not if every read is an aggregation over a million by million by who knows how many docs.
I second the idea of denormalizing your data. It feels odd because that's exactly what you're trying to avoid with relational databases, but honestly, mongo is designed to handle denormalized data
http://blog.mongodb.org/post/88473035333/6-rules-of-thumb-fo...
Want to see other replies as well - but here are some of my thoughts.
1. Run your query with explain - to confirm that your queries are indeed indexed[1]. There have been a couple of times where we discovered that a specific branch of a complex query was not hitting index - so discovering that was useful.
2. One thought on slow reads with aggregation framework and 1M+ documents is that you are bottlenecked on memory. Even though all your queries may be indexed - you may be hitting disk too often (and doing a lot of random I/O) which is expensive. Check out MMS - and specifically the page faults graph - and you will get an idea about how many times a second you hit disk. For my specific application - response time was very important - so we tried to keep our page faults close to zero. Is increasing the RAM in your instance a viable option? If so this would definitely give you some breathing room.
3. You should try and verify if your response is slow because the query processing time itself is high, or if it's because you have several concurrent queries competing for system resources (CPU / memory etc). If it's the latter - the fix is easy - just add more machines to your replicaset (balance this with adding more RAM).
4. Check your lock %. My guess - from your brief description - this will not be an issue for you. If this is an issue - you __may__ need to shard your system - or do 5.
5. I think the most scalable solution would be to rethink your data pipeline. Why do you need to apply complex aggregation queries on 1M+ documents? Can you build a processing pipeline that takes in your data - and builds "prepared views" for the queries you expect out of your aggregation pipeline? In my application - we used stream processing pipeline that aggregated all the messages we got - in all the different ways we needed - and inserted them into different databases (to work around the DB level locking :( ) in mongo. Each collection had exactly ONE type of query that would be executed - this would be an extremely simple query and would be indexed. This pattern is common - even outside of mongo.
Good luck :)
[1] http://docs.mongodb.org/manual/reference/method/cursor.expla...
1. Run your query with explain - to confirm that your queries are indeed indexed[1]. There have been a couple of times where we discovered that a specific branch of a complex query was not hitting index - so discovering that was useful.
2. One thought on slow reads with aggregation framework and 1M+ documents is that you are bottlenecked on memory. Even though all your queries may be indexed - you may be hitting disk too often (and doing a lot of random I/O) which is expensive. Check out MMS - and specifically the page faults graph - and you will get an idea about how many times a second you hit disk. For my specific application - response time was very important - so we tried to keep our page faults close to zero. Is increasing the RAM in your instance a viable option? If so this would definitely give you some breathing room.
3. You should try and verify if your response is slow because the query processing time itself is high, or if it's because you have several concurrent queries competing for system resources (CPU / memory etc). If it's the latter - the fix is easy - just add more machines to your replicaset (balance this with adding more RAM).
4. Check your lock %. My guess - from your brief description - this will not be an issue for you. If this is an issue - you __may__ need to shard your system - or do 5.
5. I think the most scalable solution would be to rethink your data pipeline. Why do you need to apply complex aggregation queries on 1M+ documents? Can you build a processing pipeline that takes in your data - and builds "prepared views" for the queries you expect out of your aggregation pipeline? In my application - we used stream processing pipeline that aggregated all the messages we got - in all the different ways we needed - and inserted them into different databases (to work around the DB level locking :( ) in mongo. Each collection had exactly ONE type of query that would be executed - this would be an extremely simple query and would be indexed. This pattern is common - even outside of mongo.
Good luck :)
[1] http://docs.mongodb.org/manual/reference/method/cursor.expla...
In addition to verifying you are hitting an index for all of your queries, you should verify that all of your indexes are in memory. Depending on the size of your dataset, trying to fit more (or all) of it in memory may be an option worth trying.
[deleted]
We moved to PostgreSQL.
Our performance problems went away, with almost no work other than the initial migration. We fixed many data integrity problems during the migration, and have not noticed any problems since. Our uptime has become near perfect. We now get to use all of the other great features that PostgreSQL offers.
Our performance problems went away, with almost no work other than the initial migration. We fixed many data integrity problems during the migration, and have not noticed any problems since. Our uptime has become near perfect. We now get to use all of the other great features that PostgreSQL offers.
I am sad to see this answer moved so far down the page. I wish others would upvote it. I worked with MongoDB for 2 years, and more lately with PostGres. I feel like PostGres is underestimated. It is the only software in the world that offers such a powerful combination of SQL and NoSql. More people should try it.
We have thought about this. We are interfacing via Mongoose to MongoDB. Is there an easy way to still use Mongoose while transitioning to PostgreSQL?
Mongoose is such a brilliant library right? If I was writing something in nodejs - the existence of mongoose itself would justify simply going for a mongodb backend.
Unfortunately, Mongoose cannot query Postgres instead of Mongo. It's a wrapper around the native Node Mongo driver and does not incorporate a database abstraction layer.
[deleted]
The OP is asking for Mongo performance suggestions, not soliciting offers for other database systems. How big is your dataset? That's nice if you can fit everything on one machine, but have you ever set up a Postgres cluster?
The most effective and affordable fix for many scaling problems is to migrate to a different product or approach.
I'd have to check with our DB folks to get the exact size of our dataset and the number of servers we're using now with PostgreSQL. At the time of the migration, it was around 85 TB, with 6 physical servers and replication being used.
I'd have to check with our DB folks to get the exact size of our dataset and the number of servers we're using now with PostgreSQL. At the time of the migration, it was around 85 TB, with 6 physical servers and replication being used.
I'd be interested to know more about setting up clusters with Postgres. Could you explain some of the issues, perhaps some general approaches people take? Is the tooling for it any good, or is the tooling the main issue?
I generally prefer relational databases, and from a theory point of view they feel much nicer and have the potential to be much faster, however, one thing MongoDB has got right is the simple primitives for scaling, and having them built into the core product, something I don't think Postgres has.
I think there's a lot that Postgres (and other relational databases) can learn from MongoDB (ease of sharding perhaps?), Cassandra (consistent hashing maybe?), HyperDex, etc. And I'm sure there's much that those products can learn from the relational world as well.
I generally prefer relational databases, and from a theory point of view they feel much nicer and have the potential to be much faster, however, one thing MongoDB has got right is the simple primitives for scaling, and having them built into the core product, something I don't think Postgres has.
I think there's a lot that Postgres (and other relational databases) can learn from MongoDB (ease of sharding perhaps?), Cassandra (consistent hashing maybe?), HyperDex, etc. And I'm sure there's much that those products can learn from the relational world as well.
The problem with PostgreSQL clustering, and really all SQL clustering is that the DB engine is just not designed with that in mind. So what you end up having to do is add an additional layer to support the clustering (or sharding). The advantage that something like MongoDB, or CouchDB, or Elasticsearch (etc) is that they're designed with sharding in mind. This makes the sysadmin's life a lot easier! Scaling Mongo up might be a pain with their config servers, routing servers, etc but at least there's a well-documented and well-supported system for scaling out horizontally.
MySQL/PostgreSQL are fantastic database systems and exceptional at solving the problems they were designed for. But its wrong to think of it as this blanket solution for every problem, or to suggest it on HN every single time someone asks a question about MongoDB.
MySQL/PostgreSQL are fantastic database systems and exceptional at solving the problems they were designed for. But its wrong to think of it as this blanket solution for every problem, or to suggest it on HN every single time someone asks a question about MongoDB.
Relational databases provide a very solid foundation upon which a wide variety of replication, partitioning and data distribution schemes can be constructed.
Just look at the different approaches that PostgreSQL supports: http://www.postgresql.org/docs/9.4/static/different-replicat...
Other relational DB systems, from Oracle, to SQL Server, and even MySQL, offer similar functionality.
Practice shows that they're very capable of successfully handling many different scenarios, even if some of these scenarios hadn't been considered decades ago when these relational database systems were first implemented. But that's just what happens when the core building blocks they do provide are so solid and fundamentally sound.
It's only anecdotal evidence, but of all of the DB professionals I've worked with over the years, the ones who complained the least were always the ones working with some sort of relational DB system. The ones who complained the most have been working with some NoSQL database system.
I've heard nothing but praise for the PostgreSQL documentation, while we repeatedly see submissions like this one asking for help with NoSQL scalability problems that just don't happen when using PostgreSQL, or Oracle, or SQL Server, or DB2, or MySQL, or even SQLite!
Just look at the different approaches that PostgreSQL supports: http://www.postgresql.org/docs/9.4/static/different-replicat...
Other relational DB systems, from Oracle, to SQL Server, and even MySQL, offer similar functionality.
Practice shows that they're very capable of successfully handling many different scenarios, even if some of these scenarios hadn't been considered decades ago when these relational database systems were first implemented. But that's just what happens when the core building blocks they do provide are so solid and fundamentally sound.
It's only anecdotal evidence, but of all of the DB professionals I've worked with over the years, the ones who complained the least were always the ones working with some sort of relational DB system. The ones who complained the most have been working with some NoSQL database system.
I've heard nothing but praise for the PostgreSQL documentation, while we repeatedly see submissions like this one asking for help with NoSQL scalability problems that just don't happen when using PostgreSQL, or Oracle, or SQL Server, or DB2, or MySQL, or even SQLite!
>all SQL clustering
Uh that's simply not true? VoltDB, for instance, easily and natively supports sharding. SQL Server's AlwaysOn system (despite being failover clustering) is incredibly simple to setup and is transparent to clients.
The OP is asking about a problem with only 1M records. Unless they're huge records or something, no one should be discussing such a small number of records as a performance limit.
Uh that's simply not true? VoltDB, for instance, easily and natively supports sharding. SQL Server's AlwaysOn system (despite being failover clustering) is incredibly simple to setup and is transparent to clients.
The OP is asking about a problem with only 1M records. Unless they're huge records or something, no one should be discussing such a small number of records as a performance limit.
Hell, for only a million records, I'd "scale" with SQLite...
Sorry but that simply isn't true, and I suspect you've not really done any serious RDBMS at scale - Teradata, DB/2, Oracle et al have had clustering baked in since the 90s, if not before. If you are talking about MySQL then sure.
I'd worked on a couple of Mongo DB projects a year and half ago. It could not scale for performance, it was supposed to be fast datastore for simple lookups and heavy updating (multiple 10s of updates/sec - spiking to hundreds of updates) (no matter what we did), eventually had to switch out to other tools (either inbuilt custom ones or off the shelf).
I won't claim that we did everything under the sun to optimize it, we could have probably optimized it to work eventually. It was just easier to switch and not deal with the headaches of maintaining/optimizing the mongoDB setup after days/weeks of headaches.
I won't claim that we did everything under the sun to optimize it, we could have probably optimized it to work eventually. It was just easier to switch and not deal with the headaches of maintaining/optimizing the mongoDB setup after days/weeks of headaches.
>heavy updating (multiple 10s of updates/sec - spiking to hundreds of updates)
In what world is 100 updates/sec "heavy"? I don't mean to come off rude, but even with a full disk write per update that fits into normal magnetic disk IOPS without even batching transactions.
For comparison, with zero work on our end, and running off a 2-disk RAID 1 array with a 10 or 15K HDD, it was trivial to pass 10K/sec updates.
In what world is 100 updates/sec "heavy"? I don't mean to come off rude, but even with a full disk write per update that fits into normal magnetic disk IOPS without even batching transactions.
For comparison, with zero work on our end, and running off a 2-disk RAID 1 array with a 10 or 15K HDD, it was trivial to pass 10K/sec updates.
It is heavy for MongoDB.
It's a weird phenomenon you see these days, people talk about 10 updates a second on a million records as some sort of intense load that needs "sharding" and God knows what else, when a DBA from 20 years ago wouldn't even blink at it.
It's a weird phenomenon you see these days, people talk about 10 updates a second on a million records as some sort of intense load that needs "sharding" and God knows what else, when a DBA from 20 years ago wouldn't even blink at it.
I'd agree that MongoDB has its fair share of performance issues, but there are many companies using it at scale in production. Trello comes to mind as a good example of a high traffic web application that is doing pretty well on it so far from what I've read.
This suggests there are things that can be done to improve performance enough, and in that case, maybe switching would be a bad idea. There is certainly a very high cost in terms of development time, learning and dev-ops to switching between two databases.
This suggests there are things that can be done to improve performance enough, and in that case, maybe switching would be a bad idea. There is certainly a very high cost in terms of development time, learning and dev-ops to switching between two databases.
I work at Trello. I believe we're seeing ~1000 updates/sec and ~10000 reads/sec at peak. That said, our data is split across 7 shards. Total documents count is in the low billions.
well by asking this question to mongo users you miss the audience of the people successfully scaling up -for less money- that may belongs to the people NOT using mongo.
Business is about costs and prices not about thinking that business is about the religious adoptions of mongo, agile, C#, python, linux ....
This question is saying that your business is ruled by the techs not anyone with the common sense needed to pay the bills (thus your paychecks).
The feedbacks I have to give you is that TCO per mb stored increase more than linearly with mongodb, with weiredly enough a decrease in SLA (due to the weired unexpected incident hard to troubleshoot) so if you look at the probability you have success, it is bad news. Customer expect cost to diminish with volume, thus prices (investors too). If the fat of your business model is around document storage, you have a problem.
You should not trust me, and do some real costs analysis of the cost per documents for your business case with more than one scenarii and make some costs analysis and some projections. Because even if someone else optimization worked for them, it may not be relevant to your business case: at one point you make money, so you issue a transaction that cannot be handled in a distributed system (CAP theorem + NoSQL==NoACID) and this point differs than fellow devs.
Mongo could be -not is- killing your business. If you do mongo because that is the only techno you know, consider that you have been taken hostage, and call SWAT to have your mongo guru arrested. They may look nice and harmless, but you are the hostages of dangerous fanatic religious techs. Maybe call an exorcist, it could be as efficient.
And then go back to thinking what is the most efficient way to have your bills paid, and how to diminish cost as your business grow. Forget technical stuff as a compass for making successful software solution. Success is about benefits only
Je dis ça, je dis rien, hein?! My .02 eurocents.
Business is about costs and prices not about thinking that business is about the religious adoptions of mongo, agile, C#, python, linux ....
This question is saying that your business is ruled by the techs not anyone with the common sense needed to pay the bills (thus your paychecks).
The feedbacks I have to give you is that TCO per mb stored increase more than linearly with mongodb, with weiredly enough a decrease in SLA (due to the weired unexpected incident hard to troubleshoot) so if you look at the probability you have success, it is bad news. Customer expect cost to diminish with volume, thus prices (investors too). If the fat of your business model is around document storage, you have a problem.
You should not trust me, and do some real costs analysis of the cost per documents for your business case with more than one scenarii and make some costs analysis and some projections. Because even if someone else optimization worked for them, it may not be relevant to your business case: at one point you make money, so you issue a transaction that cannot be handled in a distributed system (CAP theorem + NoSQL==NoACID) and this point differs than fellow devs.
Mongo could be -not is- killing your business. If you do mongo because that is the only techno you know, consider that you have been taken hostage, and call SWAT to have your mongo guru arrested. They may look nice and harmless, but you are the hostages of dangerous fanatic religious techs. Maybe call an exorcist, it could be as efficient.
And then go back to thinking what is the most efficient way to have your bills paid, and how to diminish cost as your business grow. Forget technical stuff as a compass for making successful software solution. Success is about benefits only
Je dis ça, je dis rien, hein?! My .02 eurocents.
This is an unbelievable waste of a comment and the OP should not even consider this as feedback.
You completely ignore the question, make a comment about "religious adoptions of mongo" and then do an about-face by showing your OWN religious zeal AGAINST mongo.
Mongo is not killing his business. Your comment should be deleted.
To the OPs question: can you post any more specifics? 1M documents with an index shouldn't be an issue for the aggregation pipeline. What are the specs of the machine? What are your indexes? I think it would be tough to help without having that info, and sadly a question requiring that type of info may be best suited for something like Stack Overflow.
You completely ignore the question, make a comment about "religious adoptions of mongo" and then do an about-face by showing your OWN religious zeal AGAINST mongo.
Mongo is not killing his business. Your comment should be deleted.
To the OPs question: can you post any more specifics? 1M documents with an index shouldn't be an issue for the aggregation pipeline. What are the specs of the machine? What are your indexes? I think it would be tough to help without having that info, and sadly a question requiring that type of info may be best suited for something like Stack Overflow.
I mean this as honest feedback, and please don't take it as a personal attack, but I think you are being unreasonably critical of the comment, and I suggest you take time to consider why that is.
The comment raised an interesting way of thinking about the problem that many developers would not consider, and while it might have had a fair bit of religious zeal, that is very common here and not worthy of comments like "unbelievable waste of a comment" and "your comment should be deleted".
The comment raised an interesting way of thinking about the problem that many developers would not consider, and while it might have had a fair bit of religious zeal, that is very common here and not worthy of comments like "unbelievable waste of a comment" and "your comment should be deleted".
Aye yai yai. Ya know, your response is part of the problem here. This may be a general shift in the way this site is going, and that may be fine, but what I expect from the comments in HN posts is far greater than what SFjulie1 posted. It's incoherent, off-topic, and filled with the anti-{%database%} mentality prevalent when something about MongoDB comes up.
The comment raised an interesting way of thinking about the
problem that many developers would not consider
The comment was not 'incoherent', to say so suggests that the actual points made no sense. Rather, the comment was simply written in poor English.
I think your comment only needed to be a polite "can we keep the conversation on the topic of MongoDB scaling" rather than a quite personal attack on the author.
Also, personally, I thought the main point raised about viewing the database choice from a business rather than development point of view was interesting and well worth discussion.
I think your comment only needed to be a polite "can we keep the conversation on the topic of MongoDB scaling" rather than a quite personal attack on the author.
Also, personally, I thought the main point raised about viewing the database choice from a business rather than development point of view was interesting and well worth discussion.
Why can we not have an actual discussion about scaling MongoDB when the post is about scaling MongoDB!
Because it might be more helpful to think about scaling your data needs than scaling with only a certain product.
Because it might be more helpful to think about scaling your data needs than scaling with only a certain product.
Here's why: SFJulie1's comment came off as rambling and somewhat incoherent, if not a little childish and rude. In the past few hours one other account has popped up here ( https://news.ycombinator.com/threads?id=Ronaldo777 ) posting equally trollish comments in this same specific thread...
Just to give you an idea where SFJulie1 is coming from:
http://www.reddit.com/r/programming/comments/2pwkcl/help_nee...
I think mikegioia was perfectly reasonable in calling out this comment, it is trash and he was right to recognize it as such.
Just to give you an idea where SFJulie1 is coming from:
http://www.reddit.com/r/programming/comments/2pwkcl/help_nee...
I think mikegioia was perfectly reasonable in calling out this comment, it is trash and he was right to recognize it as such.
I don't know, it may not address the OP's specific question but it may, in a very circuitous way, address a need the OP doesn't recognize he has.
It is hardly arguable that there is a segment of the developer population that has a religious zeal for their "stack" or against others (just pick any random topic on functional programming or PHP for examples of each). It is entirely appropriate to discuss and address that in topics like this on HN, in my opinion.
It is hardly arguable that there is a segment of the developer population that has a religious zeal for their "stack" or against others (just pick any random topic on functional programming or PHP for examples of each). It is entirely appropriate to discuss and address that in topics like this on HN, in my opinion.
this religious thing works the other way, too. e.g. i, after reading a couple dozen articles about mongo across the years, most of them linked from this site, have a religious zeal against mongo.
That may well be the case. HN has a community of zealots for just about any programming or tech related thing (pro-Apple, anti-PHP, anti-C++, pro-functional, pro-javascript, you name it). That doesn't mean criticism (or support) automatically comes from a place of zealotry. In this particular case the parent comment in this thread didn't read at all like religious zealotry against mongoDB to me.
I suppose one might infer from it notion of a default assumption that developers are all zealots about their particular technology choices. I'm not sure such an assumption itself is terribly bad to have, honestly. Sometimes I think that if my only insight into the world of technology came from startup job postings and HN, I ought to be forgiven for thinking that the entire industry is a kind of elaborate and humorous play on role-playing game mechanics ("Ruby Dev" <==> "Developer/Ruby" class/skill combo), which are notorious for producing fanatical devotees of one combination or another.
I suppose one might infer from it notion of a default assumption that developers are all zealots about their particular technology choices. I'm not sure such an assumption itself is terribly bad to have, honestly. Sometimes I think that if my only insight into the world of technology came from startup job postings and HN, I ought to be forgiven for thinking that the entire industry is a kind of elaborate and humorous play on role-playing game mechanics ("Ruby Dev" <==> "Developer/Ruby" class/skill combo), which are notorious for producing fanatical devotees of one combination or another.
A database that had a server-wide lock (and highly trumpeted when they got the feature of only being database-wide) and had the amazingly in-depth IO strategy of mmap, defaulted to losing people's data, silently destroyed data on 32-bit, had a Java client that shipped with "catch { if rnd() > 0.1 then .... }", and spends a lot on money trying to be "webscale"?
Perhaps it's a deserved attitude.
Perhaps it's a deserved attitude.
SFjulie1's comment is a good one, mikegioia.
Taking a look at the economics of a technology that's being used is a very sensible thing to do, especially for a business. If this analysis shows that one solution is inferior to another, it does not mean that the analyst has a "religious zeal" against the economically-inferior technology.
Taking a look at the economics of a technology that's being used is a very sensible thing to do, especially for a business. If this analysis shows that one solution is inferior to another, it does not mean that the analyst has a "religious zeal" against the economically-inferior technology.
My entire point in this entire thread is to for once keep the conversation on topic about MongoDB without everyone going into a tangent on switching to Postgres or something else. I just want one post where we can actually talk about MongoDB scaling.
Somebody came here asking for help with a real world problem.
It would be doing this person a disservice to not mention alternative technologies that have solved very similar problems that have affected many other people/organizations.
It is total absurdity to avoid bringing up these solutions to the problem merely in order to maintain some vague purity of the discussion, or to otherwise keep the discussion unnecessarily and unhelpfully focused.
It would be doing this person a disservice to not mention alternative technologies that have solved very similar problems that have affected many other people/organizations.
It is total absurdity to avoid bringing up these solutions to the problem merely in order to maintain some vague purity of the discussion, or to otherwise keep the discussion unnecessarily and unhelpfully focused.
I feel like you're just ignoring what I'm saying. It is NOT helpful to suggest alternative technologies all the time, especially in this specific post.
If I asked on a forum for help with the engine on my Dodge Dart, and you posted "Just buy a Prius", that would not be helpful. Do you see how that isn't helpful? That's what I'm trying to say, that I would like to see, for once a post stay on topic when it's about MongoDB (heck, even PHP for that matter!).
If I asked on a forum for help with the engine on my Dodge Dart, and you posted "Just buy a Prius", that would not be helpful. Do you see how that isn't helpful? That's what I'm trying to say, that I would like to see, for once a post stay on topic when it's about MongoDB (heck, even PHP for that matter!).
If using a Prius instead of a Dart would very likely solve the problem(s) being experienced, then it is a perfectly reasonable thing to suggest. It would be wrong not to suggest it.
This submission is about a problem involving MongoDB. It is not about MongoDB itself. The submitter clearly wants a variety of ideas here. That's why the submission says, and I quote, "Any tips, stories, suggestions are welcome!"
"Any" means "any", including the suggestion that some other technology be used.
This submission is about a problem involving MongoDB. It is not about MongoDB itself. The submitter clearly wants a variety of ideas here. That's why the submission says, and I quote, "Any tips, stories, suggestions are welcome!"
"Any" means "any", including the suggestion that some other technology be used.
Do you see his reply below? Do you still think his comment was a good one? Sometimes I can't believe the people commenting here. This guy is a self-proclaimed troll that you are defending!
And I must add that I also point that given a good technical solution for a business case A, it may not map correctly to another business case.
It is always a question of context. So I do think the question is lacking of business context.
The question asked is a if there is a silver bullet. The problem is optimization are always context related. You can even improve a software's sclalabity by changing your processes (who needs concurrencial access to a database if your process can ensure one and only one person is enough to do a certain provisioning that is Single Point of Performance loss?).
I mean, my life as a dev would be easier if managers where not asking for cost reductions and forcing technical solutions that increase costs and then convocating us because the company is having unexpected loss, and thus our wages have to be cut (or more often we have to do up to 50% extra hours coding for free) (and their wages are not). So yes, I am bitter, and unnice.
And even if you would be posting the "right question" here; then we would conclude that you are asking for a free audit, and also probably leaking business secrets (yep, btw, if you are on a competitive market, the concurrence and your customers now know you have a problem with scaling, and how you take costly decisions without caring about your costs... nice).
So with a brilliant reasoning ab absurdum, I can thus demonstrate that neither questions (the mongo oriented or the business oriented) are good questions to ask on HN.
It should be an heuristic on HN, and there maybe should a howto ask the right question like this one http://catb.org/~esr/faqs/smart-questions.html
Remember you are basically asking skilled people to do part of your job for free while aiming for a lot of money (I still wish you to be successful). It is quite logically itching poorly paid (or delusive (my case)) skilled coders.
And you are also leaking insider's information about your technology AND (lack of) business processes. And you may have future (and ex) cyberterrorists(joke)/concurrent/customers/partners/investors/employees here.
And you probably think security is about your 128 character gibberish SSO password OAUTH2 compliant chain you are using. Well most successful attacks are using social engineering, so now I know that I can hack you by proposing you a free software with a trojan aiming at solving your problem (using native active record so that it is not a first order backdoor). Or phone you as a contractor, and being paid to have a look at your documents because I know you are technologically a sitting duck.
I mean: seriously?
Advice #2: DON'T FEED THE TROLL (TL;DR) Yep, if you don't read until the end, you will not notice that I am serious, but choose deliberately a provocative tone because, I have fun, time, and I see this as the price to pay for my free consulting. I take the freedom to express my views in the formalism I choose. My intention is to see what will be the arbitration made by HN (and readers) between business (what I am saying is valid) and formalism (I say it in a clearly non consensual). So I intend to push my reasoning to the extreme just for measuring what is more important: business or being nice?
It is always a question of context. So I do think the question is lacking of business context.
The question asked is a if there is a silver bullet. The problem is optimization are always context related. You can even improve a software's sclalabity by changing your processes (who needs concurrencial access to a database if your process can ensure one and only one person is enough to do a certain provisioning that is Single Point of Performance loss?).
I mean, my life as a dev would be easier if managers where not asking for cost reductions and forcing technical solutions that increase costs and then convocating us because the company is having unexpected loss, and thus our wages have to be cut (or more often we have to do up to 50% extra hours coding for free) (and their wages are not). So yes, I am bitter, and unnice.
And even if you would be posting the "right question" here; then we would conclude that you are asking for a free audit, and also probably leaking business secrets (yep, btw, if you are on a competitive market, the concurrence and your customers now know you have a problem with scaling, and how you take costly decisions without caring about your costs... nice).
So with a brilliant reasoning ab absurdum, I can thus demonstrate that neither questions (the mongo oriented or the business oriented) are good questions to ask on HN.
It should be an heuristic on HN, and there maybe should a howto ask the right question like this one http://catb.org/~esr/faqs/smart-questions.html
Remember you are basically asking skilled people to do part of your job for free while aiming for a lot of money (I still wish you to be successful). It is quite logically itching poorly paid (or delusive (my case)) skilled coders.
And you are also leaking insider's information about your technology AND (lack of) business processes. And you may have future (and ex) cyberterrorists(joke)/concurrent/customers/partners/investors/employees here.
And you probably think security is about your 128 character gibberish SSO password OAUTH2 compliant chain you are using. Well most successful attacks are using social engineering, so now I know that I can hack you by proposing you a free software with a trojan aiming at solving your problem (using native active record so that it is not a first order backdoor). Or phone you as a contractor, and being paid to have a look at your documents because I know you are technologically a sitting duck.
I mean: seriously?
Advice #2: DON'T FEED THE TROLL (TL;DR) Yep, if you don't read until the end, you will not notice that I am serious, but choose deliberately a provocative tone because, I have fun, time, and I see this as the price to pay for my free consulting. I take the freedom to express my views in the formalism I choose. My intention is to see what will be the arbitration made by HN (and readers) between business (what I am saying is valid) and formalism (I say it in a clearly non consensual). So I intend to push my reasoning to the extreme just for measuring what is more important: business or being nice?
SFjulie1(1)
This is an interesting way of looking at the problem - as a 'TCO' of data stored. It's not one that I generally go to, but perhaps I should more.
I think the argument against it is that developer time and productivity is worth a considerable amount for many companies, and working with tools (DBMSs in this case, but it applies to lots more) that speed up application development is often worth more to the business than the basic cost of storing and serving data.
This isn't always the case, at Facebook scale for example, it might be that the most efficient use of money is to use (speaking entirely hypothetically here) is an Oracle database instead of MySQL, and instead hire twice as many developers because they are half as productive with it. Maybe that would be a good trade-off? However, at many (most?) companies, I suspect that the better trade-off would be to hire fewer developers and use tools that are more expensive to run.
I think the argument against it is that developer time and productivity is worth a considerable amount for many companies, and working with tools (DBMSs in this case, but it applies to lots more) that speed up application development is often worth more to the business than the basic cost of storing and serving data.
This isn't always the case, at Facebook scale for example, it might be that the most efficient use of money is to use (speaking entirely hypothetically here) is an Oracle database instead of MySQL, and instead hire twice as many developers because they are half as productive with it. Maybe that would be a good trade-off? However, at many (most?) companies, I suspect that the better trade-off would be to hire fewer developers and use tools that are more expensive to run.
There's the risk tolerance factor as well.
If product X is free, but requires hard to find skilled operators and doesn't have well established operational best practices, paying über dollars for Oracle might be the smart move.
Where I work, location and policies for hiring contract staff make a product like Mongo high risk for data we care about.
If product X is free, but requires hard to find skilled operators and doesn't have well established operational best practices, paying über dollars for Oracle might be the smart move.
Where I work, location and policies for hiring contract staff make a product like Mongo high risk for data we care about.
And after that pay big support fees to Oracle.
You're being trolled by SFJulie1.
1m records? We have 25 million+ in normalized MySql tables with proper indexing and run 20ms queries on modest Xeon E5-1650, 12gig ram dedicated to db, Samsung 840 Pro.
Seems like a drop in the bucket to be experiencing performance problems... What are your use cases? Why did you pick MongoDB i the first place?
Seems like a drop in the bucket to be experiencing performance problems... What are your use cases? Why did you pick MongoDB i the first place?
Heck, I have a SQLite dB I'm working with right now that has ~30 million rows across a few dozen tables. Queries against the indexed fields are returning in 130ms on my rMBP.
We've had slowness on https://Clara.io because of what we believe was excessive file system fragmentation as we were running on zfs and then later brtfs. We were on zfs and brtfs because it allowed for easy and quick backups via snapshots. But eventually writes were slowed leading to long lock times and making everything basically unworkable. A reinstall would always fix the issue for a while. We believe that the same reasons that backups were fast, the copy on write feature, is the same reason we got excessive fragmentation after a while.
I believe it was because we were not using xfs as our file system, which we have recently moved over to. So far the issue hasn't re-occured.
Other than this issue, MongoDB has been pretty good. Indices are important and formulating queries so that they are index based is also important (the same result can be obtained in multiple ways, some of which likely do not use the indices, while other formulations will use them.)
We use Redis as a cache in front to reduce the unnecessary repetitive load on MongoDB for common operations.
We have MongoDB cluster setup and it has saved us a number of times. Also it is really just a live backup system -- we've had a few hardware failures and we still have never had to restore for an archive backup, we just resync from one of the failover servers. But using that is likely a no-brainer, and you are likely already doing it.
I believe it was because we were not using xfs as our file system, which we have recently moved over to. So far the issue hasn't re-occured.
Other than this issue, MongoDB has been pretty good. Indices are important and formulating queries so that they are index based is also important (the same result can be obtained in multiple ways, some of which likely do not use the indices, while other formulations will use them.)
We use Redis as a cache in front to reduce the unnecessary repetitive load on MongoDB for common operations.
We have MongoDB cluster setup and it has saved us a number of times. Also it is really just a live backup system -- we've had a few hardware failures and we still have never had to restore for an archive backup, we just resync from one of the failover servers. But using that is likely a no-brainer, and you are likely already doing it.
We had similar issues with MongoDB on btrfs. While the snapshot features were nice, the write performance was abysmal. We moved to xfs at some point and got a massive increase in write throughput.
So far I've worked at two separate companies deploying MongoDB into production and we've never had any real issues, mostly because in both cases the lead engineers really knew what they were doing, had read the manual front-to-back, and they knew exactly what kind of OS and filesystem setup to use for maximum effectiveness.
So far I've worked at two separate companies deploying MongoDB into production and we've never had any real issues, mostly because in both cases the lead engineers really knew what they were doing, had read the manual front-to-back, and they knew exactly what kind of OS and filesystem setup to use for maximum effectiveness.
Definitely take a look at Mongo's recommendations:
http://docs.mongodb.org/manual/administration/monitoring/#di...
This has gotten much better since I last used mongo heavily. I ended up using the slow query log, but it looks like the tooling has improved significantly and now there is a list of common problems
There are ways to get around the latency you're seeing without (or while) moving away from Mongo. If you rewrite your queries as mapreduce operations, then you can run them at a scheduled interval to update a reports table with the previous interval's data. This is called incremental mapreduce and Mongo has a great guide on how to get started: http://docs.mongodb.org/manual/tutorial/perform-incremental-...
You won't get real time data any more, but switching to batch processing will speed up your users' experience.
The cynic in me, however, wants to just tell you that you've outgrown Mongo and should move on. It happens; not every database is designed to support every need, and Mongo happens to be a spectacularly shitty fit for the kind of OLAP scenario you're describing.
If you absolutely must have real-time analytics crunched on the fly, I recommend that you look at data stores that are a better fit for your application. InfluxDB is great for time series data. Aerospike is getting a lot of press in the AdTech industry. And old standbys like MySQL or Postgres can handle queries over a million records without breaking a sweat.
You won't get real time data any more, but switching to batch processing will speed up your users' experience.
The cynic in me, however, wants to just tell you that you've outgrown Mongo and should move on. It happens; not every database is designed to support every need, and Mongo happens to be a spectacularly shitty fit for the kind of OLAP scenario you're describing.
If you absolutely must have real-time analytics crunched on the fly, I recommend that you look at data stores that are a better fit for your application. InfluxDB is great for time series data. Aerospike is getting a lot of press in the AdTech industry. And old standbys like MySQL or Postgres can handle queries over a million records without breaking a sweat.
[deleted](1)
Use TokuMX (I don't have a stake in TokuMX, I'm just a happy customer). It's mostly a drop-in for Mongo 2.6 (though converting data is a downtime operation). To your point specifically, we're running aggregation stuff on collections with >1 billion documents and it hums along great. Massively better disk usage, clustering keys, ACID guarantees, document-level locking, tunable compression, and a bunch of other small improvements make it a really excellent choice.
TokuMX does have one significant problem in that the technology they use for indexes experiences performance degradation over time if you do lots of deletes from the head of the collection (ie, if you're processing data that gets inserted and deleted roughly chronologically), but it is fixable with reIndex() (which is an async operation analogous to OPTIMIZE TABLE). That's really the only large problem we've had with it, and it's surmountable with regular maintenance. In the case of monotonically-ordered data though, you can address the problem with partitioned collections, which lets you split a collection up into chunks, and then drop chunks as they become unnecessary, without having to do expensive things to the whole collection.
It looks like WiredTiger should provide many of the same benefits, so you may evaluate that first before taking the plunge, but I haven't personally compared it with our tokumx setup yet.
Outside of that, you probably need to start sharding your dataset. Figure out a shard key that will let you distribute your data well, and start splitting your load among multiple replica sets. Figure out if your issues are IO, CPU, or whatever - there's a lot of documentation on how to do this, so I won't rehash it here, but in general, you first need to understand what is slow before you can fix it. Chances are very good that your culprit is disk IO, though (because it always is in databases), and you can probably get some of that performance back with more RAM (to keep more of your working set in RAM at a given point), faster disks, and/or a striped disk array. Depending on your platform, some kernel tuning may help, as well, though it's going to be specific to your platform and setup.
You might also look at other tools and see if they're better-suited for your purposes. We use Elasticsearch as our ad hoc query layer on top of TokuMX, and are doing some stuff with Cassandra and BigQuery for bulk analysis of historical data. I'm not suggesting that you replace your whole stack, but if you're trying to shove a square peg through a round hole, you're just going to be frustrated. Scaling is all about specializing your toolset to fit your problem domain - there is almost never a single solution that is the best thing for all use cases. If you need to add another tool to the stack to offload one part of your requirements, do it!
TokuMX does have one significant problem in that the technology they use for indexes experiences performance degradation over time if you do lots of deletes from the head of the collection (ie, if you're processing data that gets inserted and deleted roughly chronologically), but it is fixable with reIndex() (which is an async operation analogous to OPTIMIZE TABLE). That's really the only large problem we've had with it, and it's surmountable with regular maintenance. In the case of monotonically-ordered data though, you can address the problem with partitioned collections, which lets you split a collection up into chunks, and then drop chunks as they become unnecessary, without having to do expensive things to the whole collection.
It looks like WiredTiger should provide many of the same benefits, so you may evaluate that first before taking the plunge, but I haven't personally compared it with our tokumx setup yet.
Outside of that, you probably need to start sharding your dataset. Figure out a shard key that will let you distribute your data well, and start splitting your load among multiple replica sets. Figure out if your issues are IO, CPU, or whatever - there's a lot of documentation on how to do this, so I won't rehash it here, but in general, you first need to understand what is slow before you can fix it. Chances are very good that your culprit is disk IO, though (because it always is in databases), and you can probably get some of that performance back with more RAM (to keep more of your working set in RAM at a given point), faster disks, and/or a striped disk array. Depending on your platform, some kernel tuning may help, as well, though it's going to be specific to your platform and setup.
You might also look at other tools and see if they're better-suited for your purposes. We use Elasticsearch as our ad hoc query layer on top of TokuMX, and are doing some stuff with Cassandra and BigQuery for bulk analysis of historical data. I'm not suggesting that you replace your whole stack, but if you're trying to shove a square peg through a round hole, you're just going to be frustrated. Scaling is all about specializing your toolset to fit your problem domain - there is almost never a single solution that is the best thing for all use cases. If you need to add another tool to the stack to offload one part of your requirements, do it!
People who are currently using MongoDB in production, what kinds of performance optimizations did you have to make in order to deploy it in scale? Any tips, stories, suggestions are welcome!