Unix sort is a merge sort, using fixed RAM
uniq -c is line by line
sed is line by line
A superior solution in every way
If you were going to do this in Python (or another similar language), you need to write a sort function that operates on a file, not a Python collection, as the collection is always bound by RAM, which would at best be re-implementing the sort command. Once you can sort a file, everything else is trivial, as counts can be done line by line, or more siply, via uniq -c.
Of course, if you only care about things that fit in memory, you can do it in Python, but it is still far easier to use command line for these type of problems.
Our use case goes like this: a member database of over a 100M records, and a number of content databases each with tens or hundreds of millions of records (reviews, video, lists, wiki, this, that , the other thing, one content table with over a billion records, where all the content records had a member id. Our primary usage pattern is to grab a set of content records (say 10-200 at a time) with their member information.
Putting everything in one database and doing the join there does not scale for us, and severely reduces flexibility. We would need to continue to scale up our hardware to handle the sum of the content sets, and new content sets are being created on a regular basis. By putting these all into different databases you then have the choice (not the necessity) of keeping them on one or more machines. You can put on one machine a bunch of content sets that are relatively small, and put the big ones on their own machine. You can also scale the hardware to individual content sets - infrequently accessed content sets do not have to be on powerful machines, very frequently accessed sets can be scaled on bigger machines.
There are downsides, the two-query hit being the least significant, the extra query on a tuned database is on order of 1ms. Even if the hit was larger, I would still live it, scalability != performance
"At the end of the day hire good ppl. They'll do the right thing most of the time." - absolutely agree.
No system is perfect, ours is not. You need to decide what is important to your organization and emphasize it - they way we do things here at TA has its downsides, but overall, it works very well for us and what we want to do. The benefits for us significantly outweigh the drawbacks.
It is critical to have a boss who understands how software development works, and that mistakes get made. If anything, my boss wants my team to make more mistakes than we do :-)
(disclosure - I am the author of this article)
I think it is a matter of judgment. Bugs that bring the site down,I agree. Minor bugs that most people will not notice on a consumer site where we can quickly patch them, I would rather get more projects out. Where you land in between these positions depends on a lot of things.
(disclosure - I am the author of this article)
I am not sure about "faster", but it is more scalable and more flexible - especially considering that the datasets to be "joined" are small, page-sized datasets. We have a lot of different content types, with a central member database. Not having to have all of our content on one machine and not having the memory demand of doing joins allows us to scale more easily. This is not to say we "never" do joins (our member database has multiple tables), it is a matter of find the most appropriate modularity.
@latch - Yes, we saw it - would be a fun project, but we are software hackers :-) There are also concerns about heat, vibration, hot swapping, etc. I would happily spend $78K for that same storage instead of $7.8K if it was reliable and was shipped as a complete package.
Closest company doing something like this is CORAID, which might be an option, but they have a number of proprietary interfaces/systems to buy in to.
Need disk storage, would like to use 7200rpm for real-time access to logs, 15k for database/email apps
Would be a lot of fun to roll our own, but we are resource constrained, and while money is not an issue for us, I don't like complicated/expensive solutions, I like simple/inexpensive
Looking at NetApp for example, but having a hard time accepting the price point.
A superior solution in every way
If you were going to do this in Python (or another similar language), you need to write a sort function that operates on a file, not a Python collection, as the collection is always bound by RAM, which would at best be re-implementing the sort command. Once you can sort a file, everything else is trivial, as counts can be done line by line, or more siply, via uniq -c.
Of course, if you only care about things that fit in memory, you can do it in Python, but it is still far easier to use command line for these type of problems.