This screenshot was taken on my dev box which has plenty of other activity happening. On a production memcache box, seeing 1Gb/s, I see ~2% packet loss.
Since I wrote memkeys, maybe I can clarify a few things.
First, dropping packets matters. If you see only 30-40% of your traffic you can't guarantee that you have enough data to know what your hot keys actually are. This is especially true when you are interested in (for instance) sorting keys by bandwidth usage. You might have a key that gets half as many hits as the hottest key but is 4x the size and causing network link saturation. In this case, depending on how much data you're able to capture, you may or may not even see this data point. Also, the follow-up comment from corresation about patching memcache doesn't make sense to me.
Second, this was no 'jab' at etsy. I know the etsy guys incredibly well and we're all friends. We've collaborated on work in more than one occasion. The jab comment seems like unnecessary speculation. The comment about seeing how memkeys affects performance is of course spot on. In this case, one thread will peg a CPU core for packet capture but besides that will not be CPU intensive. Since it uses packet capture, memkeys doesn't actually interact with memcached directly so the impact should be minimal. We used it at Tumblr.
Third, fixing the packet loss issue in mctop wasn't feasible as the problem is with ruby-pcap not with mctop. Additionally, while Tumblr has plenty of ruby code in production we don't generally use it for building 'real-time' applications. There are better languages for the job.
I built memkeys because it solved a problem we had, and was fun. That's it.
I'm just going to point out that this site is hosted by wisegeek.com, a site that was hit hard by the panda update (see http://www.quantcast.com/wisegeek.com). Although the data is accurate, I wouldn't feel sorry for most of these sites.
Full disclosure. I used to work for ChaCha. I am no longer associated with the company, but in my time there I know a lot of time/effort/money went into producing original content.
An aside. Pandalized was using a domain proxy so I connected to port 80 via telnet which gave me back the following banner (which gave up the hostname): Apache/2.2.8 (Debian) DAV/2 SVN/1.4.2 PHP/5.2.5-3+lenny2 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.2 Perl/v5.8.8 Server at strongwiki.wisegeek.com Port 80
Hey Steve, we do have plans to open source the code in two parts. The first part to likely be open sourced will be the high performance logging library that was written for it. After that, the proxy component. Keep an eye on the Tumblr engineering blog for a notification.
The upper band essentially represents new users. At the current rate of growth, we push into a new redis shard every 2-3 days. When we first start writing/reading against that shard, it appears to take a bit of time to warm up.
I worked on Staircar so maybe I can answer a couple of questions.
Machine failures: failures we'll likely address in a future post but basically each redis instance can have a slave which can be failed over to.
Performance: Staircar isn't nearly as fast as redis, but the primary project goal was making the redis infrastructure opaque to any clients. The secondary goal was performance, which is still quite good.
Early Optimization: In a large infrastructure (thousands of machines, multiple data centers, etc) you have locality issues, slow clients, machine failures and other operational considerations. Again, redis performance is fine and we weren't concerned with that. We were interested in creating a high performance proxy to a pool of redis instances. Also, with Staircar we can make online changes to the redis pool without needing to notify clients of those changes.
Remember the underpants gnomes from South Park? Their plot was simple. 1. Steal underpants, 2. ???, 3. Profit. Most YC people have something similar going on: 1. Come up with great idea, 2. Get YC to fund it, 3. ???, 4. Profit, 5. Get a Girlfriend. You've already jumped to step 5, skip 1-4.
If you can afford it, UltraDns (Neustar) is the best. This is the service that Amazon uses, as well as a number of other very large sites. They have a 100% uptime SLA and a latency one as well. We haven't had an outage with them in almost two years. They also have an API which is nice. Oh, and they can do geo-distribution of requests, as well as a variety of other more advanced functions.
We use The Planet (http://www.theplanet.com) along with EC2 from AWS. I've used The Planet for about 7 years now, for both business and personal use. Good prices, decent support and a top notch network.
Having used RoR and Spring fairly extensively, I was convinced that Groovy, and more specifically Grails, would be great to work with. Fun? Check. New? Check. Bleeding edge unstable? Check. Extremely difficult to figure out how to do things you already knew how to do with Spring or RoR? Check.
I just wasn't able to ever get efficient with the platform and ended up having to figure out how to expose Spring internals to do the stuff I wanted to do.
I would recommend building something with Grails if it's a personal project but not for anything commercial or production.
I've used JMeter (http://jakarta.apache.org/jmeter/) for a few years now. It's easy to use and extend. The only thing I don't like is it doesn't play nice with AJAX.
I'm thinking this might be a 'bad' idea unless you are also hosting the site on EC2. Having significant latency in your session handling will bring your site to a crawl due to IOWAIT on your web servers. Obviously making a request across the net for session data will be slow compared with a request across a local network.
Now, assuming you are also hosting your web servers on EC2, I would recommend using neither S3 or SimpleDB for hosting sessions. S3 is more appropriate for static data, and SimpleDB is still in early Beta (as in, the API is subject to change and you have no SLA).
Session data is largely meant to be ephemeral. Consider using a standard MySQL (or similar) session backend with a circular hash for distribution to different nodes. Unless you are doing a ton of traffic, you can get by with a single machine running the session backend. The MySQL backend scales to a lot of concurrent users.
There's a lot of overhead in running a MySQL instance compared with running a memcached instance (in terms of system resources, personnel, building it redundantly). I would revisit your problem and see if you can't get memcached to fit before you decide to reinvent the wheel.
If you're set on MySQL, assuming you are on Linux (or a similar platform) you should consider perhaps using BDB or MyISAM for your storage engine and just keeping the data on a ram disk. This will give you wicked fast reads.
I think Amazon should be part of your hosting strategy, but not the entirety of it.
We use Amazon (EC2, S3, SQS) but only where appropriate. EC2 for compute intensive but ephemeral data (logging, etc since your storage can disappear). S3 for hosting data files served up to HTTP clients (images, video, music) and SQS for, well, queuing :)
Most importantly though, we host the primary application in a typical data center. It's reliable, supported, we have an SLA and we can expand in any way we need to.
Like I said, use Amazon for part of your hosting strategy but not the entirety of it.