Ask HN: My host has me over a barrel. I've placed myself in a pickle. How to handle?
42 comments
You're paying quite a lot for a very weak VPS--compare at slicehost or linode. So your first step might should be to spend less for a more powerful server at one of the other VPS providers. OTOH, I'm not sure linode or slicehost even offer the kind of support package that you have paid for with midphase--you'd be on your own. OTOOH, if I read your post right you're paying midphase $130/month currently. You can get a whole lot of server for a lot less than that.
For some combination of Drupal's general heavy-weightness, lack of caching, slow server, your front page performance is sporadic and mostly very poor. I get times mostly in the 0.6 - 1.4 sec range just for loading the HTML, and that's with about 0.15 sec as the baseline for just loading a similar-sized static image from the site. I'm not a Drupal expert even a little, but if you could fully cache the front page (so that apache serves a static file directly) and update it, e.g., once every 10 min, you'd be flying, comparatively.
But about what's going on, I'd lay money it's all of the above: you have a low-resource server, drupal isn't fast/light by default and you haven't made it any better, you're not using any caching where it would really help. You should work on fixing all of those.
For some combination of Drupal's general heavy-weightness, lack of caching, slow server, your front page performance is sporadic and mostly very poor. I get times mostly in the 0.6 - 1.4 sec range just for loading the HTML, and that's with about 0.15 sec as the baseline for just loading a similar-sized static image from the site. I'm not a Drupal expert even a little, but if you could fully cache the front page (so that apache serves a static file directly) and update it, e.g., once every 10 min, you'd be flying, comparatively.
But about what's going on, I'd lay money it's all of the above: you have a low-resource server, drupal isn't fast/light by default and you haven't made it any better, you're not using any caching where it would really help. You should work on fixing all of those.
Excellent analysis given little info. Although it is possible the VPS host is not managing its resources well and its possible Apache is configed poorly, it is just as likely the app itself is a pig and needs lots of cleanup.
If you don't have the skill set be a good sysadmin and you "dont know" if its your app, you are in for a world of hurt. I highly suggest you find help: a good sysadmin and maybe an outsider to look at your code and web page structure. I have not heard enough to say your hosting company is screwing you. And its maybe wise that you are paying $60/month for better tech support but you still only get $60/month worth of tech support and that's not much ;(.
If you don't have the skill set be a good sysadmin and you "dont know" if its your app, you are in for a world of hurt. I highly suggest you find help: a good sysadmin and maybe an outsider to look at your code and web page structure. I have not heard enough to say your hosting company is screwing you. And its maybe wise that you are paying $60/month for better tech support but you still only get $60/month worth of tech support and that's not much ;(.
If you're willing to handle the stuff yourself, and it isn't that bad, SliceHost is a charm. I've hosted my django based blog (600-1000 uniques per day, ~6000 some peak days) on a 512 megabyte vps (Apache2 with mod_python, nginx for static files and proxying other requests to the Apache2, memcached instances with maybe 32 megs (not a lot of content to cache), and postgres) for the past 8 or 9 months and it's been great. On a 256 meg vps it used to crash, but 512 has been golden with my traffic levels.
Oh, and its $38/month for 516 meg slice. Can't say enough kind words about 'em (if you're willing to fire up SSH yourself:).
Oh, and its $38/month for 516 meg slice. Can't say enough kind words about 'em (if you're willing to fire up SSH yourself:).
That's the problem with Slicehost for me. I did sign up for a Slice, about a month ago, based on the reviews here. I spent three days trying to learn how to "make it go" and after some thought and cost/benefit analysis, decided against migrating over. Perhaps I should bite the bullet and spend a month learning sysadmin basics...
We have a lot of articles at http://articles.slicehost.com which will walk you through a lot of typical server installations. Additionally, even though our service is unmanaged, we're usually in a campfire chat and will answer questions if we're able (and direct you toward the answer if we're not).
Slicehost is amazing. I wish they had pre-built LAMP stack images ready, but it's not that hard to set up (esp if you follow their tutorials)
You need a SysAdmin. Now, this SysAdmin can be you (really, knowing the basics of server janitoring helps you as a developer and as a web designer, and as the guy who makes hosting decisions. Learning at least the basics of Linux is a good career move for almost everyone) but if not, you need to find someone you can depend on.
The other option is to find a host that specializes in hosting drupal sites and is willing to take care of all that for you. The biggest problem with this is that you can only use the features they install for you. Of course, they will also charge more than generic hosting, but it's still cheaper than getting a SysAdmin.
The other option is to find a host that specializes in hosting drupal sites and is willing to take care of all that for you. The biggest problem with this is that you can only use the features they install for you. Of course, they will also charge more than generic hosting, but it's still cheaper than getting a SysAdmin.
http://articles.slicehost.com. A godsend.
How about asking here if anyone is a Drupal/apache shark and is willing to help you tune your site and migrate to Slicehost?
You could maybe offer to pay for (some of) the help with a sponsored link on your site. That could be reasonably attractive to a Chicago-based contractor with a low pipeline.
You could maybe offer to pay for (some of) the help with a sponsored link on your site. That could be reasonably attractive to a Chicago-based contractor with a low pipeline.
Ok.
ahem
Is anyone a Drupal/apache shark and willing to help me tune my site and migrate to Slicehost?
We were featured on the HuffingtonPost, the Freakonomics blog, Playboy.com and numerous other "name" sites last week. Would be happy to discuss price or a link.
ahem
Is anyone a Drupal/apache shark and willing to help me tune my site and migrate to Slicehost?
We were featured on the HuffingtonPost, the Freakonomics blog, Playboy.com and numerous other "name" sites last week. Would be happy to discuss price or a link.
We've also got a great community of people on our forums (http://forum.slicehost.com) and our IRC channel #slicehost on Freenode.
From what you've said, I doubt host swapping is a good idea - yet. First you need to get a good understanding of what's causing the system to run out of memory. One way to do this is to watch vmstat and top when the system is failing, or to install one of many system monitoring packages.
From what you've stated, I'll take a stab at what's going on. Apache processes are big, from a memory usage standpoint. A single Apache + mod_php could use 25 MB or more. If you have a dozen or more requests that hit at once -- or processes that are feeding slow clients -- blam, you've run out of memory.
One solution is to switch to lighttpd, nginx, or another webserver with a smaller footprint. Here a single process (or small set) serves a pool of PHP processes. But that's a bunch of work for a guess at what's going wrong.
Instead, try running nginx as a proxy in front of Apache, and let it spoonfeed slow clients. It's likely you will not need to change Drupal at all, and make only minor changes to Apache to do this. A good description is here: http://blog.kovyrin.net/2006/05/18/nginx-as-reverse-proxy/
From what you've stated, I'll take a stab at what's going on. Apache processes are big, from a memory usage standpoint. A single Apache + mod_php could use 25 MB or more. If you have a dozen or more requests that hit at once -- or processes that are feeding slow clients -- blam, you've run out of memory.
One solution is to switch to lighttpd, nginx, or another webserver with a smaller footprint. Here a single process (or small set) serves a pool of PHP processes. But that's a bunch of work for a guess at what's going wrong.
Instead, try running nginx as a proxy in front of Apache, and let it spoonfeed slow clients. It's likely you will not need to change Drupal at all, and make only minor changes to Apache to do this. A good description is here: http://blog.kovyrin.net/2006/05/18/nginx-as-reverse-proxy/
384 MB of RAM is really low for a site (apache + db i assume). I host my Drupal site ( http://esciencenews.com ) at ServInt on a 90$/month package with 784 MB of RAM. Dont know how I would do it with 384, looks impossible to me. Higher traffic daily than your spike.
Hey there! I forgot you hang out here (confession: I read your esciencenews Drupal showcase writeup about once a week for inspiration).
Ok. So 384 looks low. That's helpful, objective advice. Thanks.
Ok. So 384 looks low. That's helpful, objective advice. Thanks.
I'm paying for a dedicated server at the moment with 1GB ram for 39 euros/month, an upgrade to 2gb ram (total) is 10 euros/month extra from LeaseWeb (http://leaseweb.com), which has half decent routing. I know the exchange rate sucks (being in the US myself), but an option to consider, perhaps.
I've always been partial to Slicehost, but my main need uses a lot of bandwidth and disk space and they absolutely can't compare to overseas hosts for bw/$. slicehost's most generous plan is 1600gb transfer at $280 and our 4000gb for 79 euros from leaseweb is vastly cheaper, although you have to mix and match some of the other specs.
Honestly, do not go with midphase dedicated servers. If I remember correctly they're just reselling softlayer(.com), and softlayer is a LOT better in every aspect compared to midphase. I've had some friends use midphase and have so much trouble with their customer support when something goes wrong...softlayer in my experience has been everything great about a company providing these kinds of services.
I've always been partial to Slicehost, but my main need uses a lot of bandwidth and disk space and they absolutely can't compare to overseas hosts for bw/$. slicehost's most generous plan is 1600gb transfer at $280 and our 4000gb for 79 euros from leaseweb is vastly cheaper, although you have to mix and match some of the other specs.
Honestly, do not go with midphase dedicated servers. If I remember correctly they're just reselling softlayer(.com), and softlayer is a LOT better in every aspect compared to midphase. I've had some friends use midphase and have so much trouble with their customer support when something goes wrong...softlayer in my experience has been everything great about a company providing these kinds of services.
1) You need to understand this stuff yourself. You need to learn about Apache and Drupal and the other technical bits - maybe not to the point of becoming an expert, but enough to work with them a bit.
2) Go elsewhere. Slicehost (or linode) has better VPS deals, and Layered Tech used to have some pretty good deals on 'real' servers.
2) Go elsewhere. Slicehost (or linode) has better VPS deals, and Layered Tech used to have some pretty good deals on 'real' servers.
I'd ask them to clarify what they mean by a crash. Is it a system crash or is it Apache going out to lunch or is it something else? A "crash" can mean so many different things because it is somewhat vague. I've seen plenty of cases where if you press someone for a root cause of a "crash", they can't give it to you because they don't really know what went wrong. You've got to understand the problem before you can solve it or be sure what they are recommending makes sense. If you want some more input, feel free to e-mail me.
After every outage I receive an automated e-mail telling me that httpd has failed on my VPS. The way this usually works is I notice the site goes down. It comes back up 8-10 minutes later and I find another e-mail in my inbox saying that apache failed and has restarted.
First, you should know that many types of caching will increase your memory usage, not decrease it. The whole point of a cache is to use fast storage (RAM) to keep a local copy of data that would be slower to retrieve from its persistent source (filesystem or DB).
Second, 384 MB just isn't much memory for a medium-traffic Drupal site. I would recommend at least a gig of memory, and perhaps more if you use plugins heavily. Drupal is neither particularly lean nor blazingly fast, so you're going to need a lot of Apache listeners working to handle that load, with each process probably consuming 30-50 MB of RAM.
Third, you're right to think that you need someone on-call who knows Apache, Drupal, and database tuning and configuration inside and out. A $60/mo. support package isn't going to do it, though. If you think about it, that's really only paying for an hour or two of tech time each month, so expecting to get 24/7 interactive performance tuning and optimization for free along with your support contract is pretty unreasonable. Put a performance guru on retainer, and establish an SLA with them if necessary to maintain your site uptime.
Finally, opening >30 tickets in less than a week is not the best way to get quality support. It's like the stereotypical impatient kid on a road trip, asking "are we there yet?" over and over again.
My suggestion would be for your to find an experienced LAMP admin who is willing to manage your hosting for you, and pay them a reasonable fee for doing so.
Second, 384 MB just isn't much memory for a medium-traffic Drupal site. I would recommend at least a gig of memory, and perhaps more if you use plugins heavily. Drupal is neither particularly lean nor blazingly fast, so you're going to need a lot of Apache listeners working to handle that load, with each process probably consuming 30-50 MB of RAM.
Third, you're right to think that you need someone on-call who knows Apache, Drupal, and database tuning and configuration inside and out. A $60/mo. support package isn't going to do it, though. If you think about it, that's really only paying for an hour or two of tech time each month, so expecting to get 24/7 interactive performance tuning and optimization for free along with your support contract is pretty unreasonable. Put a performance guru on retainer, and establish an SLA with them if necessary to maintain your site uptime.
Finally, opening >30 tickets in less than a week is not the best way to get quality support. It's like the stereotypical impatient kid on a road trip, asking "are we there yet?" over and over again.
My suggestion would be for your to find an experienced LAMP admin who is willing to manage your hosting for you, and pay them a reasonable fee for doing so.
I once had to deal with a hosting company that had a ticket tracking system that would mark tickets as closed if there was any kind of customer reply, and they had a fat fingered support person or two that also marked tickets as closed..and of course, if a ticket was open for too long they'd automatically be closed, so any ticket I send on a Friday gets closed by Monday morning. I HAD to open a couple dozen tickets and replies to tickets before someone finally replied to me and apologized for the whole thing :) Needless to say that was one reason why I am no longer with them..
But I can sort of commiserate with the OP. Midphase support is not exactly the best...I've spent hours trying to get them to understand the most trivial things, like when a friend's website wouldn't work because the mysql server was awol and the support person didn't understand what was wrong with nothing showing on the website. Yeaaaahhhhh.
But I can sort of commiserate with the OP. Midphase support is not exactly the best...I've spent hours trying to get them to understand the most trivial things, like when a friend's website wouldn't work because the mysql server was awol and the support person didn't understand what was wrong with nothing showing on the website. Yeaaaahhhhh.
Ok. I'm going to call the folks at ServInt and Linode and talk to them. I am concerned about migrating from one host to another as I've never done that before but I can figure it out.
I'm also going to look more into improving my site's performance. I have enabled Drupal's page caching, but I guess it's not doing much to help. I'll pursue that further.
I'm also going to look more into improving my site's performance. I have enabled Drupal's page caching, but I guess it's not doing much to help. I'll pursue that further.
Well the first you can do, is to configure apache to restart an httpd process after serving some number of clients. Under Apache 2.x this is called MaxRequestsPerChild and the default is 10000 requests. Since you are not bumping into CPU problems, try reducing this number to say 500 requests. After a child process has served 500 requests it will exit, and another process will be started.
Before slicehost I was on Mediatemple's gridserver... the latency was terrible and there was more downtime than I would have liked, but it was ok. I suspect their service has improved somewhat in the interim and they provide more bandwidth than any other shared, managed hosting provider for the price. If you really don't want to manage your own server, they are probably worth looking into.
Host if on Amazon Web services. Will cost you $72 / month and you can scale up from there at peak times if you need.
You get 1.7GB of memory and roughtly a 2.5GHz processor.
If you need help setting this up use Scalr.net or we (cloudincode.com can help you for free)
Drop a line if you are interested bjorn At Stadil.com
Move your static files to a different subdomain / host or vhost. (AWS?)
Turn Keep-Alive Off. (If you're site doesn't have tightly packed multiple requests). The default is 15 seconds, I saw a 50% speed increase when I disabled this. Set MaxRequestsPerChild to 500, in case your code is leaking.
Turn Keep-Alive Off. (If you're site doesn't have tightly packed multiple requests). The default is 15 seconds, I saw a 50% speed increase when I disabled this. Set MaxRequestsPerChild to 500, in case your code is leaking.
Better yet, set Keep-Alive to something very low (1 or 2), so that when users load a page, any initial assets can still reuse the same connection, but you'll still drop the connection quickly to serve other users.
Really important advice - at least for us (Xobni), turning off keep-alive on one of our content-delivery servers freed up a ton of resources.
I agree with most of the above. Move to a more cost effective host, optimize your Apache server, and use page caching in Drupal. I think it's as simple as a checkbox in the settings somewhere, but it's been a few months since I've worked with Drupal.
http://drupal.org/getting-started/6/admin/settings The manual says caching is under the Performance section and it's disabled by default.
http://drupal.org/getting-started/6/admin/settings The manual says caching is under the Performance section and it's disabled by default.
Their plans seem a bit expensive. Depending on how much bandwidth you use, slicehost.com might be a better option - you get 1GB of RAM for the same price.
For another $10-$20 you could have a physical dedicated server with 1GB memory on ServerBeach. I highly recommend them. You can get $100 off if you use my referral code 7XYHDMBU8A but even if you don't use my code I still recommend it!
http://www.serverbeach.com/catalog/cust_ref_landing_new.php?...
http://www.serverbeach.com/catalog/cust_ref_landing_new.php?...
tweak your apache configuration to see if that will fix your swapping problems. there are tons of articles on what you need to do to do this available online. sounds like you're just spawning a lot of children that are sticking around to fill up memory and swap space.
i had similar problems with a default config of apache on a 256 slice. if you tweak things correctly, you'll be fine.
i had similar problems with a default config of apache on a 256 slice. if you tweak things correctly, you'll be fine.
you have 2 options.
1. buy a much more powerful server/VPS.
for seventy some odd dollars a month you can just leave your amazon ec2 image up all the time, and that's something like 1.7Gb ram. if you are willing to use a small provider (like me) you can do even better- I'll give you 2GB ram for $64/month. Having more ram makes a lot of problems just go away. (using a Xen based host also means that you can use swap, which means that your out of memory events would be slowdowns rather than crashes.)
2. get a skilled SysAdmin (possibly you) to optimize your system from a systems standpoint.
ultimately, you want both. But often just buying a better server is the cheapest option.
But yeah. no matter how much you like your host, I recommend that you always have a hot copy of your website/application at another server, ready to switch the dns over at the first sign of trouble. you should test this periodically.
1. buy a much more powerful server/VPS.
for seventy some odd dollars a month you can just leave your amazon ec2 image up all the time, and that's something like 1.7Gb ram. if you are willing to use a small provider (like me) you can do even better- I'll give you 2GB ram for $64/month. Having more ram makes a lot of problems just go away. (using a Xen based host also means that you can use swap, which means that your out of memory events would be slowdowns rather than crashes.)
2. get a skilled SysAdmin (possibly you) to optimize your system from a systems standpoint.
ultimately, you want both. But often just buying a better server is the cheapest option.
But yeah. no matter how much you like your host, I recommend that you always have a hot copy of your website/application at another server, ready to switch the dns over at the first sign of trouble. you should test this periodically.
How about EC2? 1.7GB on their lowest virtual server - Works out at about $72~ per month I think?
I had the same problem four years ago. It killed my startup. The volume numbers weren't there for a dedicated box and the VM-type hosting was continuously (and randomly) rebooting.
I know HN discourages superfluousness, but I'd just like share my moment of zen as I contemplate how this is exactly the kind of post that keeps me coming to HN.
Happy to help! I was honestly worried I'd get booted for posting about hosting when most here can host their own sites.
Couple quick things for Drupal:
Do you have Optimize CSS files (CSS file aggregation) enabled?
Do you have Caching mode (Normal) enabled?
Cheers
Do you have Optimize CSS files (CSS file aggregation) enabled?
Do you have Caching mode (Normal) enabled?
Cheers
Yes and yes.
I'm also using the blockcache module for most of my blocks and have js aggregation switched on (though I have to omit tinymce and a few other js files in order for them to work).
I'm also using the blockcache module for most of my blocks and have js aggregation switched on (though I have to omit tinymce and a few other js files in order for them to work).
FastCGI and mod_xcache? Made all the difference on a couple of TYPO3 (PHP CMS) site I've run.
check out http://novanix.com/
I realize the website looks shady as all hell. But we use them for EVERYTHING and we've never had a problem. Their support is also top notch. Usually answers+fixes <30 mins.
I realize the website looks shady as all hell. But we use them for EVERYTHING and we've never had a problem. Their support is also top notch. Usually answers+fixes <30 mins.
linode.com asap
Seconded. You could probably host this site quite easily on their $40/mo package - if not lower.
While we had 5,000+ uniques per day last week, on average we've been getting around 1200-1500 per day.
This doesn't seem like a lot of traffic to me. However for the last 4 days, apache has been failing every 3-4 hours and restarting about 10 minutes later, meaning 10 minutes of downtime for each crash. My host is saying the site is hitting its memory limit all the time. Needless to say, this is a problem.
For the last three days I've been filing tickets with my support guys (I bought a 5-minute response support package for $60/month thinking it would free me up to do more writing/editing) every time there's an outage. I've now filed 34 tickets. Their responses say the same thing every time "We are making apache configuration changes and will monitor your performance." Last night we had about 5 hours without a crash. I thought things were fixed.
They just went down again.
I've spoken with a manager at Midphase. He said that he doesn't see any obvious problems with how their server is configured or how my site is configured. It's just using up too much memory. His suggestion (of course) is to move to a dedicated server with Midphase at $200/month.
Now, I'm more than willing to concede that my site could be better optimized. I'm working on that and have some Drupal pro's who've offered to give it a look sometime in the near future. I'm also aware that I should be planning for future growth with our server setup.
I just don't want to get ripped off.
It seems like I've got three options.
1. Ask my hosting guys the right questions and find out what's causing the problem and get it fixed.
2. Upgrade to a dedicated Midphase server, potentially encounter the same problems.
3. Go elsewhere, potentially encounter the same problems.
Are there some specific questions I should ask my hosting people to help me get a better picture of the situation?