Ask YC: Scalability advice for web apps

7 pointsby Frocervor 18 Jahren3 comments

I am currently hosting my web app on Media Temple's Dedicated Virtual servers. It has been working great, but since we are pretty early, we aren't getting crazy amount of traffic either. Our web app is basically a social network, and there's a matchmaking algorithm which could potentially become a bottleneck (think Match.com or eHarmony's matching system).

Given that, and as we are growing about 25% every month. I am just curious on what kind of scaling issue we should be worried about? Are there any best practices in both back-end / database design to accommodate future scaling issues?

I read database performance is usually the bottleneck for most web apps -- and since our matching algorithm does hit the DB pretty hard, what kind of things I could do to improve DB performance and scale it accordingly (besides the obvious, e.g. caching and index).

Thanks for any advice!

5 comments

umangjaipuriavor 18 Jahren

a) Find out how to partition your data.

b) Can you pre-process some of the queries and make the data available to the algorithm in a BDb or some such?

c) Can you make your heavier components into independent services?

d) You know what they say about premature optimization... :)

HTH.

mg1313vor 18 Jahren

Well, I guess this website can answer to your question: www.highscalability.com

Hope this helps.

[deleted]vor 18 Jahren

yanvor 18 Jahren

I think you just might have a legit candidate for a MapReduce-like algorithm here since your work seems to be a great candidate for parallel work. Check the Hadoop article that was posted here not too long ago.

It will let you scale the work over many nodes in a fairly linear fashion.

You might not even have to go that route if you properly shard your data among your computers. If you store user profiles based on locality or the like, the work is guaranteed to be local. Also, you can try segmenting your users based on general factors that group them (a course version of your matching algorithm) to preserve locality.