Ask HN: Why and how is Hacker News so fast?
109 comments
It's relatively straightforward. Take a look at what Pingdom tells you it does when the site loads:
http://tools.pingdom.com/fpt/#!/cnYAuq/https://news.ycombina...
5 requests, Total size 12.6kB, total time 661ms. 1 request for the content, 1 for css, and then 3 gifs, all from a single domain (so just one DNS request). If you look at the profile, 47% of the time was spent doing that one DNS lookup, so subsequent requests will be even quicker (plus caching will handle those gifs just fine)
That's almost no work for the end client to do. No javsacript for the client to download, parse and process, just straight content to render and done.
The reduced request count is especially important when it comes to the mobile experience, where bandwidth might be great, but latency is terrible. Every request you have to make to render a page significantly impacts the loading time (HTTP2 helps here). Along with trying to reduce the number of resources being loaded on the screen, it's especially important to reduce the number of domains you're getting them from so you keep the DNS queries down.
http://tools.pingdom.com/fpt/#!/cnYAuq/https://news.ycombina...
5 requests, Total size 12.6kB, total time 661ms. 1 request for the content, 1 for css, and then 3 gifs, all from a single domain (so just one DNS request). If you look at the profile, 47% of the time was spent doing that one DNS lookup, so subsequent requests will be even quicker (plus caching will handle those gifs just fine)
That's almost no work for the end client to do. No javsacript for the client to download, parse and process, just straight content to render and done.
The reduced request count is especially important when it comes to the mobile experience, where bandwidth might be great, but latency is terrible. Every request you have to make to render a page significantly impacts the loading time (HTTP2 helps here). Along with trying to reduce the number of resources being loaded on the screen, it's especially important to reduce the number of domains you're getting them from so you keep the DNS queries down.
There was a thread about slow web: https://news.ycombinator.com/item?id=9791606
I prefer a static site without JavaScript any time of day! (over a JS bloated webs(h)ite)
I prefer a static site without JavaScript any time of day! (over a JS bloated webs(h)ite)
Might be able to get that down to 5 requests if they use .pngs instead of .gifs and throw them into a sprite sheet.
"... (so just one DNS request)."
Are you sure?
request #1
1 news.ycombinator.com: 207 bytes, 1+0+4+2 records, response, noerror query: 1 news.ycombinator.com authority: ycombinator.com 172800 NS ns-225.awsdns-28.com
request #2
1 news.ycombinator.com: 225 bytes, 1+1+4+0 records, response, authoritative, noerror query: 1 news.ycombinator.com answer: news.ycombinator.com 300 CNAME news.ycombinator.com.cdn.cloudflare.net authority: ycombinator.com 172800 NS ns-1411.awsdns-48.org
request #3
1 news.ycombinator.com.cdn.cloudflare.net: 367 bytes, 1+0+5+10 records, response, noerror query: 1 news.ycombinator.com.cdn.cloudflare.net authority: cloudflare.net 172800 NS ns1.cloudflare.net
request #4
1 news.ycombinator.com.cdn.cloudflare.net: 89 bytes, 1+2+0+0 records, response, authoritative, noerror query: 1 news.ycombinator.com.cdn.cloudflare.net answer: news.ycombinator.com.cdn.cloudflare.net 300 A 198.41.190.47
A HOSTS file entry for HN will avoid this routine.
Are you sure?
request #1
1 news.ycombinator.com: 207 bytes, 1+0+4+2 records, response, noerror query: 1 news.ycombinator.com authority: ycombinator.com 172800 NS ns-225.awsdns-28.com
request #2
1 news.ycombinator.com: 225 bytes, 1+1+4+0 records, response, authoritative, noerror query: 1 news.ycombinator.com answer: news.ycombinator.com 300 CNAME news.ycombinator.com.cdn.cloudflare.net authority: ycombinator.com 172800 NS ns-1411.awsdns-48.org
request #3
1 news.ycombinator.com.cdn.cloudflare.net: 367 bytes, 1+0+5+10 records, response, noerror query: 1 news.ycombinator.com.cdn.cloudflare.net authority: cloudflare.net 172800 NS ns1.cloudflare.net
request #4
1 news.ycombinator.com.cdn.cloudflare.net: 89 bytes, 1+2+0+0 records, response, authoritative, noerror query: 1 news.ycombinator.com.cdn.cloudflare.net answer: news.ycombinator.com.cdn.cloudflare.net 300 A 198.41.190.47
A HOSTS file entry for HN will avoid this routine.
maybe I should rephrase that as one outbound DNS query? The answer should be cached locally, or at least at your next hop resolver, and the TTL of 300 seconds is long enough to make it effective
HN pages are small, the HTML is generated on the server, and they render nearly progressively. That is, the renderer doesn't need to block waiting for many CSS, JavaScript, or font files to download.
Most development practices considered "modern" prevent progressive rendering. Common tools like Bootstrap or Jquery, as normally used, cause page rendering to block until large libraries are downloaded, despite the fact that only a tiny portion of bootstrap or Jquery are actually used by the page.
The largest reason of all is that most developers dismiss such concerns as old school worries and like to harp on and on about how tools like Angular enforce wonderful software engineering practices, ignoring that most such frameworks, as commonly used, bloat pages and cause incredible rendering delays.
The difficulty with improving performance is that most "modern" pages have numerous render-blocks. The page doesn't get fast until you fix all of them. This is the reason most wise suggestions to speed up a page make little difference when tested individually.
Most development practices considered "modern" prevent progressive rendering. Common tools like Bootstrap or Jquery, as normally used, cause page rendering to block until large libraries are downloaded, despite the fact that only a tiny portion of bootstrap or Jquery are actually used by the page.
The largest reason of all is that most developers dismiss such concerns as old school worries and like to harp on and on about how tools like Angular enforce wonderful software engineering practices, ignoring that most such frameworks, as commonly used, bloat pages and cause incredible rendering delays.
The difficulty with improving performance is that most "modern" pages have numerous render-blocks. The page doesn't get fast until you fix all of them. This is the reason most wise suggestions to speed up a page make little difference when tested individually.
"The largest reason of all is that most developers dismiss such concerns as old school worries and like to harp on and on about how tools like Angular enforce wonderful software engineering practices, ignoring that most such frameworks, as commonly used, bloat pages and cause incredible rendering delays."
Alternatively, you could say that many people look at the tradeoffs and decide they're comfortable with trading off a certain amount of speed for the user, in order to increase development speed.
Whether these people are right or wrong really depends on the context.
Alternatively, you could say that many people look at the tradeoffs and decide they're comfortable with trading off a certain amount of speed for the user, in order to increase development speed.
Whether these people are right or wrong really depends on the context.
Users sitting on gigabit links close to the server make the tradeoffs, and people on a high latency end of a 3G link, or an airport wifi with a weak signal and 50% packet loss need to live with those tradeoffs.
Sometimes, yes, in which case the tradeoff might not make sense.
But sometimes it does - e.g. I think in most cases, something existing but being slow is better than something not existing in the first place, which might be the case if the costs of making it go up.
But sometimes it does - e.g. I think in most cases, something existing but being slow is better than something not existing in the first place, which might be the case if the costs of making it go up.
Loading this webpage I downloaded ~12KB. For comparison, I went to the NYT homepage and downloaded ~1.2 MB. For people with shitty downlinks like you and I, the absence of exorbitant quantities of Javascript and images makes a lot of difference - a lot more than time-to-first-byte.
It's not just small but the HTML part is also dead simple to render.
Or for a bigger offender, I noticed a couple of days ago that Rally (https://www.rallydev.com/) loaded over 5 MB of JavaScript from 3 scripts...being loaded 3+ times each (each one is over 600 KB)! Each script seemed to contain their own copy of ExtJS. Each script returns a 200, and doesn't use a 304 either on each load :( .
[deleted]
that 1.2Mb gets cached. Infact that will make the site faster because once the .js are downloaded only json/partials will be fetched when visited again.
It must be the backend ,ie when tasked to render the same page ,which backend would respond faster
It must be the backend ,ie when tasked to render the same page ,which backend would respond faster
I have a fast FIOS connection, and the New York Times webpage still takes around 6 seconds to fully load according to the Chrome Network panel. In that time it makes 226 requests and downloads 168 KB. This is after several page refreshes, so I'm fully taking advantage of browser caching. Simply put, there's way more images, fonts, and network callbacks on that site than on HN. It's way more heavyweight. HN, by contrast, downloads all of the content in only SIX requests, at 14.8 KB total, and in under a second.
It's the number of requests that's killing the NYT site. HN is very simple and old school, and doesn't do a single thing that isn't explicitly necessary to render exactly the content you see on the page, which is presented cleanly and without frills.
It's the number of requests that's killing the NYT site. HN is very simple and old school, and doesn't do a single thing that isn't explicitly necessary to render exactly the content you see on the page, which is presented cleanly and without frills.
That 1.2Mb gets partly cached on the desktop. Mobile clients cache very little. Safari on my iPad is currently storing only 6.1Mb of website data and I use it every day on a variety of websites.
Client-side caching is a nothing but a lie bad front-end developers tell themselves to justify producing bad websites.
Client-side caching is a nothing but a lie bad front-end developers tell themselves to justify producing bad websites.
If you want to see an even faster "forum", see
http://forum.dlang.org/
http://forum.dlang.org/
That's impressive. What is their architecture?
http://forum.dlang.org/help#about says
> This website is a web frontend to the DigitalMars NNTP server and mailing lists. It is part of DFeed, which is also a D news aggregator and IRC bot.
> DFeed was written mostly by Vladimir Panteleev. Portions (including style and graphics) are Copyright © by Digital Mars. The source code is available under the GNU Affero General Public License on GitHub: https://github.com/CyberShadow/DFeed
Probably as static as possible, speed is easy then.
> This website is a web frontend to the DigitalMars NNTP server and mailing lists. It is part of DFeed, which is also a D news aggregator and IRC bot.
> DFeed was written mostly by Vladimir Panteleev. Portions (including style and graphics) are Copyright © by Digital Mars. The source code is available under the GNU Affero General Public License on GitHub: https://github.com/CyberShadow/DFeed
Probably as static as possible, speed is easy then.
>Probably as static as possible, speed is easy then.
With an empty cache, every asset except font awesome is getting to me - here in Poland - in under 200ms, with some under 150ms.
For 22 requests - even with a total payload of "only" 155kb - whether static or not, that is seriously impressive.
With an empty cache, every asset except font awesome is getting to me - here in Poland - in under 200ms, with some under 150ms.
For 22 requests - even with a total payload of "only" 155kb - whether static or not, that is seriously impressive.
This one is crazy. How are they so fast?
No, forum.dlang.org is not crazy-fast. Other sites are crazy-slow.
Write compact and easy-to-render HTML, it's no rocket science. Write some trivial CSS, you don't need Bootstrap. Make something dead-simple and call it flat design. If you have more than one stylesheet, concatenate them into one. Consider minifying. You don't need a 200 kB JS framework in order to fade in some hover menu. If you do need more than one JS file, concatenate them into one. Minify/uglify and move the <script> tag to the bottom of your HTML.
You don't need 2 MB worth of stock photos. Recently, there was a page on HN with an uncompressed 12 MB banner photo. That is insane.
You don't need those heavyweight social plugins, they ruin your day (performance-wise). If your content is great, people will find a way to share. If your boss specifically asks for Facebook and Twitter buttons, just link to https://www.facebook.com/sharer/sharer.php?u=http://www.exam... and https://twitter.com/home?status=This%20is%20great:%20http://.... [1]
Don't just embed YouTube iframes, they are even worse. Use yt-embed.js [2].
Is a web font really necessary? If it is, do you really need Bold Italic? Sure as hell you don't need Ultralight+Light+Semilight+Regular+Bold, just because there is one ubercool Ultralight tagline in the footer.
Make sure Nginx or Apache gzip HTML, CSS and JS and send far-future Expires headers for all your static assets. Make sure that there are no redirects, they add latency.
If you follow these simple rules, your page weighs under 50 kB and needs no more than 10 HTTP requests. You don't even need complicated CMS caching and CloudFlare. But hey, there are some 10,000 cache plugins for WordPress, just install one.
[1] http://www.sharelinkgenerator.com/
[2] https://gist.github.com/do18/09761dfabf60689b2cd8
Write compact and easy-to-render HTML, it's no rocket science. Write some trivial CSS, you don't need Bootstrap. Make something dead-simple and call it flat design. If you have more than one stylesheet, concatenate them into one. Consider minifying. You don't need a 200 kB JS framework in order to fade in some hover menu. If you do need more than one JS file, concatenate them into one. Minify/uglify and move the <script> tag to the bottom of your HTML.
You don't need 2 MB worth of stock photos. Recently, there was a page on HN with an uncompressed 12 MB banner photo. That is insane.
You don't need those heavyweight social plugins, they ruin your day (performance-wise). If your content is great, people will find a way to share. If your boss specifically asks for Facebook and Twitter buttons, just link to https://www.facebook.com/sharer/sharer.php?u=http://www.exam... and https://twitter.com/home?status=This%20is%20great:%20http://.... [1]
Don't just embed YouTube iframes, they are even worse. Use yt-embed.js [2].
Is a web font really necessary? If it is, do you really need Bold Italic? Sure as hell you don't need Ultralight+Light+Semilight+Regular+Bold, just because there is one ubercool Ultralight tagline in the footer.
Make sure Nginx or Apache gzip HTML, CSS and JS and send far-future Expires headers for all your static assets. Make sure that there are no redirects, they add latency.
If you follow these simple rules, your page weighs under 50 kB and needs no more than 10 HTTP requests. You don't even need complicated CMS caching and CloudFlare. But hey, there are some 10,000 cache plugins for WordPress, just install one.
[1] http://www.sharelinkgenerator.com/
[2] https://gist.github.com/do18/09761dfabf60689b2cd8
>No, forum.dlang.org is not crazy-fast. Other sites are crazy-slow.
You're ignoring that this site does use jQuery and webfonts and loads all of those pretty avatars separately. 22 requests, at my count.
For the reasons you list - i.e. words to the effect of "don't use all of this stuff" - I think that it is impressively fast, and their achievement shouldn't be hand-waved away quite so readily.
You're ignoring that this site does use jQuery and webfonts and loads all of those pretty avatars separately. 22 requests, at my count.
For the reasons you list - i.e. words to the effect of "don't use all of this stuff" - I think that it is impressively fast, and their achievement shouldn't be hand-waved away quite so readily.
I'm not ignoring anything, I'm describing the philosophy behind sites like these: It seems that perfection is attained not when there is nothing more to add, but when there is nothing more to remove. [1]
Many seem to believe that there is some secret sauce involved in making web pages load fast. There isn't. Server-side caching is no rocket science. The key is to keep everything so simple that you have the overview over every delivered byte at all times.
Take a look at the HTML of forum.dlang.org. It's so simple, if it wasn't minified, I could immediately start working on this site. When someone asks me to work on some existing wiki or blog template, I usually spend an hour removing unused code first and I still struggle to find my way around the codebase. At that point, maintaining efficiency becomes impossible.
When you have a simple codebase, you may add a font, avatars, and jQuery. The trick is to think about whether you need it, not throw it in "just in case." Specifically, some tiny avatars won't hurt overall performance as long as the page doesn't need to be redrawn. Therefore, always specify width and height attributes for your img elements in HTML or CSS.
[1] https://en.wikiquote.org/wiki/Antoine_de_Saint_Exup%C3%A9ry#...
Many seem to believe that there is some secret sauce involved in making web pages load fast. There isn't. Server-side caching is no rocket science. The key is to keep everything so simple that you have the overview over every delivered byte at all times.
Take a look at the HTML of forum.dlang.org. It's so simple, if it wasn't minified, I could immediately start working on this site. When someone asks me to work on some existing wiki or blog template, I usually spend an hour removing unused code first and I still struggle to find my way around the codebase. At that point, maintaining efficiency becomes impossible.
When you have a simple codebase, you may add a font, avatars, and jQuery. The trick is to think about whether you need it, not throw it in "just in case." Specifically, some tiny avatars won't hurt overall performance as long as the page doesn't need to be redrawn. Therefore, always specify width and height attributes for your img elements in HTML or CSS.
[1] https://en.wikiquote.org/wiki/Antoine_de_Saint_Exup%C3%A9ry#...
Lot of good points, thanks.
>You don't need 2 MB worth of stock photos. Recently, there was a page on HN with an uncompressed 12 MB banner photo. That is insane.
That seems to be a big trend nowadays and a huge pain.
>You don't need 2 MB worth of stock photos. Recently, there was a page on HN with an uncompressed 12 MB banner photo. That is insane.
That seems to be a big trend nowadays and a huge pain.
What you're determining to be "needed" wrt design is very draconian and anti-progress. The web was ugly in 95 and appealed to few... progress in all areas -- especially design -- has made it what it is today.
I find that the prevalent user-friendly, minimalist design philosophy is very much compatible with performance principles like those that I have outlined.
Performance is not the opposite of UX, it is an important part of it.
Performance is not the opposite of UX, it is an important part of it.
What it is today, designwise, is awful, with screens littered with irrelevant content and extremely aggressive ads that are like xroach on steroids.
To some of you who are testing with your browser: Observing load times in your browser is fine for just fun but latency is irrelevant when evaluating the speed we're talking about here. A loading time benchmark of server-side code should naturally be calculated server-side, a server-side measurement of how long it took to execute the backend code. Nobody can accurately benchmark D over the net using their browser. If D is also being used to code the web server, then a browser should be used to benchmark the code but not across the net unless you can somehow subtract your latency. If not, it should be done locally or over a LAN where latency is extremely consistent but probably locally. I always benchmark my sites in this way.
As for D being quite fast, the consensus is that yes, it is. From what I gather, you'd be hard-pressed to beat or match it using any other language not counting perhaps assembly. It also seems to have a relatively low memory footprint comparable to that of C and C++.
IMO, more people need to be using D. It seems to outperform every other language in every speed benchmark I've seen, is packed with modern features, and doesn't have a steep learning curve. I'm not someone who has a bias toward D either. I've only recently become interested in it and I come from a background of C, C++, and C# (As well as many scripting languages).
Try D, I am.
As for D being quite fast, the consensus is that yes, it is. From what I gather, you'd be hard-pressed to beat or match it using any other language not counting perhaps assembly. It also seems to have a relatively low memory footprint comparable to that of C and C++.
IMO, more people need to be using D. It seems to outperform every other language in every speed benchmark I've seen, is packed with modern features, and doesn't have a steep learning curve. I'm not someone who has a bias toward D either. I've only recently become interested in it and I come from a background of C, C++, and C# (As well as many scripting languages).
Try D, I am.
> It seems to outperform every other language in every speed benchmark I've seen
> It seems to outperform every other language in every speed benchmark I've seen
I like D. A lot. But please don't do this. It a) does not help D's popularity (if that is what you are trying to do , b) it makes you look like a troll or makes you look stupid (sorry)
> It seems to outperform every other language in every speed benchmark I've seen
I like D. A lot. But please don't do this. It a) does not help D's popularity (if that is what you are trying to do , b) it makes you look like a troll or makes you look stupid (sorry)
I said that because when I lookup the benchmarks, that's what I see. I'm sure if I setup similar benchmarks of my own I'd probably get the same results (Unless the people who posted those that I've seen are lying and for some reason I doubt that).
You should think before calling someone a troll or saying that they sound stupid when they're citing information from an outside source. If I didn't think Dlang had a performance benefit I'd have a bit less incentive to use it. Are you saying such benchmarks are false? They seem legitimate to me. Search and view them yourself. If I were a troll I'd do something like you did, citing no evidence. Commenting in such a way without citing data makes YOU sound stupid (Sorry).
You should think before calling someone a troll or saying that they sound stupid when they're citing information from an outside source. If I didn't think Dlang had a performance benefit I'd have a bit less incentive to use it. Are you saying such benchmarks are false? They seem legitimate to me. Search and view them yourself. If I were a troll I'd do something like you did, citing no evidence. Commenting in such a way without citing data makes YOU sound stupid (Sorry).
To be fair, I'll run my own benchmarks at some point. For now, I have to rely on what's out there.
Wow quiet fast indeed, in my experience the fastest dynamic site I browse is kickass torrents (=> no link for obvious reason)
With cache enabled they both take less than 150ms to render a full page which actually should be a standard but seems really really faster than any sites I visit everyday
I have been puzzled by this being said about dlang forever and today about HN. These sites are certainly not noticeably slow, but I don't notice anything remarkably fast about them either. Am I not visiting enough of the slow sites ?
My working hypothesis is that I don't have a good understanding of web-development to appreciate the nuances. Much akin to me listening to Indian classical music, Jazz: I would often encounter trained listeners breaking out into a (silent) applause. At those instances I feel bad because I realize I have missed something but I wish I didn't. So all pointers appreciated.
My working hypothesis is that I don't have a good understanding of web-development to appreciate the nuances. Much akin to me listening to Indian classical music, Jazz: I would often encounter trained listeners breaking out into a (silent) applause. At those instances I feel bad because I realize I have missed something but I wish I didn't. So all pointers appreciated.
How fast is your network link, both in terms of throughput and latency for the sites you visit? The OP is browsing over a 256 kbps connection of unknown origin. They are likely impacted by latency and throughput constraints, which will exacerbate issues with sites that require a lot of requests or are resource heavy.
Making a site fast boils down to:
1) Making sure your server responds quickly.
2) Reducing the number of requests required for a page to load.
3) Reducing the number of bytes required for a page to load.
Making a site fast boils down to:
1) Making sure your server responds quickly.
2) Reducing the number of requests required for a page to load.
3) Reducing the number of bytes required for a page to load.
With an empty cache, for me, hacker news loads in ~700ms.
Under the same conditions, the dlang site loads in ~200ms.
That's half a second faster, which is definitely noticeable when clicking around.
Under the same conditions, the dlang site loads in ~200ms.
That's half a second faster, which is definitely noticeable when clicking around.
Thanks for the numbers. A difference of 500ms would definitely be noticeable, especially if its in ones daily workflow
> Am I not visiting enough of the slow sites?
Do you use NoScript? I feel the same way because I usually run with JS turned off. Most things are pretty zippy despite using a computer from 2009.
Do you use NoScript? I feel the same way because I usually run with JS turned off. Most things are pretty zippy despite using a computer from 2009.
Ah! that and flashblock might explain it.
Wow, really fast, feels like fully loaded single page app and only downloading jsons
In addition to being so small (text-based, minimal JS), HN is cached by CloudFlare, which grabs copies of dynamic pages and pushes those copies out to their CDN edge servers around the globe so they're closer to end users.
When your browser asks for the IP to news.ycombinator.com, you get the address of a cloudflare CDN server. Which web server the DNS server will send you is based on a GeoIP lookup of your own IP address so that they can give you the address to the server physically closest to you, and therefore the lowest possible latency.
Edit: Apparently this is not how CF does things (see below explanation by benjojo12) but is not uncommon in load-balancing and distribution.
When your browser asks for the IP to news.ycombinator.com, you get the address of a cloudflare CDN server. Which web server the DNS server will send you is based on a GeoIP lookup of your own IP address so that they can give you the address to the server physically closest to you, and therefore the lowest possible latency.
Edit: Apparently this is not how CF does things (see below explanation by benjojo12) but is not uncommon in load-balancing and distribution.
news.ycombinator.com. 296 IN CNAME news.ycombinator.com.cdn.cloudflare.net.
news.ycombinator.com.cdn.cloudflare.net. 299 IN A 198.41.190.47
news.ycombinator.com.cdn.cloudflare.net. 299 IN A 198.41.191.47Pretty close!
We don't actually use GeoIP at the DNS level, we use the routing table of the internet to direct people to the closest location we have to them by announcing the same IP addresses in the routing table in many places (eg, anycast) You can find a blog post primer and general more explanation here: https://blog.cloudflare.com/cloudflares-architecture-elimina...
the upshot of this is that TCP and SSL connections start faster because of lower latency to our location, and we can pull items out of cache faster and give them to the client.
We don't actually use GeoIP at the DNS level, we use the routing table of the internet to direct people to the closest location we have to them by announcing the same IP addresses in the routing table in many places (eg, anycast) You can find a blog post primer and general more explanation here: https://blog.cloudflare.com/cloudflares-architecture-elimina...
the upshot of this is that TCP and SSL connections start faster because of lower latency to our location, and we can pull items out of cache faster and give them to the client.
Awesome, thanks for the additional info. On a related note, I really appreciate how good CloudFlare is about knowledge sharing. Not a lot of companies would be so willing to share information that's so integral to their business.
although it's useful to note that use of anycast is impossible to hide. any looking glass will tell you that CF has tons of routes to each apparent IP, e.g. https://ssp.pme.gin.ntt.net/lg/lg.cgi?query=bgp&sourceIP=IP&...
On the other hand, they do some stupid stuff, disregarding the standard and how useful it is [0].
[0] https://blog.cloudflare.com/deprecating-dns-any-meta-query-t...
[0] https://blog.cloudflare.com/deprecating-dns-any-meta-query-t...
Disregarding standards is not uncommon when you're actually running a high-performance system that faces attacks. Plenty of stuff goes into specs that just doesn't make sense while operating. Their explanation of why they are disabling ANY seems perfectly fine.
All I can see is that you've never needed to debug problems with DNS.
So every time a new comment is posted, is the CloudFlare cache invalidated and replaced with a new version of the comments page?
Actually for now, HN is not caching their HTML with us. we put a "CF-Cache-Status" header for when we are caching things
So when hitting a comment page, HN's backend is being hit in some way, whether by reading a db or by concatenating flat files or something else? Or are they just using a local cache that isn't pushed out to CloudFlare because there would be so much churn that pushing to CF wouldn't be a benefit?
Are they using Railgun to only transmit changes? Railgun seems like the neatest thing CF offers, as, AFAIK, you get the bandwidth benefits of caching without having to deal with invalidation.
Not sure, but cloudfare supposedly speeds up content delivery even for dynamic content where you need a new db hit each time. The idea is that it leaves open a persistent connection with the webserver, which means A) less work creating and destroying connections B) each new connection isn't starting off as throttled.
They have tech to do even better. See here https://www.cloudflare.com/railgun
That is how you can make it faster even when somebody is logged in and you are delivering a somewhat personalized experience.
That is how you can make it faster even when somebody is logged in and you are delivering a somewhat personalized experience.
Not pulling in a ton of external javascript, images, and ads is probably the biggest, but one thing that helps is that most comment threads have a relatively low number of comments. This page currently only pulls down down 12KB (compressed to 3.75KB).
A thread like https://news.ycombinator.com/item?id=9976298 with ~600 comments pulls down 785KB (compressed to 143KB)
In general, looking at the network tab in the developer tools will show why one site loads faster than another:
https://developer.chrome.com/devtools/docs/network
https://developer.mozilla.org/en-US/docs/Tools/Network_Monit...
A thread like https://news.ycombinator.com/item?id=9976298 with ~600 comments pulls down 785KB (compressed to 143KB)
In general, looking at the network tab in the developer tools will show why one site loads faster than another:
https://developer.chrome.com/devtools/docs/network
https://developer.mozilla.org/en-US/docs/Tools/Network_Monit...
Can someone buy the .text TLD already and make a whole part of the internet where this approach is standard?
Sorry for the offensive domain name (it's not mine), but this is relevant here:
http://motherfuckingwebsite.com/
http://motherfuckingwebsite.com/
I prefer http://bettermotherfuckingwebsite.com/ which was my inspiration for my blog design http://benlowery.co.uk/ :)
You know, after 20 years of www, one'd think browsers would be able to competently format text files already. The fact that we need that small amount of css (instead of it just being available) means the web is still broken.
Check out Firefox reader view.
There should be a doctype declaration that automatically triggers this.
As I said last time that link was posted:
> First thing I did, as I usually do with sites that have annoyingly small text width, was inspect element to increase contrast / width / remove the doublespacing. Then I read it. ...irony.
> First thing I did, as I usually do with sites that have annoyingly small text width, was inspect element to increase contrast / width / remove the doublespacing. Then I read it. ...irony.
The red font on dark grey background you used is off-putting.
Or, even better, mandatory.
Wow, really? Has it become such a generational sea change? Have people really failed to understand the nightmare that modern javascript, full-motion animated gifs, web videos, and ultra high-resolution jpegs represent to bandwidth? Nevermind their local CPUs?
Javascript is really the black hole, though. When people defer their client-side libraries to whatever CDN, and then let that library manage whatever it pleases for dependencies, forget it. That's like saying, I don't even care if people's computers catch fire and explode, when they try to watch this awesome star field simulation.
Oh yeah, and all the ads and spyware analytics too, ya know...
Javascript is really the black hole, though. When people defer their client-side libraries to whatever CDN, and then let that library manage whatever it pleases for dependencies, forget it. That's like saying, I don't even care if people's computers catch fire and explode, when they try to watch this awesome star field simulation.
Oh yeah, and all the ads and spyware analytics too, ya know...
As many others have mentioned, the secret sauce is probably quite simply this: No ads, no trackers, no js libraries. I use Firefox with the 'disconnect' plugin. On HN, disconnect stays completely inactive, nothing to be blocked, just good old html to be rendered with a sprinkle of css. It's beautiful, it's the way it should be.
I've found that not including Google Analytics, Twitter buttons and that sort of stuff plays a major part in loading time. Despite the fact that those scripts have the 'async' tag, they still impact loading time.
Heck, I get the feeling that they impact loading time more than my own CSS, images and Javascript. I already heavily minify and concatenate everything, as well as use SPDY for improved concurrent requests, but it seems Google and Twitter add 1-2 seconds.
Heck, I get the feeling that they impact loading time more than my own CSS, images and Javascript. I already heavily minify and concatenate everything, as well as use SPDY for improved concurrent requests, but it seems Google and Twitter add 1-2 seconds.
The fun part is when the google pagespeed tool starts nagging about google's own resources (fonts, analytics etc.) -- so meta.
It's because it's only text with little to no JavaScript, and I haven't looked at the source behind HN, but presumably, the HTML for every URL is cached upon each change (comment, upvote) and requires very little processing time per user customization.
The site is just a bunch of text without any async loading. I think all the database is cached in RAM. So a page load is just "render this tiny HTML with a few template substitutions and spit it out over the network".
I kind of wish the whole web worked that way. Yahoo was like that in the old, old days (akebono.stanford.edu).
I kind of wish the whole web worked that way. Yahoo was like that in the old, old days (akebono.stanford.edu).
I haven't done any real analysis of it, but I've always assumed it's because it renders up a single small HTML document rather than large core content and dozens of external references (ads, analysis, etc.).
Looking at this page in my browser's network analyzer, I see a request for the page itself (3.8k), followed by requests for the css and a few images which don't impact layout. On top of that, JS isn't used for layout, so the HTML renderer can do its job.
Looking at this page in my browser's network analyzer, I see a request for the page itself (3.8k), followed by requests for the css and a few images which don't impact layout. On top of that, JS isn't used for layout, so the HTML renderer can do its job.
And the JS and images are probably cached anyway, since they don't change like the rest of the design.
For your need to simulate a 256kbps connection, two things have worked for me. Both require a good few mbps connection to begin with though:
1. Use Chrome's device emulator mode. You can ignore all other settings like screen size, user agent, etc. and just use the speed restriction. 2. Use a specialized WiFi router. I use the ones from TP-Link in which I can restrict speed for a particular device IP. DD WRT based routers can also help here.
1. Use Chrome's device emulator mode. You can ignore all other settings like screen size, user agent, etc. and just use the speed restriction. 2. Use a specialized WiFi router. I use the ones from TP-Link in which I can restrict speed for a particular device IP. DD WRT based routers can also help here.
For OS X and iOS users, Apple has a "Network Link Conditioner" utility in the developer tools that can simulate different throughput as well as packet loss.
Just tried it. Looks great.
In fact, it can simulate the scenario better as compared to Chrome as the speed is shared by everything on the system and not just that Chrome tab.
In fact, it can simulate the scenario better as compared to Chrome as the speed is shared by everything on the system and not just that Chrome tab.
Server-side is a different beast than the whole server->client part.
HN does not load particularly fast for me, 300-500ms is fairly slow.
HN does not load particularly fast for me, 300-500ms is fairly slow.
Compared to most newspapers and other sites using megabytes of Javascript, HN is blazingly fast.
Whenever I find myself on a shitty connection with high response time and (moderate) packet loss, trying to read a newspaper online is just painful, whereas HN mostly loads fine anyay.
Whenever I find myself on a shitty connection with high response time and (moderate) packet loss, trying to read a newspaper online is just painful, whereas HN mostly loads fine anyay.
Same for me. I don't think HN's particularly fast. Barely faster than rockpapershotgun.com, which has lots of images, but not much. Given that it's all pure html text, I would expect more. RPS might have better caching that's closer to me though.
> Barely faster than rockpapershotgun.com, which has lots of images, but not much
rockpapershotgun.com takes 10 seconds to load entirely and serves 1.6 MB of data. Which is OK compared to others Like Polygon and Co which serve more that 5MB on their homepage and take 20sec to load on average.
rockpapershotgun.com takes 10 seconds to load entirely and serves 1.6 MB of data. Which is OK compared to others Like Polygon and Co which serve more that 5MB on their homepage and take 20sec to load on average.
> HN does not load particularly fast for me, 300-500ms is fairly slow
slow Compared to what? Show me a social network/forum that you visit everyday and loads everything under 300ms.
slow Compared to what? Show me a social network/forum that you visit everyday and loads everything under 300ms.
Flat files are very fast. The OS caches the files most of the time in memory. OS caching only becomes a problem when the hot data exceeds the memory.
By the nature of HN, there isn't that much hot data. Only the couple dozens of links in the HN front page are popular. May be couple hundreds or couple thousands people actively voting. All those don't take too much space and can fit in memory comfortably so the entire data set can be operated inside memory essentially.
The front page has 30 articles. Assuming 100k per one yields 3MB. It has about 1000 vote points. Assuming 50 bytes per vote, it's about 50k. So you can fit the entire front page data in under 4MB. That's tiny amount of data to work with.
By the nature of HN, there isn't that much hot data. Only the couple dozens of links in the HN front page are popular. May be couple hundreds or couple thousands people actively voting. All those don't take too much space and can fit in memory comfortably so the entire data set can be operated inside memory essentially.
The front page has 30 articles. Assuming 100k per one yields 3MB. It has about 1000 vote points. Assuming 50 bytes per vote, it's about 50k. So you can fit the entire front page data in under 4MB. That's tiny amount of data to work with.
I'm late to the thread, but for what it's worth: Hacker News is too damn slow.
It's fast because everything is text based. Very little CSS. Very few files to download.
The language which generates the page is irrelevant, when the page you're viewing changes rarely (doesn't change from user to user). It's very cacheable which means you might just be loading a static file served by a regular web server. Combine that with the above, and you get your answer.
Compare this to http://qbixstaging.com/Groups, what do you see?
The language which generates the page is irrelevant, when the page you're viewing changes rarely (doesn't change from user to user). It's very cacheable which means you might just be loading a static file served by a regular web server. Combine that with the above, and you get your answer.
Compare this to http://qbixstaging.com/Groups, what do you see?
Primarily it's the simplicity of what it sends back and the low number of requests.
Pre-rendered pages fed over the wire with minimal assets are fast, my blog http://benlowery.co.uk/ loads under 70ms on my home connection, it downloads 3 assets (and by far the biggest hit is the webfont).
I think we've grown so used to pages taking a couple of seconds to pull hundreds of assets and then render them that fast pages stand out :).
Pre-rendered pages fed over the wire with minimal assets are fast, my blog http://benlowery.co.uk/ loads under 70ms on my home connection, it downloads 3 assets (and by far the biggest hit is the webfont).
I think we've grown so used to pages taking a couple of seconds to pull hundreds of assets and then render them that fast pages stand out :).
PPS: I just clicked the serverfault link (I pulled it out of my address bar history without loading it, because bandwidth) and realized it was [on hold]. I don't seem to have "gotten" the Stack Exchange thing, this happens to most of my questions, heh.
So, feel free to reply here instead of the SF question if you have a response for that.
So, feel free to reply here instead of the SF question if you have a response for that.
I live in Sweden and for some reason HN takes several seconds to load for most of the times. I do not know why but it is frustrating.
I just came to know that the site is using flat files and surprised. Is that still the case ? Why was such a design selected ?
Why is loading 5k faster than loading ~500k?
Everything cached, static html pages, simple design and the big one: no tracking cookies/tracking scripts.
It does not load 10,000 trackers and 10,000 Javascript libs.
That's the whole answer.
That's the whole answer.
Without looking I can tell you it's two reasons:
1) It's text based and, therefore, can keep its total response small.
2) The page is cached on the backend so database lookups are kept infrequent and small.
Those two reasons alone make it faster than, say, 80% of most web sites.
1) It's text based and, therefore, can keep its total response small.
2) The page is cached on the backend so database lookups are kept infrequent and small.
Those two reasons alone make it faster than, say, 80% of most web sites.
Non-text sites of old weren't as slow as the current ones. Progressive rendering and not bringing in a MB of JavaScript and fonts and other shit is what makes it fast. HN could easily add images and still stay fast.
That this is even a question is just sad. It means the slow web is just accepted. Most web pages aren't particularly more functional than they were 15 years ago. Sure, maybe search boxes have auto-complete. But taking a screenshot to compare output? You could achieve much the same stuff, pixel-per-pixel, back with table-based layout. Yet even though we're running machines over 10x more powerful, things are laggy.
It's embarrassing, annoying, and dumb.
That this is even a question is just sad. It means the slow web is just accepted. Most web pages aren't particularly more functional than they were 15 years ago. Sure, maybe search boxes have auto-complete. But taking a screenshot to compare output? You could achieve much the same stuff, pixel-per-pixel, back with table-based layout. Yet even though we're running machines over 10x more powerful, things are laggy.
It's embarrassing, annoying, and dumb.
> Without looking
You risk being wrong this way. Your #1 is OK, but #2 is false: there's no database (in the sense of RDBMS or NoSQL) behind HN. It's just plain files on disk. Which, of course, makes fetching data blazingly fast, thanks to how filesystems are already handled and cached by the OS.
Anyway, I long suspected that "gzip and cache the hell out of your site" is not the best way to go about optimizing websites. From my experience websites come in two flavours: tiny with no need for optimization at all and huge, where you need to optimize everything and you know it's not going to be enough unless you're willing to rewrite some code in C.
You risk being wrong this way. Your #1 is OK, but #2 is false: there's no database (in the sense of RDBMS or NoSQL) behind HN. It's just plain files on disk. Which, of course, makes fetching data blazingly fast, thanks to how filesystems are already handled and cached by the OS.
Anyway, I long suspected that "gzip and cache the hell out of your site" is not the best way to go about optimizing websites. From my experience websites come in two flavours: tiny with no need for optimization at all and huge, where you need to optimize everything and you know it's not going to be enough unless you're willing to rewrite some code in C.
You risk being wrong this way. Unless you are involved in the site and can testify to that. In any case, the page could be, or should be, cached in some form, either on the server or with the browser.
Which makes me wonder with your second paragraph. gzipping and caching are the easiest and one of several ways to optimize the downloading of a web page. Server side code written in C plays no part in that except if the pages are created on the fly and anything else really does slow down delivery of the page; that is, page creation takes longer than 80ms or so, but a properly coded site won't let that happen if it can be helped.
And after running my own web dev business for 11 years, I didn't have to look at anything to determine any of that.
Which makes me wonder with your second paragraph. gzipping and caching are the easiest and one of several ways to optimize the downloading of a web page. Server side code written in C plays no part in that except if the pages are created on the fly and anything else really does slow down delivery of the page; that is, page creation takes longer than 80ms or so, but a properly coded site won't let that happen if it can be helped.
And after running my own web dev business for 11 years, I didn't have to look at anything to determine any of that.
I always just thought it was because Paul Graham is a wizard. some might say they don't know why that would have an effect on site loading times, but I know it does.
Speed isn't everything. Hacker News has been recently one of the most unreliable website I frequent.
The whole point of this whole thread in a nutshell is that speed is greatly underrated.
Besides this thread we had a bunch of "We discovered that adblockers clean up and speed up browsing experience by x%. Now why is that?" threads in the last month posted here. There's a reason for that.
As for reliability... I don't have complaints, honestly. Maybe I spend less time here than you.
My only gripe with HN is the search stuff they use (Algolia). It's almost unusable on my mobile, broken in Opera Mini, and just crappy on my desktop browser.
Besides this thread we had a bunch of "We discovered that adblockers clean up and speed up browsing experience by x%. Now why is that?" threads in the last month posted here. There's a reason for that.
As for reliability... I don't have complaints, honestly. Maybe I spend less time here than you.
My only gripe with HN is the search stuff they use (Algolia). It's almost unusable on my mobile, broken in Opera Mini, and just crappy on my desktop browser.
Agree, it should be improved on mobile. I would love to hear more about your desktop experience, what would be the improvement you would love to see?
What about it has been unreliable for you?
About once every few weeks, the site is hard-down for posting for several hours.
Downvote me all you want, https://twitter.com/HNStatus has all the data supporting my assertion.
HN hasn't been down for that long in a year.
it's down pretty often.
What problems are you referring to? I've never had any issues.
I understand that HN is built on top of Arc (http://arclanguage.org), a Lisp variant, and that the code (https://github.com/wting/hackernews/blob/master/news.arc) uses flat files to store post and vote data. This is generally speaking quite an unusual/unexpected architecture for a site presumed to serve a consistently moderate load, but HN manages to do so remarkably well, presumably because the site itself is so incredibly lightweight. I don't think the minimal HTML is the only factor though, although it probably counts for the majority.
I'm interested to understand as much as possible/practicable/relevant about the server(s') configuration, uplink bandwidth, other tunables, etc, so I can get an idea of what makes HN so exceptionally responsive and "a cut above 99% of everything else". Does CloudFlare really make that much of a difference? :P
I know there's some "secret sauce" in there somewhere, because virtually-verbatim clones of HN such as http://firespotting.com/ seem almost as fast... but just not as instantaneous as HN.
(PS: On the occasions I get hit by it, my ISP's shaping config seems quite involved/nuanced, and I actually want to profile it because it's so catalyzing and would be very useful to apply to my own projects for "worst-case" testing. For the interested, the details I have thus far can be found here: http://serverfault.com/questions/709529/how-can-i-profile-my-isps-bandwidth-shaping-settings)