Server is smoking, trying to figure out why(twitter.com)
twitter.com
Server is smoking, trying to figure out why
https://twitter.com/HNStatus/status/1401980524428627970
62 comments
> It's a difficult habit to quit
LOL
LOL
> Something went wrong. Try reloading.
Seems like Twitter isn't doing much better.
Seems like Twitter isn't doing much better.
I interpreted this much more literally. When the updated tweet about a code revert fixing it came out, it took me a second to figure out what was meant.
Update provided
> Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry!
https://mobile.twitter.com/HNStatus/status/14020112741043609...
> Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry!
https://mobile.twitter.com/HNStatus/status/14020112741043609...
Besides applying security patches, what exactly has needed to be changed in the HN codebase over the last 14 years? Featurewise, it's been essentially static for its whole existence (which is a large part of its appeal to many, including myself).
I think I've noticed changes a few times. For example,
- the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point.
- we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code.
- code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs.
- the "past" link in the navbar was added
- the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating.
These are relatively recent changes.
- the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point.
- we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code.
- code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs.
- the "past" link in the navbar was added
- the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating.
These are relatively recent changes.
> the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation.
Hasn't been a problem for me (my browser (Firefox on Linux) matches ASCII dashes against en dashes and em dashes too), but you could try using a Compose key shortcut? On Linux at least, the Compose key shortcuts[1] for en dash and em dash are easy: minus-minus-period and minus-minus-minus respectively.
1: https://fsymbols.com/keyboard/linux/compose/
Hasn't been a problem for me (my browser (Firefox on Linux) matches ASCII dashes against en dashes and em dashes too), but you could try using a Compose key shortcut? On Linux at least, the Compose key shortcuts[1] for en dash and em dash are easy: minus-minus-period and minus-minus-minus respectively.
1: https://fsymbols.com/keyboard/linux/compose/
> Hasn't been a problem for me (my browser (Firefox on Linux) matches ASCII dashes against en dashes and em dashes too)
Huh, you're right. I could've sworn that didn't work. I wouldn't have even noticed the character change otherwise. Maybe it started working again when Firefox added the ability to ignore diacritics. I guess that goes further than diacritics and works by general transliteration to ASCII.
Huh, you're right. I could've sworn that didn't work. I wouldn't have even noticed the character change otherwise. Maybe it started working again when Firefox added the ability to ignore diacritics. I guess that goes further than diacritics and works by general transliteration to ASCII.
There's also the "second chance" list (few months ago?)
This one, right?
https://news.ycombinator.com/pool
Doesn't seem to be linked anywhere, though. I wonder if there are other pages like that.
https://news.ycombinator.com/pool
Doesn't seem to be linked anywhere, though. I wonder if there are other pages like that.
In the footer is "Lists", which goes here and includes that: https://news.ycombinator.com/lists
Others have added more here in 2009 that aren't in either place: https://news.ycombinator.com/item?id=1024293
Others have added more here in 2009 that aren't in either place: https://news.ycombinator.com/item?id=1024293
[deleted]
> the subthread collapse link [...] is my least favorite change because [...]
To each their own, but yours was to me a fun practical example of that famous xkcd strip called "workflows" that is sometimes linked in HN, about how every possible change is seemingly always able to break someone's way of using your product :)
To each their own, but yours was to me a fun practical example of that famous xkcd strip called "workflows" that is sometimes linked in HN, about how every possible change is seemingly always able to break someone's way of using your product :)
[deleted]
When you hit scale every edge case becomes a reality.
If the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.
[deleted]
Automated spam/voting ring detection and ranking systems.
Ajax upvotes, upvote used to be just a link iirc.
If they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.
My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things!
This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
Does fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
Fixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.
Nicotine is a hell of a drug.
If you think about it, it's impressive how such a high traffic site works out using a niche language which what I assume is not such a big ecosystem.
(On the other hand there's moderate complexity but low page sizes and nothing like modern websites with hundreds of cookies, heavy js, etc)
(On the other hand there's moderate complexity but low page sizes and nothing like modern websites with hundreds of cookies, heavy js, etc)
something about deep comment thread folding?
Yesterday I saw a strange artifact in a comment. Instead of the usual "[-]", I saw "[2 more]". Did not seem right.
I also saw that yesterday. What's worse, the only way to expand the comments hidden behind that seems to be to temporarily unblock Javascript, reload the page, and click on that.
That's what it says after you use the [-] to collapse a comment and it has a single reply. It also counts replies to replies (and so on).
[deleted]
But I do not use Javascript and I only ever see "[-]".
I use a text-only browser and all comments are left-justified. A flat view, no hidden comments. Looks very different than in a browser that runs Javascript. Seeing "[2 more]" seemed unusual.
I use a text-only browser and all comments are left-justified. A flat view, no hidden comments. Looks very different than in a browser that runs Javascript. Seeing "[2 more]" seemed unusual.
I really thought it was going to be someone complaining about their restaurant server having a smoke break. Thought it must have been covid related to make the front page. But don't ask me how.
I had an older Dell give up the magic smoke recently and literally - https://i.imgur.com/WHYODQS.jpeg
Amusingly enough I smelled it weeks ago and it finally shut off a few days ago.
Amusingly enough I smelled it weeks ago and it finally shut off a few days ago.
Those are buck converters, the MOSFETs in them tend to die either due to overheating or being run at the edge of design limits for long enough. It's rare that they die excitingly, however.
Yeah I've been paged before due to a crashed server failing ping and found out the datacenter tech was already on the job because it was suffering from actual smoking. I was remote so I don't know what they did, presumably pull the power and hit it with a CO2 extinguisher?
If it’s any consolation, I thought my wireless provider had booted me before I realized HN could be having issues. It’s generally so fast and reliable that I use it as my “is my internet broken?” sanity check page.
Ahahah, similar situation here. I thought maybe our IT department had noticed my activity here and decided to throttle my connection (which would still be an odd solution instead of just outright blocking it), but still it took me a few minutes to suspect that maybe the problem was with HN itself.
Also assumed this, was only convinced otherwise when I ran into the same issue on my personal mobile phone, on a data network separate from my work network.
It was worse for me, as I had just replaced my router!
Same here. :-)
"Smoking" meaning running high CPU, presumably, not literally smoldering
Was the last outage really in 2015
No, but three months ago it wasn't their fault: https://news.ycombinator.com/item?id=26469116
I thought Replit started ddosing the site
My google foo failed and I wasn't able to find the correct @hnstatus account when I was having trouble. Perhaps a ”status" link in the footer would make this easier to find?
BeOS came up with an function for this: https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_Sys... but “Smoldering doesn't count.”
I don’t mean to be rude, but is HN a refutation of the Lispy vision it was meant to demonstrate? There has been a bug where comment pages can only show a limited number of entries for years, and the wonderful mods like dang continually report it is being worked on. It’s hard for me to believe it wouldn’t be fixed much faster if it occurred in a more practical language.
> There has been a bug where comment pages can only show a limited number of entries for years, and the wonderful mods like dang continually report it is being worked on.
It's not a bug per se, it's a concession to conserve server resources, which will be removed once some optimization work has been completed.
You might respond "well why is HN so resource-hungry?" Answer: it's not! HN is run off a single core of a single CPU.
https://news.ycombinator.com/item?id=23187455
> The software for both HN and YC was just a single Arc program (and not a large one) for the first 9 years of their existence, during which they went from nothing to massively successful to industry-changing. Written by one person, programming part-time. That is a staggering achievement. The power of using the right language for your project goes far further than most people dream. Our imagination about this is crippled by path dependence, social proof, and the conditioning that comes from only ever doing things the same few ways, like those fish in experiments (which may be urban legends?) who stick to their corner of the aquarium even after a glass barrier has been removed. The solution space of software and programming is so much larger than most of us want to imagine that it is. Sad.
It's not a bug per se, it's a concession to conserve server resources, which will be removed once some optimization work has been completed.
You might respond "well why is HN so resource-hungry?" Answer: it's not! HN is run off a single core of a single CPU.
https://news.ycombinator.com/item?id=23187455
> The software for both HN and YC was just a single Arc program (and not a large one) for the first 9 years of their existence, during which they went from nothing to massively successful to industry-changing. Written by one person, programming part-time. That is a staggering achievement. The power of using the right language for your project goes far further than most people dream. Our imagination about this is crippled by path dependence, social proof, and the conditioning that comes from only ever doing things the same few ways, like those fish in experiments (which may be urban legends?) who stick to their corner of the aquarium even after a glass barrier has been removed. The solution space of software and programming is so much larger than most of us want to imagine that it is. Sad.
A post from a week or two ago about APL and cellular automata (I cannot find it) perhaps illustrated OP's sentiment better: it's easy for a language to do something easily and elegantly, but quickly become fantastically difficult to accomplish with the introduction of new—and sometimes mundane—constraints. The post showed how it was trivial to implement certain cellular automata in APL, but massively difficult to do when the automata became slightly more complex (while being trivial to update a Java implementation to accommodate the new complexity).
The practical difference between running on one server with one core and five servers with four cores is nothing—to YC it's a rounding error either way. If there's a meaningful problem (e.g., comments being computationally expensive to paginate well) that the language/runtime/system/etc cannot solve easily enough for it to be remedied in O(months), it does call into question whether the choice of language is the right one. Yes, it can be patched over by a human moderator. But it still highlights a significant weakness in the technology's core function.
Perhaps ironically, most YC companies (and by consequence, YC) would fail miserably if they chose purism and principle over pragmatism when it came to technology decisions.
The practical difference between running on one server with one core and five servers with four cores is nothing—to YC it's a rounding error either way. If there's a meaningful problem (e.g., comments being computationally expensive to paginate well) that the language/runtime/system/etc cannot solve easily enough for it to be remedied in O(months), it does call into question whether the choice of language is the right one. Yes, it can be patched over by a human moderator. But it still highlights a significant weakness in the technology's core function.
Perhaps ironically, most YC companies (and by consequence, YC) would fail miserably if they chose purism and principle over pragmatism when it came to technology decisions.
Here's the post:
https://news.ycombinator.com/item?id=27231978
> If there's a meaningful problem (e.g., comments being computationally expensive to paginate well) that the language/runtime/system/etc cannot solve easily enough for it to be remedied in O(months), it does call into question whether the choice of language is the right one.
I could be wrong, but I get the sense that HN doesn't have any full-time developers. In fact, I think it might be entirely dang—he's talked before about how moderating HN leaves him with very little HN development time.
I could be wrong, but I get the sense that HN doesn't have any full-time developers. In fact, I think it might be entirely dang—he's talked before about how moderating HN leaves him with very little HN development time.
On the other hand, I see spam and flamewars very rarely, and I know they have protection in place against it. I'd take that over seeing all the comments in one page.
With Arc and HN and now Bel, I think pg is still exploring what a system could look like without major compromises to accommodate the limits of today’s hardware. For example, the first cut at pagination put something like a continuation ID into a URL, which only worked until that continuation was GC’d or the one and only server process got bounced.
If you wanted a highly scalable HN and nothing more, you’d let Postgres or some graph database do the heavy lifting, add a Racket or Clojure frontend, and call it a day. But that’s pretty boring plumbing work, and doesn’t get you closer to the hundred-year language he’s aiming for.
If you wanted a highly scalable HN and nothing more, you’d let Postgres or some graph database do the heavy lifting, add a Racket or Clojure frontend, and call it a day. But that’s pretty boring plumbing work, and doesn’t get you closer to the hundred-year language he’s aiming for.
What's the bug? Isn't there a "more..." At the bottom of those long comment pages?
I wouldn't necessarily call it a bug, but the way the pagination works does not work well for active threads which is probably why they want to replace it at some point with code than can handle long threads on a single page.
A problem with pagination as implemented on HN is that the algorithm to display page N seems to be functionally (no pun intended!) equivalent to this:
So you read page 1 of an active discussion. You get to the end of the page and hit "more". But while you were reading page 1, other people were commenting, and other people were voting. Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2.
Or maybe what was just after page 1 when you were reading page 1 has moved up due to voting, and when you go to page 2 you end up past it.
It's like if you had a book that went through several editions, with significant changes between editions (including rearranging the order of chapters), and someone put together a printing by taking the first 10 pages from the first edition, the second 10 pages from the second edition, and so on.
This could be fixed while retaining pagination by saving a timestamp when you go to page 1, and then when you go to subsequent pages use a snapshot of the comment database from that timestamp instead of the current comments. If you want to update to the current comments you'd have to hit refresh or go back to page 1.
Another approach while retaining pagination would be keep track of what top level comments you've seen. If when rendering page N it sees that due to comment order changes one of the comments that is going to end on the page is one you've already seen it could omit it. Or maybe omit it unless the tree starting at that comment has changed since you last saw it. (If going this route, I'd also suggesting adding something to the UI to highlight what has changed in the tree and/or dehighlight what you've already seen).
That second approach could still be annoying. You could still get stuck on a single comment that is getting a lot of relies. That could be fixed, but then you are getting into probably needing to add new controls or settings.
I'm sure they have thought of both of those approaches and more, but probably want to keep the interface clean and simple and retain minimal state, hence the plan to someday make it handle very long comment threads on a single page.
A problem with pagination as implemented on HN is that the algorithm to display page N seems to be functionally (no pun intended!) equivalent to this:
Render all the comments as one long virtual page
Split it into smaller pages
Render the Nth page from that set of smaller pages
So you read page 1 of an active discussion. You get to the end of the page and hit "more". But while you were reading page 1, other people were commenting, and other people were voting. Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2.
Or maybe what was just after page 1 when you were reading page 1 has moved up due to voting, and when you go to page 2 you end up past it.
It's like if you had a book that went through several editions, with significant changes between editions (including rearranging the order of chapters), and someone put together a printing by taking the first 10 pages from the first edition, the second 10 pages from the second edition, and so on.
This could be fixed while retaining pagination by saving a timestamp when you go to page 1, and then when you go to subsequent pages use a snapshot of the comment database from that timestamp instead of the current comments. If you want to update to the current comments you'd have to hit refresh or go back to page 1.
Another approach while retaining pagination would be keep track of what top level comments you've seen. If when rendering page N it sees that due to comment order changes one of the comments that is going to end on the page is one you've already seen it could omit it. Or maybe omit it unless the tree starting at that comment has changed since you last saw it. (If going this route, I'd also suggesting adding something to the UI to highlight what has changed in the tree and/or dehighlight what you've already seen).
That second approach could still be annoying. You could still get stuck on a single comment that is getting a lot of relies. That could be fixed, but then you are getting into probably needing to add new controls or settings.
I'm sure they have thought of both of those approaches and more, but probably want to keep the interface clean and simple and retain minimal state, hence the plan to someday make it handle very long comment threads on a single page.
> Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2.
Reddit pagination works the same way. When you click the next page link the order may have moved things around and you might see entries you have already seen.
Reddit pagination works the same way. When you click the next page link the order may have moved things around and you might see entries you have already seen.
I believe Reddit pagination only updates once every half hour... or at least that's what I heard about /all. Perhaps HN has more continuous updates resulting in more problems (the main page certainly feels more dynamic on HN.)
I wonder what database or file schema they use. Many would find that interesting, given the scale that hn has reached.
> Something went wrong. Try reloading.