When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config.
However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.
As far as I remember from building these things with others within the async rust ecosystem (hey Eliza!) was that there was a certain tradeoff: if you wouldn’t be able to select on references, you couldn’t run into this issue. However you also wouldn’t be able run use select! in a while loop and try to acquire the same lock (or read from the same channel) without losing your position in the queue.
I fully agree that this and the cancellation issues discussed before can lead to surprising issues even to seasoned Rust experts. But I’m not sure what really can be improved under the main operating model of async rust (every future can be dropped).
But compared to working with callbacks the amount of surprising things is still rather low :)
Some other material that has been written by me on that topic:
- Proposal from 2020 about async functions which are forced to run to completion (and thereby would use graceful cancellation if necessary). Quite old, but I still feel that no better idea has come up so far. https://github.com/Matthias247/rfcs/pull/1
> When a NOTIFY query is issued during a transaction, it acquires a global lock on the entire database (ref) during the commit phase of the transaction, effectively serializing all commits.
It only serializes commits where NOTIFY was issued as part of the transaction, right? Transactions which did not call NOTIFY should not be affected?
I don't get that part. If it can receive an event/delegate, it means the object needs to be referenced by another object which invokes the event. If that is the case - how would be eligible for GC at all?
Vancouver BC might be in the same range now, where things like this show up for 2M CAD. And the conversion rate won't make up for the differences in salaries
Gravel bike was my favorite purchase in the last year too.
I was already mountain-biking for the last 15 years. Things got more and more gravity oriented, where rides mostly had been one long uphill and technical trails down. While this is pretty cool it got a bit monotonic over time.
With the Gravel bike I enjoy doing far longer rides and checking out places I hadn't been before. 70km flat on an enduro mountain-bike are rather cumbersome, but fun on the faster gravel bike.
Germany guy living in Canada at the moment: German internet now seems super cheap for me. I had UnityMedia for years. 120Mbit/s and unlimited home phone for 25€ per month. You can triple that sum for Canada (90$ plus tax) if you don’t get any special deal. And deals are annoying, since internet providers randomly increase their prices even if you are on a contract. So even if are not constantly out on the watch for the next deal and complain regularly at customer service you will simply pay a lot.
Besides that the Canadian plans are capped (E.g. at 1 TB/month even at high prices), whereas I never had a capped plan in Germany.
Mobile phone plans are even worse, without even talking about coverage.
So yes, Germany is not great, but it can be worse.
That’s right. In addition: CBOR can’t automatically compress field names, since those are strings which need to get fully serialized. gzip can compress them too, so it has a chance to trim the size of data further down in exchange for the additional cost of a 2nd encoding.
Cap‘N‘Proto, protobuf and co can replace field names by IDs as indicated through Schemas and will thereby most space efficient in general.
If you want it to be infinitely streaming compatible (which CBOR is) it raises another question: For how long are identifiers valid, and do they get invalidated or updates at some point of time. The header compression in HTTP2 solves such a problem, but also introduces quite a bit of additional complexity
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.