When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config.
However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.
As far as I remember from building these things with others within the async rust ecosystem (hey Eliza!) was that there was a certain tradeoff: if you wouldn’t be able to select on references, you couldn’t run into this issue. However you also wouldn’t be able run use select! in a while loop and try to acquire the same lock (or read from the same channel) without losing your position in the queue.
I fully agree that this and the cancellation issues discussed before can lead to surprising issues even to seasoned Rust experts. But I’m not sure what really can be improved under the main operating model of async rust (every future can be dropped).
But compared to working with callbacks the amount of surprising things is still rather low :)
Some other material that has been written by me on that topic:
- Proposal from 2020 about async functions which are forced to run to completion (and thereby would use graceful cancellation if necessary). Quite old, but I still feel that no better idea has come up so far. https://github.com/Matthias247/rfcs/pull/1
> When a NOTIFY query is issued during a transaction, it acquires a global lock on the entire database (ref) during the commit phase of the transaction, effectively serializing all commits.
It only serializes commits where NOTIFY was issued as part of the transaction, right? Transactions which did not call NOTIFY should not be affected?
I don't get that part. If it can receive an event/delegate, it means the object needs to be referenced by another object which invokes the event. If that is the case - how would be eligible for GC at all?
If you are open to other things, I wouldn't worry about it. Programming languages are tools, and learning others tools isn't that hard. People can jump from C++ to Rust, Java or C# in a small time. And this especially applies if people are coming from more powerful languages like C++ - so that other language features are often more a subset than a superset.
I won't comment on the rest of the blog post, but I can fully relate to this point:
> Despite using golang, working with the absolute impenetrable monster that is kubernetes, taking on a helpdesk like on-call experience for one week at a time tending to configuration files over code, and not touching C++ at all, I feel more rewarded in my work than I ever did writing any amount of C++ in my entire life.
I actually experienced the same in the last couple of years. Doing ops work, and sometimes really tiny code changes which still help to improve a product and get recognized by people can feel really rewarding. Standardization work or any kind programming language / library development sometimes not that much (and that is independent of C++).
I think the reason for this is that in ops/support/etc environment one is working towards a known goal, which is well understood by people (peers, customers). If the goal has been reached everyone is happy, and one will likely get positive feedback. And even on the road towards the goal people will understand why it's important to get something done and help to get to the goal. E.g. code changes might not have to be perfect in order to get merged, and there might not be an extreme amount of bikeshedding.
Working on fundamentals can be different: In a setting like a standard library working group often a variety of people come together which do not share the same goal, and where goals are also not necessarily "customer driven". Instead of that, people more often have their personal goals they want to achieve, and they have their own set of motivations for it. This makes it harder to get buy-in from others. Goals are also not that quantifieable (like "fix a crash" or "improve latency by 10ms"), which makes it additionally harder to argument why something is important or less important.
Vancouver BC might be in the same range now, where things like this show up for 2M CAD. And the conversion rate won't make up for the differences in salaries
Gravel bike was my favorite purchase in the last year too.
I was already mountain-biking for the last 15 years. Things got more and more gravity oriented, where rides mostly had been one long uphill and technical trails down. While this is pretty cool it got a bit monotonic over time.
With the Gravel bike I enjoy doing far longer rides and checking out places I hadn't been before. 70km flat on an enduro mountain-bike are rather cumbersome, but fun on the faster gravel bike.
> I'm suprised it wasn't something native like QT.
The question is what you really would gain from using QT compared to Chrome/HTML/JS.
Performance won't really matter for this. It it's good enough on the hardware they have, then further improving it does not make a difference. Provided they are running everything really mission-critical and real-time outside of the UI and on an RTOS or specialized hardware anyway (I really really hope so).
One advantage might be audibility and the hope for less defects due to a smaller codebase and less dependencies. But I think even QT is already far beyond that point. And since Chrome is nowadays wider deployed, it might have a higher level of maturity than eg. the QT/QML tech.
It's interesting to hear. I have not seen a lot of Macbooks in any of the companies I worked for nor the one we partnered with in germany. At least in software development, in creative domains (photo, video) things are different.
In north america Macbooks seem to be more or less the default hardware (also mainly my personal impressions - since I moved there).
Just a guess - but you are in the US, right? In most other places of the world I have not seen macbooks being used for non Apple related development tasks due to pricing.
Germany guy living in Canada at the moment: German internet now seems super cheap for me. I had UnityMedia for years. 120Mbit/s and unlimited home phone for 25€ per month. You can triple that sum for Canada (90$ plus tax) if you don’t get any special deal. And deals are annoying, since internet providers randomly increase their prices even if you are on a contract. So even if are not constantly out on the watch for the next deal and complain regularly at customer service you will simply pay a lot.
Besides that the Canadian plans are capped (E.g. at 1 TB/month even at high prices), whereas I never had a capped plan in Germany.
Mobile phone plans are even worse, without even talking about coverage.
So yes, Germany is not great, but it can be worse.
That’s right. In addition: CBOR can’t automatically compress field names, since those are strings which need to get fully serialized. gzip can compress them too, so it has a chance to trim the size of data further down in exchange for the additional cost of a 2nd encoding.
Cap‘N‘Proto, protobuf and co can replace field names by IDs as indicated through Schemas and will thereby most space efficient in general.
If you want it to be infinitely streaming compatible (which CBOR is) it raises another question: For how long are identifiers valid, and do they get invalidated or updates at some point of time. The header compression in HTTP2 solves such a problem, but also introduces quite a bit of additional complexity
Indeed. Using plain structs one can't just add a single field with a reasonable default and still be backward compatible - which is a very very common case in API design.
You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work.
Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.
If an echo would require 114Mbps, it would barely work for any household, since internet with that speed is far from common. The real required bandwidth should be in the region that is required for transmitting a bit of audio (< 10mbps), and only if audio is active.
Technically it's not even required for that. HTTP/1.1 bodies can be streamed just fine. The main reason not to do this in a browser is that one real TCP connection would be required for each stream, and the number of concurrent outgoing connections from a website is strictly limited. With HTTP/2 this problem goes away.
The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".
The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.