That is kind of the point of this repo. If your API implementation is too expensive for lambda move the same container to run on EKS/ECS fargate or ec2. The opposite holds as well - if you can save money by moving it from fargate to lambda then you can use the same container.
Ion has a binary format but is not specifically about real-time streaming. It is a JSON replacement.
Ion originated 10+ years ago from the Amazon catalog team - the team that kept data about the hundreds of millions of items available on Amazon. Nearly every team in the company called the catalog to get information about items all the time - scanning the entire catalog, parts of the catalog, millions of individual item lookups every second, etc.
They did the math and some very large percentage of network traffic in Amazon Retails data centers was catalog data. If that data, currently in XML or JSON format, was sent in a more compact format it would save some ridiculous millions of dollars every year. So Ion was born and eventually open sourced.
The intent of the stored schema isn't really for self-description. A typical use case for Avro is data storage over long periods of time.
It is expected that the schema will evolve at some point during this time. Therefore you still need to specify a target schema to read the data into which is allowed to be different than the stored schema. Avro then maps the stored data into the target schema by using the stored schema. Most avro libraries expect you to get the target schema from a separate source before reading data.
Ion has functions to turn Ion into JSON which will, of course, lose information. Annotations are dropped, decimals turn into a JSON number type which may lose precision, etc.
That's not true. Avro at least existed at the time. However they wanted something self-describing to replace JSON/XML usage. Avro is better suited as a data storage format rather than a transit oriented format. Of course, both Ion and Avro can be used for either, but Avro will give you better compression on disk, but Ion is less cumbersome since it doesn't require a schema
Finally! I've had to live the JSON nightmare since I left Amazon.
Some of the benefits over JSON:
* Real date type
* Real binary type - no need to base64 encode
* Real decimal type - invaluable when working with currency
* Annotations - You can tag an Ion field in a map with an annotation that says, e.g. its compression ("csv", "snappy") or its serialized type ('com.example.Foo').
* Text and binary format
* Symbol tables - this is like automated jsonpack.
* It's self-describing - meaning, unlike Avro, you don't need the schema ahead of time to read or write the data.
I realize your previous comment was more in reference to transient network partitions so my comment is out of place. But whether the mechanism is manual or automatic once a network partition is discovered the reshuffle begins.
Really it doesn't matter and your point only goes to show one of Cassandra's flaws. When either a network partition or a server failure happens Cassandra starts reshuffling data amongst multiple hosts and filling network pipes. Contrast this with a setup where you have a static partitioning of hosts to partition and a leader per partition. Then you only need to (possibly) elect a new leader and carry on.
This is especially relevant when you need to do these things because of unexpected load increases or the loss of hosts in your cluster.
What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook (DynamoDB from AWS is actually not based on Dynamo tech except in terms of what not to do).
It doesn't. As an average developer it's my public/private cloud providers job to take my images/dockers and schedule them correctly on whatever underlying fleet of hardware they run.
I can only assume Tectonic and related tech are aimed at those developers who run hybrid or private clouds for the rest of their company.
Don't get too excited. This proposal is more a codification of the "easy" part of managing dependencies for top level/binary packages that much of the community already does.
This proposal does not (and isn't intended) to cover how dependencies work when vending libraries and does not address shared package and diamond shaped dependencies.
That's not to say having a standard for vendoring isn't a good thing.
Nothing is magic but in this case moving from a language with a GIL to a language built for (single machine) concurrency probably scales a lot.
The OP says as much in the comments:
"For us, one of the biggest latency wins comes from the fact that go can truly execute sql statements in parallel (whereas python's GIL serialized these parallelizable operations). In general, single-threaded go is at least 5x faster than pure python (without c-module)."
"If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical."
This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typically, any future causes of the service process dying are hidden unless someone happens to be looking at the logs one day.
I would love to see smart ways to surface "this will be a problem soon" on alerting systems.