the scheduler is probably simpler if coroutines can't bounce between cores. you can have a single thread per core that runs the scheduler to multiplex a bunch of coroutines on the single thread, which lines up with the example linked in the repo to a redis clone. redis runs a single thread (i think technically it has some multithreading stuff now but that was the model for a while) but can concurrently process a bunch of requests like when 1 or more requests are blocked on blpop, a pubsub thing, xread with block arg given, etc. nginx does something similar with forking a process per core that runs single threaded (ignore the threading for stuff that reads from disk).
i got caught up on that statement too. i interpreted it as they can spin up more servers with their own volumes but thats not really 1 ebs volume anymore. maybe at their current load they only need 1? the op mentioned copying stuff to s3 at a certain size so it sounds like the disk isn't very big at any moment. i don't think there would be much to do if another server joins.
> Amazon EBS offers a higher durability volume (io2 Block Express), that is designed to provide 99.999% durability with an annual failure rate (AFR) of 0.001%, where failure refers to a complete or partial loss of the volume.
if they take snapshots often enough to feel comfortable with that low failure rate, it does seem kind of reasonable to me. really low risk of a given volume failing.
if i'm reading the op right, they kind of use ebs as a buffer for fresh data until it ages out to s3. they use a "local" disk to hold the stuff used by the queries that people actually make and the queries run quick. they let the old stuff rot in s3 where its almost never used. that sounds like a good idea to save money plus the stuff that's done often is fast.
the ebs slas look reasonable to a non expert like me and you can take snapshots. it sound like you need to be careful when snapshotting to avoid inconsistencies if stuff is only partially flushed to disk. so you'd need to pause io while it snapshots if those inconsistencies matter. that sounds bad and would encourage you to take less frequent snapshots...? you also pay for the snapshot storage but i guess you wouldn't need to keep many. i like that aws defines "SnapshotAPIUnits" to describe how you get charged for the api calls.
with aurora, it looks like you can synchronously replicate to a secondary (or multiple secondaries) across azs in a single region. it sounds nice to have a sync copy of stuff that people are using. op says the'yre ok with a few seconds of data loss so i'm wondering how painful losing a volume right before taking a snapshot would be.
i wonder if anything off the shelf does something similar. it sounds like people are suggesting clickhouse. i saw buffer table in their docs and it sounds similar https://clickhouse.com/docs/en/engines/table-engines/special.... it looks like it has stuff to use s3 as cold storage too. i even see geo types and functions in the docs. i've never used clickhouse so i don't know if i'm understanding what i read, but it sounds like you could do something similar to whats described in the post with clickhouse if the existing geo types + functions work and you are too lazy to roll something yourself.
yeesh i didnt realize there was a patent on that. your link lines up with the authors predictions of cdns coming up with and patenting different ways to accomplish this. i guess it makes sense and maybe it isn't a bad thing, but i never thought about it. i wonder how anyone infringing on that is even caught? is something like that ever enforced?
edit: i was blind and missed that it is marked as abandoned "ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION". i never look at these so maybe thats a normal ending, but it is interesting to me.
this guy sounds uhhh exactly the same https://patents.google.com/patent/US11706292B1/en...? reading these brings up the duration of a connection as a relevant dimension. in my bubble i've only dealt with relatively short lived conns. long lived conns must be more likely to get caught up in routing changes so a redirect to unicast can help.
the "Stupid DNS Tricks" section says using dns to map clients to a nearby pop is a trick. they predicted this trick would be used for decades and it seems like they were right. i know cloudfront uses this in some fancy form.
i don't know if i'd call it a trick though. if you have multiple pops, dns feels like a natural place to control what traffic goes to which pop. you will need resolvers to be well behaved which will never be the case. not all will respect ttls and use the client subnet extension but a lot do. dns gives you a nice knob and hooks to apply rules to control the traffic to each pop. this paper i think describes the idea well https://www.sigcomm.org/sites/default/files/ccr/papers/2015/....
if using dns like this is a trick, what is the right way to map a client to 1 of multiple pops? anycast?
the SHED_LOAD thing offered by ns1 looks cool. anyone have experience with it? https://help.ns1.com/hc/en-us/articles/360020683413-Configur.... seems like something juicy missing from route53. ns1 even has an integration with cloudwatch (not to say cloudwatch is great) as a source for the load stats.
it seems obvious at the level of like an individual load balancer to consider the load of the nodes backing an upstream when it decides which one to send reqs to. i didn't think of lifting that to the record that points to the load balancers.
i'm trying to think how you'd do it with route53. maybe with health checks that take load into account? i don't think you'd get the low and high watermarks like in ns1s thing. maybe a weighted record with something wired up to to tweak weights based on load? sounds just annoying enough to get right and working safely that if route53 implemented it well, people would pay for it...
it has decent lua hooks to customize behavior but i use it in the dumbest way possible to hammer a server at a fixed rate with the same payload over and over.
i run it by hand after a big change to the server to make sure nothing obviously regressed. i used to run it nightly in a jenkins job but 99% of the time no one looked at results. it was nice to see if assumptions on load a single node could handle didn't hold anymore.
your library looks well written and clean. thanks for sharing.
if anyone else wants to go coroutine spelunking, these were interesting to me:
https://github.com/higan-emu/libco/tree/master
https://github.com/Tencent/libco
https://github.com/hnes/libaco
https://kernel.googlesource.com/pub/scm/virt/kvm/qemu-kvm/+/...
https://tia.mat.br/posts/2012/09/29/asynchronous_i_o_in_c_wi...
https://www.cs.uml.edu/~bill/cs516/context_paper_rse-pmt.pdf