rsanders

w zeszłym rokudiscuss

I'd say that medicine has been a net benefit to bettering the lives of those who have access to it. I'm biased, though - "drugs" have kept me alive and able for the last 30 years.

rsanders

w zeszłym rokudiscuss

The gray market for peptides is pretty fascinating. I've spent the last month exploring it and interacting with some of the Chinese entrepreneurs involved, as well as the informal regulatory functions provided by self-organizing groups of buyers who pool their resources to test the products and share experiences. It's a surprisingly wholesome kind of shady.

An enterprising shopper can stock years worth of semaglutide or tirzepatide, or even the newer medicines still in trials like cagrilintide, retatrutide, mazdutide, survodutide, et cetera -- for a monthly treatment cost of less than $100. Heck, you can get max dose sema for $5/month. Which, putting aside the legal/IP and risk issues for a moment, is a good thing for the people who will likely be on these medicines for the rest of their lives.

Having a less obese population would have so many positive impacts on our quality of life, healthcare system, and economy. I hope that somehow these medicines become cheap and plentiful enough that people can take them prophylactically.

rsanders

2 lata temudiscuss

I'm pretty new to Rust so forgive me if I'm mistaken, but it seems to me that this crate doesn't require the use of tokio.

rsanders

2 lata temudiscuss

Which ones would you recommend?

rsanders

5 lat temudiscuss

Swapping in a multi-vendor world also implies standardization. It’s too early in this technology to standardize - that would stifle progress.

Swap stations would also be far more expensive to build and manage inventory for, and would carry higher liability for whatever automated mechanism moved the thousand pound pack packs around when it e.g. went out of alignment and crushed somebody’s car frame.

rsanders

6 lat temudiscuss

If CSV were being used just to exchange data with Excel, we probably wouldn't be using CSV. Many systems neither need nor know that ="01" should be treated as the string "01".

If Excel were the only intended consumer, .xlsx would be a preferable file format. At least it's mostly unambiguous.

rsanders

6 lat temudiscuss

That's a good point. But then you're not benchmarking C++ as a distinct language. So what would sufficiently distinguish a C++ program from a C program? Let's assume it's not just minor incompatibilities introduced to prevent compilation by a C compiler.

There must have used some definition that is not explicit in the paper, but you can see in this code sample that the author used various C++ standard data types (std::string, std::array), iterators, classes, concurrency (std::thread). I'm no judge of C++ style, but perhaps it's "C++ as a C++ developer circa 1997 would have written it".

https://github.com/greensoftwarelab/Energy-Languages/blob/ma...

rsanders

6 lat temudiscuss

Ha! I picked up the original project (dosemu1) from an abandoned skeleton created by Matthias Lautner. I was a larval programmer and Linux was very much in its infancy. I really wanted to run Civilization on my one computer without keeping the DOS partition around -- staying on 24/7 Linux was a point of pride back then. While I got quite a few programs running well under dosemu, I never had much time to play Civilization after picking up the project.

Thankfully, when I got out of my depth and then joined a fledgling new ISP (MindSpring) as the first engineer, the worthy James MacLean took over and turned it into a solid system. Not sure why I'm not listed in the THANKS, but perhaps it's revenge for the poor code quality I left behind. :-)

It was one of the most enjoyable projects of my career, despite or perhaps because of how little I knew and how much I had to learn.

- root @ hrothgar

rsanders

6 lat temudiscuss

Can you expand on this: "AWS healthchecks each kubernetes node, but not your pods themselves".

Are you talking about a keepalive connection to an unhealthy pod which is reused for multiple requests? So the failure modes are, if I understand you correctly, a) the ALB keeps sending requests through an established keep-alive HTTP connection which terminates in an unhealthy pod, but which it sees as healthy because the node is healthy and can route traffic to another, healthy pod, and b) the health of an established HTTP keepalive connection is perceived to be that of the node rather than the destination pod, so nodes which become unhealthy can cause the ALB to unnecessarily terminate a keepalive connection.

We had to switch to using target-type=instance because of issues with pods not being deregistered. I'd prefer to use target-type IP but it seemed like preventing 500s on rollouts required a bit of testing and tuning with a very specific approach. e.g. introducing a longish delay on pod termination with a lifecycle hook and using the pod readiness gate support recently added to alb-ingress-controller.

rsanders

6 lat temudiscuss

Fewer features and fewer lines of code, and those LOC are written in Go, which is the language in which CockroachDB is written and which, presumably, for which their team and tooling are best optimized. It's a reasonable thesis.

rsanders

6 lat temudiscuss

I wrote a little fork+spinloop program w/100 subprocesses and deployed it with a low (100m) CPU request and no limit. It's certainly driving CPU usage to nearly all 8 of the 8 cores on the machine, but the other processes sharing the node are doing fine.

Prometheus scrapes of the kubelet have slowed down a bit, but are still under 400ms.

Prometheus scrape latency for the node kubelet has increased, but not it's still sub-500ms.

Note that this cluster (which is on EKS) does have system reserved resources.

    [root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/system.slice/cpu.shares
    1024
    [root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/kubepods/cpu.shares
    8099
    [root@ip-10-1-100-143 /]# cat /sys/fs/cgroup/cpu/user.slice/cpu.shares
    1024

rsanders

6 lat temudiscuss

That may be more likely with limits, but it doesn’t require a limit. I’ve had lots of fun with that in Elasticsearch pods with no limit. And then you get to enjoy a nice cascading failure.

rsanders

6 lat temudiscuss

I should have said that it seems like the wrong fix to the problem. But I have since learned that limits can cause excessive throttling. And of course you may want your pods to be burstable, but that would just be a question of setting appropriate limits.

Live and learn!

rsanders

6 lat temudiscuss

Can you explain how having a CPU limit set (at any level) has a negative effect on latency? That's an important factor to understand.

The arguments for allowing containers to burst makes plenty of sense to me. I do it on most of my services!

thockin's reddit post for reference: https://www.reddit.com/r/kubernetes/comments/all1vg/on_kuber...

Another interesting bit of context describing some of the non-intuitive impacts of CPU limits: https://github.com/kubernetes/kubernetes/issues/51135

Edit: added links

rsanders

6 lat temudiscuss

A container with a request but without a limit should be scheduled as Burstable, and it should only receive allocations in excess of its request when all other containers have had their demand <= request satisfied.

A container without either request or limit is twice-damned, and will be scheduled as BestEffort. The entire cgroup slice for all BestEffort pods is given a cpu.shares of 2 milliCPUs, and if the kernel scheduler is functioning well, no pod in there is going to disrupt the anything but other BestEffort pods with any amount of processor demand. Throw in a 64 thread busyloop and no Burstable or Guaranteed pods should notice much.

Of course that's the ideal. There is an observable difference between a process that relinquishes its scheduler slice and one that must be pre-empted. But I wouldn't call that a major disruption. Each pod will still be given its full requested share of CPU.

If that's not the case, I'd love to know!

rsanders

6 lat temudiscuss

Excluding kernel bugs, CPU limits just provide an upper bound on burst capacity. That controls oversubscription of CPU on a node. As with any other kind of oversubscription of a resource based on variable demand, there is a tradeoff. Allowing one pod to burst over its request is both unreliable and potentially impacting other neighboring pods. Whether that improves your cluster efficiency or introduces intolerably high variability in service latency and throughput depends on your mix of workloads and how the scheduler distributes your various pods.

Buffer's solution of having different flavors of node, onto which mutually compatible workloads are scheduled in isolation from incompatible ones, is a very reasonable thing to do, even if this particular case is a bit of a head-scratcher.

rsanders

6 lat temudiscuss

Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside...

I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods.

Run "systemd-cgls" on a node and you'll see two toplevel slices: kubepods and system. The kubelet process lives within the system slice.

If you run "kubectl describe <node>" you can see the resources set aside for system processes on the node. Processes in the system slice should always have (cpu_capacity - cpu_allocatable) available to share, no matter what happens in the kubepods slice.

    Capacity:
        cpu:                         8
        ephemeral-storage:           83873772Ki
        memory:                      62907108Ki
    Allocatable:
        cpu:                         7910m
        ephemeral-storage:           76224326324
        memory:                      61890276Ki
        pods:                        58

Granted, it's not a large proportion of CPU.

rsanders

6 lat temudiscuss

This advice is confusing. CPU is a "compressible" resource -- pods don't get killed for (trying to) exceed it. Pods don't get evicted from nodes based on CPU starvation, so autoscaling your node count won't help if you end up with a set of pods on a node that need more CPU than the node can provide. They'll just starve each other.

If your service allows horizontal scalability, you can use autoscaling of pods with Horizontal Pod Autoscaler (ideally also with a cluster autoscaler) to increase pod count for a given service when some percentage of the requested CPU is exceeded, whether or not you set a CPU limit. Setting the cpu_request appropriately for your pods is critical to ensure that node CPU is not oversubscribed by the Kubernetes pod scheduler.

Pods where mem & CPU requests = limits are given the highest class of service ("guaranteed"). For your most critical and latency sensitive services, this is the best approach when also coupled with HPA. Assuming a 4.19 kernel or later, I suppose.

https://medium.com/better-programming/the-kubernetes-quality...

rsanders

6 lat temudiscuss

Thanks! I didn't realize they were such distinct personalities.

Submissions

Nebula, the open source global overlay network from Slack

comments