compumike

10 gün öncediscuss

I don’t see any other impact-narrowing criteria.

The default kubelet `syncFrequency` is 1 minute (per Pod). (There can be additional event-driven ones, but this is a floor.) Back of the envelope: 30 Pods * 1440 * 7 = 300k calls to startPodSync in a week.

I’d guess a lot of production clusters don’t run the latest release of Kubernetes, and 1.36 was only released in late April, less than two months before I reported the issue.

Given that the issue shows up as linear memory growth on the order of maybe 1 GiB/month per node, there really is a time-based discovery process here. (Could be artificially accelerated for a CI test, I imagine!) And any restarted nodes or clusters that upgraded point releases would restart that clock.

The only unusual contributing cause that made the issue more visible to me was trying to run everything on a tiny 2 GiB RAM node!

compumike

10 gün öncediscuss

It does seem like a serious tripping point.

I took a quick look at "containedctx" and it seems like for this case, it would almost be backwards: it would flag the (not-memory-leaking) struct-stored "status.ctx", but wouldn't flag when there is a stored "status.cancelFn" only (which resulted in the memory leak).

Could Go implement a runtime leaked-context detector, like the data race detector? https://go.dev/doc/articles/race_detector

compumike

10 gün öncediscuss

What kind of health checks? In my case, the kubelet process was staying alive and responsive to queries, I believe due to:

  # cat /proc/$(pgrep kubelet)/oom_score_adj
  -999
  
  (from OOMScoreAdjust=-999 in /etc/systemd/system/kubelet.service)

With this score, the Linux OOM killer wouldn't touch it, but any of my Pods were fair game.

compumike

10 gün öncediscuss

Thanks. I know there's a `go vet` tool that's run as part of Kubernetes CI, and one of its checks is:

  lostcancel: check cancel func returned by context.WithCancel is called

I'm not 100% sure why `go vet` didn't catch this issue, but storing the cancelFn in the struct is probably part of the reason. Any Go experts know if that's the case?

compumike

11 gün öncediscuss

Author here! If you're running a Kubernetes cluster, I recommend you check `kubectl version` and see if you're running "Server Version: v1.36.[0,1,2]". If so, you may want to use the one-liner at the end of the article to check your "process_resident_memory_bytes" on each node, and consider restarting kubelet as a temporary workaround to tame the memory leak until v1.36.3 is released.

compumike

geçen aydiscuss

Here are some N-year rolling total inflation charts to put this datapoint in a longer-term perspective: https://totalrealreturns.com/inflation . Zooming out always smooths the noise.

compumike

geçen aydiscuss

> Redis (and memcache) are memory caches and should be treated like that

If you haven't come across Kvrocks yet, it may be worth a look: https://github.com/apache/kvrocks https://kvrocks.apache.org/ . It's a database with a Redis-compatible wire protocol, but the database is stored on disk. This means your working set is not limited by RAM and can be a few orders of magnitude larger! On modern SSDs this is still very fast. I think it improves the durability story as well. But the big win is the orders of magnitude larger database space.

As I've been improving my side project https://totalrealreturns.com/ recently I've ended up using both Redis and Kvrocks together. Redis is great for small global state that needs to be super fast. Kvrocks is great for larger bulk data storage (large precomputed datasets), but also supports a lot of the Redis data structures as well as Lua scripts.

compumike

2 ay öncediscuss

Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:

If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!

If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.

Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.

compumike

2 ay öncediscuss

You can cheaply and readably give a lot of clues to both agents and humans with some assertions at the start of a method:

  raise ArgumentError.new("...") unless ...

which can include type assertions but also a lot more. The agents seem to do well with this.

I've also had good results using agents to write Crystal https://crystal-lang.org/ which is Ruby-like but does have the static types and produces blazing fast static binaries. Might be a sweet spot for coding agents if you're building some backend services. But I'd still pick Ruby on Rails for a new full stack project.

compumike

2 ay öncediscuss

Check out this map: https://api.phillymesh.net/map for live data from the Philadelphia area.

The edges drawn are between nodes that have been able to hear each other in the last 24 hours, based on observed traceroute packets.

(Even then, it’s only a subset of the actually-connected nodes: the map only shows nodes that have published their position on the public channel, and have set a flag that their data is okay to uplink to a server over MQTT.)

compumike

3 ay öncediscuss

It can be dangerous to backfeed (which is why you're supposed to have an interlock for a generator inlet, ensuring utility power is disconnected). But:

1. These grid tie inverters are designed and tested to shut off completely if there's no grid power. (This is a big design tradeoff: it means they don't provide any power during a grid power outage, even if it's very sunny out.)

2. Even if I had a beefy generator that was unsafely backfeeding my house while the utility power was still connected, the generator would be trying to power not just my house, but all my neighbors too! And the circuit breaker and/or inverter on the generator would likely trip and shut down almost instantly.

There's still a possible risk from #2, especially if the downed wire being repaired is relatively local (i.e. your house only).

But I think #1 and #2 mitigate this risk very well.

compumike

3 ay öncediscuss

Just wait until you see "kW/h" :)

But I think plug-in / balcony solar will be pretty cool. And I think there's a path to inexpensive, larger, safer grid-tie inverters which never backfeed, but prioritize solar input first and make up the difference with grid power.

For example, I'm imagining a box that would plug in to the wall, have a DC input from solar panels, and a power strip for loads supporting up to, ideally, a full 15A normal US 120V circuit.

Currently this box exists in the form of battery power station units (Bluetti, Ecoflow, Anker etc). But I think there could be a much less expensive form that could exist without the battery.

compumike

5 ay öncediscuss

> Schematics also help explain your circuit because the idioms of drawing them communicate intent.

I think this is a key point. One could imagine three layers:

(1) NETLIST: a list of text-only descriptions, like "R1 1 0 1k", or even a sentence description "connect a 1k resistor between node 1 and node 0"

(2) SCHEMATIC: a 2D drawing with canonical symbols, straight lines for wires, and labels/notes

(3) LAYOUT: a 3D (or multi-layer 2D) physical representation for PCB, or in this case breadboarding

All three layers are useful. (Obviously you need layout to make a PCB, and you need a netlist for simulation.)

But for most humans, where we have 2D visual representations baked in, if you're trying to understand or communicate what's going on:

- It's really really hard to keep track of a bunch of text sentences like a netlist and node numbers/names for all but the simplest circuits -- maybe 3-5 elements?

- It's really really hard to follow a 3D layout of PCB tracks that leads to pads, and then having to remember pin orders etc.

- It's easiest to follow a schematic diagram. It's browsable. It contains "idioms", as you say, about signal flow, logical block groupings, etc.: purpose and intent and functionality, in a way that netlists and physical layouts don't.

FYI, for medium-large digital circuits, I don't think this is true: probably just reading VHDL/Verilog, like reading source code, makes more sense. This is closer to the "netlist" level. I think that's because you'd name modules and inputs/outputs in a way similar to how you'd name functions and arguments in software, which doesn't really apply to "Resistor" or "Capacitor" as primitives.

But for a pretty big practical range of mixed-mode and analog things, I'd argue that schematics really are the easiest level for our brains.

(Disclosure: I'm one of the founders of CircuitLab https://www.circuitlab.com/ (YC W13) where we've been building an online circuit simulator & schematic editor for a long time. Although I'm mostly on the simulation engine / netlist side. My cofounder and other teammates have done most of the schematic GUI work.)

IMHO solderless breadboards still have their place for prototyping some slow circuits, ballpark maybe < 1 MHz signals, if you're aware of the extra capacitance and limitations. :)

compumike

5 ay öncediscuss

The big thing that articles like this miss completely is that we are no longer in the brief HTTP/1.0 era (1996) where every request is a new TCP connection (and therefore possibly a new DNS query).

In the HTTP/1.1 (1997) or HTTP/2 era, the TCP connection is made once and then stays open (Connection: Keep-Alive) for multiple requests. This greatly reduces the number of DNS lookups per HTTP request.

If the web server is configured for a sufficiently long Keep-Alive idle period, then this period is far more relevant than a short DNS TTL.

If the server dies or disconnects in the middle of a Keep-Alive, the client/browser will open a new connection, and at this point, a short DNS TTL can make sense.

(I have not investigated how this works with QUIC HTTP/3 over UDP: how often does the client/browser do a DNS lookup? But my suspicion is that it also does a DNS query only on the initial connection and then sends UDP packets to the same resolved IP address for the life of that connection, and so it behaves exactly like the TCP Keep-Alive case.)

compumike

6 ay öncediscuss

It's very local here. I'm in the suburbs of Philadelphia, in one of the highest income counties in the state, two blocks from a major hospital, one block from a suburban downtown. Despite that, I've experienced one or two 4-6 hour long power outages per year the past few years. (Mostly correlated with weather.) One outage in June 2025 was 50 hours long!

Many larger homes in this area have whole-house generators (powered by utility natural gas) with automatic transfer switches. During the 50-hour outage, we "abandoned ship" and stayed with someone who also had an outage, but had a whole-house generator.

Other areas just 5-10 miles away are like what you describe: maybe one outage in the past 10 years.

compumike

6 ay öncediscuss

> If something goes wrong, like the pipeline triggering certbot goes wrong, I won't have time to fix this. So I'd be at a two day renewal with a 4 day "debugging" window.

I think a pattern like that is reasonable for a 6-day cert:

- renew every 2 days, and have a "4 day debugging window" - renew every 1 day, and have a "5 day debugging window"

Monitoring options: https://letsencrypt.org/docs/monitoring-options/

This makes me wonder if the scripts I published at https://heyoncall.com/blog/barebone-scripts-to-check-ssl-cer... should have the expiry thresholds defined in units of hours, instead of integer days?

compumike

7 ay öncediscuss

100%, I've run into this too. I wrote some minimal scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to send a request and alert if the expiration is less than 14 days from now: https://heyoncall.com/blog/barebone-scripts-to-check-ssl-cer... because anyone who's operating a TLS-secured website (which is... basically anyone with a website) should have at least that level of automated sanity check. We're talking about ~10 lines of Python!

compumike

7 ay öncediscuss

Thought you might find this story interesting, concerning the Total Real Returns website which originally launched on HN a while ago. https://news.ycombinator.com/item?id=32081943

This is probably the same kind of "one new row per month" assumption that many data pipelines with any sort of primary date/time column make!

compumike

8 ay öncediscuss

Lots of people are speculating that the price spike is AI related. But it might be more mundane:

I'd bet that a good chunk of the apparently sudden demand spike could be last month's Microsoft Windows 10 end-of-support finally happening, pushing companies and individuals to replace many years worth of older laptops and desktops all at once.

compumike

8 ay öncediscuss

There's a tradeoff and the assumption here (which I think is solid) is that there's more benefit from avoiding a supply chain attack by blindly (by default) using a dependency cooldown vs. avoiding a zero-day by blindly (by default) staying on the bleeding edge of new releases.

It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.

While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.

On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.

If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!

Submissions

Fixing a kubelet memory leak in Kubernetes 1.36

A Government Shutdown and a 1913 Data Assumption Caused an Outage in 2025

AWS us-east-1 outage on 2025-10-20: 7% of teams were paged

comments