25 years in industry, as staff engineer and CTO. Experienced across fields from safety critical embedded systems to distributed machine learning pipelines. Cross-disciplinarian, rabid learner, expert on system design and architecture.
Not impressed with the LLM hype or whatever latest crypto token you're hockin'
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
> I’ve been scraping dating sites and feeding them to ChatGPT, and it’s amazing how few of the profiles still say anything about the person.
Yeah, it's common knowledge now that you can't put many identifying details into a dating profile - there are creeps out there doing things like harvesting them and feeding them into ChatGPT.
It’s just an example of the fact that programming is far more than code. It is also managing dependencies, build systems, making architectural decisions, developer support, etc.
It is very possible to be good at slinging code and bad at everything that is required to support deployed code.
Based on geohotz history, I wouldn’t be surprised if he falls in that bucket - hacking and algorithm skills are great but don’t lend themselves at all to building twitter. That’s why it is foolish to pretend like one guy is great or even good at everything.
Have you just discovered the concept that humans are all humans, and all our achievements are just a mountain of knowledge we stand upon, from first principles?
Humans are human, someone being born in Africa doesn't magically mean that they will never understand technology.
Threads generally have less memory overhead and context switching between them is theoretically faster, but as I suspect you know it is situational and not as simple as just s/process/thread/g. There are many tradeoffs, including the loss of memory isolation, and since Postgres is a network application which holds a rather important position in most architectures, proper memory isolation is very important.
Switching to threads doesn't guarantee weaker isolation, just like it doesn't guarantee better performance, but those nuances are why "switching to threading yields performance improvements" is over simplified to the point of uselessness. It simply reeks of the same kind of ignorance that used to drive statements like "rust doesn't have vulnerabilities" and "rewrite it in javascript for web scale performance".
Ah yeah, what I figured. You clearly have no idea what you're talking about. Don't get all mad when someone calls you out on it.
By the way, just so you have some concept of what the actual problem is despite your resistance to education:
Simply switching from processes to threads will not yield the claimed performance increases. A 300x improvement on analytic workflows? From a direct transliteration? Your BS alarms should be going off. They should be screaming "5 Alarm Fire".
The only way they got that increase was by breaking the synchronization mechanisms that provide ACID guarantees in Postgres, otherwise a direct rewrite would expect very similar performance.
If you’re going to make a confident blanket claim, be ready to back it up - and asking for clarification is not trolling, by the way. You should be ready to engage in technical conversations if you want to make technical claims.
No I'm not missing anything. The existing specifications and tests provide a huge validation surface to work against for this kind of migration - they're the foundational pieces of the control loop. Without those two things, this kind of rewrite would not be possible.
We have known for decades that the hard part in programming isn't programming, it's making sure you have the right target! Bad designs and bad specifications have caused more bugs and broken software than anything else, and the time taken to develop the testing and validation suites (that is - the javascript spec tests) this rewrite used must be considered part of the engineering effort.
> and btw, if it can write code, it can write tests and test suite
You must be very early in your career. Again, the hard part is not writing the test, it's knowing which tests to write.
> I wished the two phases would have been tackled in reverse order.
Well, tackling them in reverse order would require the humans behind this to develop an actual understanding of the existing code and architecture before starting the project, instead of just asking claude to do it. So, here we are.
> Aiming for postgres compatible database with a 2026 architecture
Except you didn't improve the architecture, did you? You just asked an LLM to copy what was already there. Making real improvements to the database architecture requires understanding the database architecture, not just asking a calculator to do the work for you.
Better benchmark performance means nothing if the underlying guarantees break, and a 300x improvement sure makes me suspicious. I would look at something like this if it passes a Jepsen test, otherwise you simply will not be able to convince me that it's worth my time.
Not impressed with the LLM hype or whatever latest crypto token you're hockin'