Agreed. From a quick skim (especially of the CLI interface) it looks to be a device to impersonate an NFC card, so you can then put it on a reader (eg. A hotel room door) and try to reverse engineer the handshake.
Back in the early days of Heroku (when I worked there), we were all fairly deep into the Ruby community. Ruby has never had a great reputation for performance, but... it seemed like almost a running joke that any time you went down a rabbit-hole trying to understand some weird performance issue you'd eventually discover that @tmm1 had already identified the same issue months earlier, patched it in core, and given an hour long talk about it somewhere. Despite his ability and willingness talk publicly about quite deep technical topics Aman always came across as an incredibly quiet and humble in person. Every Ruby developer has benefited from his attention to finding and fixing performance issues. I'm sure the same can probably said for every GitHub user (where he worked for years).
Congrats to the entire Cursor team! I don't know all of their stories, but I do like to smile and celebrate a little when I see people who are often hidden in the shadows quietly making things x% better for all of millions of us every day for many years getting reward for that effort.
I do suspect it was built with some form of AI though because the handful of links I've tried to dig into have all linked to the wrong place/are invalid :/
It's been a lot of trial & error. A quick aside: running these tests/evals/call them what you will at scale has been fascinating to me. Going back and trawling through the logs has been like speed-running through hundreds of usability tests with people, full of the same types of "aha! Of course you'd try and do that, why didn't I think of that already?" moments of insight and inspiration.
Which is also how we've gone about working out how to improve the CLI. It's usually one or more of:
* rethinking the subcommands and hierarchy to something more obvious and aligned to the task
* providing clear documentation upfront (i.e, in the skills file)
* keeping help text concise, but not too concise. You can't assume the reader is already a power user and it's simply looking for a reminder/reference. So include usage examples for common use cases
* where possible on errors, suggest the likely commands the person meant.
* In general offer affordances on what likely next steps will be. This goes for help output, success, and errors.
> cli help text is usually massive
That doesn't have to be true.
> could eat a lot of the savings on retries
This doesn't have to be true either. You don't need to give the same full help output on every single error, once they've got it once they've got it. Also the size of the entire help output for most CLIs is generally insignificant compared to even just a couple of source files in most repos.
I did exactly that and it's all covered in the blog post. There's no hidden eval harness, it's in the same codebase as the CLI so others can reproduce and/or extend as they see fit. It also includes code editing tasks and measures them too. The only asterisk on the code editing is I didn't automate the reporting of accuracy because the test only uses Claude and having it judge it's own work seemed dubious, and having our existing parsers + policy checks verify Claude's output in a benchmark test like this might look like we were cooking the books in our favor (i.e., we're testing and verifying using our own system which obviously we will always get 100% on). Writing up a whole new independent Terraform parser or test harness to verify the results was beyond the scope of what I was willing to do for this just right now. So I opted for a "just assume Claude always gets it right", and we reported on just the token differences to get there.
I'm not sure I follow, which profile do you mean? My profile on HN?
I don't know if we'll keep dissecting every incremental improvement we make as (so far) the general approach is the same as documented in the existing blog post: document common use cases -> benchmark them -> identify bottlenecks/expensive hot spots -> fix them -> repeat
The main thing changing right now is observing new more frequent use cases (either because we're adding new capabilities, or users are doing things we didn't entirely predict) and adding them to the test cases.
We've a lot of experience doing this! Also while this feeds into and supports LLMs and non-deterministic systems, our recommendations are entirely deterministic. So it's pretty rare to have a "wrong" recommendation given they've essentially been implemented + reviewed by actual people.
What can definitely happen though is you get one that is inappropriate in a given context. An example here might be a recommendation from an m5.2xlarge to an m6g.2xlarge instance. Same vCPUs and memory, lower cost, but... also a switch from Intel -> ARM architectures. For a lot of companies their build pipelines make it easy enough to make that change. For others there may be some specific dependency on Intel for that workload which means changing the architecture isn't viable. In that case you can simply dismiss the recommendation and we'll stop suggesting it.
We do cache the results locally so that we're not repeatedly hitting our pricing API. The LLM doesn't access that cache directly though as it'd suffer the token tax you mention. Instead we optimised our CLI to return agent optimised results. We're constantly iterating and improving on it, but it already reduces the tokens usage very significantly. I wrote about it here: https://www.infracost.io/resources/blog/we-cut-claude-s-toke...
We've found even more improvements since that post so those will be shipping soon too.
What do you mean by "Terraform cross compatibility"? Pulumi was (initially) built upon the same underlying providers so had the same capabilities.
I'd posit the main difference between the two was Terraform's declarative approach provided more consistency and predictability in how infrastructure was defined and provisioned. The constraints it imposed were the benefit vs a sprawling estate with a hundred different bespoke ways to provision a given service in your preferred language.
Yikes. If you showed me this car and asked me to guess the brand I'd probably say Renault. Which isn't meant to be shade on Renault, and I don't exactly hate the design and might even take a look at it if I was in the market given the expectations I have around the price point of a new Renault.
This is absolutely not a car that screams "Ferrari" though.
Yeah, while the "average" person might be able to gracefully handle these situations there's still a lot of people who do things that to me seem obviously silly and avoidable.
Locally there's a bridge that is regularly hit by human drivers. A bridge! Not a rare weather pattern, not some temporary and surprising change in conditions. A physical structure that has literally been there for over 100 years. The approach has numerous warnings, flashing lights, and swinging poles that will hit your vehicle and alert you that you're too high to clear the underpass if you continue. And yet... it's so common that there's websites and instagram tags and all manner of things to track and laugh at the people that continue to do it anyway.
not OP, but apparently all it takes is 2 kids that are independently into multiple sports + both parents being actively involved in the clubs/work schedules that allow us to regularly make every training session and game means we're regularly playing taxi for multiple families and having to split ourselves across locations when fixtures clash. Also all their kit just takes up a heap of space too. I wanted a sedan when we upgraded one of our cars late last year but it just wasn't going to work given all of the above.
I remember meeting a founder who had a moderately successful adtech business who was venting about the difficulty getting very large brands with huge spend on board. This is probably almost 2 decades ago now before what many of us take for granted in terms of analytics, so direct attribution between spend and return wasn't particularly common.
He was having great success with small and mid-tier companies, then he'd run a pilot with a massive global brand and the results would be even more stark than what he'd see with his existing users. But basically could not close a deal.
Because of exactly what you've shared here: "they measured on how much the spend, not on how much they bring in. Because they've never been able to do that. Their job is so much easier if nobody can ever see that latter figure they just go year to year asking for more budget and the more they spend the more everyone thinks they're doing a great job". I was naive enough at the time to think he was wrong and that couldn't possibly be true. But I've seen enough in the years since to realise he was right.
I mostly agree with what you said (the diff being we've still done the "pretty" output and progress bars for the human-centric outputs). And I found it a fun exercise as trawling through the log output of the LLM tests to see why things were slow at times felt like watching a bunch of usability tests. Various approaches to solve problems that seem obvious in hindsight but not at all paths we'd optimised for. If you've ever done usability testing on something you've built you probably have a sense for what I'm talking about.
And yes, one of the outcomes of this was also ditching the human output for something more dense and LLM friendly.
Most big companies still have travel agencies/companies manage their corporate travel. I can’t remember who we used when I was at Amazon, but I made a similar complaint to my manager once given I could fly cheaper in a higher class on a different airline (also one I had heaps of points with so I would have preferred it because I’d be able to upgrade further and/or use the lounge).
Turns out the price I saw in the booking portal isn’t actually what Amazon paid. It’s kinda more like a rack rate listing. But then there’s all kinds of discounting/cash back that happens on the backend based on the amount of travel booked each month.
Zitron’s narrative around AI revenues is that somehow the people Anthropic/OpenAI etc are pitching to, people who meet the sophisticated or wholesale investor tests (i.e., the only people actually able to trade on this information) do not know what ARR means, how it’s different from revenue, and are unable to read financial statements for themselves. And that he is seemingly the only one with insight, with the partial bits of information he has access to, that can tell them the truth of the situation.
I used to love his stuff. Until he just wouldn’t stop beating this drum so breathlessly. Now I wonder how much I’d suffered Gell-Man amnesia with the other content of his I’d previously enjoyed.
I recently had (and then lost/left on a plane!) a Lumenate Nova[1] and found it was very helpful at quickly getting me away from the mind going state. I work very late to overlap with distant timezones and would often find it difficult to get to sleep once I went to bed given I've been staring at screens and on calls only minutes before hitting the pillow. This was great.