HackerTrans
TopNewTrendsCommentsPastAskShowJobs

robbie-c

563 karmajoined 11 lat temu

Submissions

I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

posthog.com
155 points·by robbie-c·17 dni temu·72 comments

Autoresearch found a 3-year-old bug in our query engine

posthog.com
5 points·by robbie-c·w zeszłym miesiącu·0 comments

comments

robbie-c
·przedwczoraj·discuss
Nothing has changed, just OP found the mirror of the main repo with the ee/ folder removed. That repo has been up for most of the lifetime of the company.
robbie-c
·przedwczoraj·discuss
PostHog has always been open source. I'm not sure why this has been shared on HN at all.
robbie-c
·16 dni temu·discuss
That's valid criticism, I kinda hand-waved the "months" part. I read everything I could about parsers while building this (I have a CS background but hadn't thought about parsers in a long time) and came across this blog post https://lakesail.com/blog/sql-parser-in-one-week/ which talked about building a toy parser in a week, so I scaled that up to months for a production one.
robbie-c
·16 dni temu·discuss
The previous parser is mostly a declarative grammar file, which is extremely readable. It codegens a C++ parser, which is hard to read. It depends which of those you count as the previous parser's source code!

In the future, we'd make changes by modifying the ANTLR parser first, then using the same approach as in the blog post to get the new parser to parity. We have no plans to get rid of the C++ parser as an oracle!
robbie-c
·17 dni temu·discuss
Yeah, one of the interesting parts to me while working on this is that the breakpoint for when it's worth writing your own parser vs accepting ANTLR's slowness has shifted massively. Previously it would have been someone's full-time job to maintain. Now with this approach you can get the best of both worlds.
robbie-c
·17 dni temu·discuss
:D

I was hoping to avoid the criticism that I was picking the most favourable number and being misleading!
robbie-c
·17 dni temu·discuss
In what way? This was a geometric mean of the improvements from a small test corpus. In production, where it only parses longer SQL that didn't hit the parser cache, the mean parse time went down by 454x, across millions of parses.
robbie-c
·17 dni temu·discuss
Ha I did consider that! But 70x is plenty fast enough (we still have to query an actual database!) and the parser runs in a shared process on untrusted input, so it wasn't worth the security risk
robbie-c
·17 dni temu·discuss
It took about 2 days to get a proof of concept, and about a week to get something I could ship to production.

I skipped a few features for the PoC (like XML tag support, token positions), so most of the delta was adding those back in!
robbie-c
·17 dni temu·discuss
Our SQL is very similar to ClickHouse SQL, in that we used ClickHouse SQL as a starting point as that's what our underlying DB is. We needed to have our own parser so that we could add additional language features on top.
robbie-c
·2 miesiące temu·discuss
We pay github quite a bit of money and it's down for us too
robbie-c
·2 miesiące temu·discuss
I did that first, which didn't work for a couple of reasons. Many "big and tall" stores are the intersection only, and many of the stores didn't quite have the right vibe. I do have some of my wardrobe from places like 2tall.com but I was looking for something very silly for an in-joke for a friends vacation.
robbie-c
·2 miesiące temu·discuss
Undecided. One of the sites didn't have an LT but the LLM flagged that chest dimensions on their large were narrower than others, so could be worth trying.
robbie-c
·2 miesiące temu·discuss
Huh, I shopped for clothes using AI today.

Not super relevant to the Googlebook ad, but in case the perspective is interesting to you: I'm quite tall (194cm) but not very wide, so I usually struggle with buying clothes online. I used AI to scrape a bunch of clothing stores to see whether they sold a men's shirt with an LT or slim fit size, in stock, and matching a particular vibe.
robbie-c
·4 miesiące temu·discuss
Wait - are you missing all the context on this? Anthropic pushed back against this hard, there was a whole back and forth. I'm on mobile and can't look it up for you atm but if you google about this scenario, Anthropic definitely come out of this looking a lot better than OpenAI and xAI
robbie-c
·4 miesiące temu·discuss
Quite a lot worse. Both OpenAI and xAI were among the largest donors of Trump's campaign

Musk was the largest individual political donor of the 2024 election [1] and Greg Brockman was the largest donor to Trump's "MAGA Inc" super PAC [2]

[1] https://www.washingtonpost.com/technology/2024/12/06/elon-mu...

[2] https://www.theverge.com/ai-artificial-intelligence/867947/o...
robbie-c
·7 miesięcy temu·discuss
There's proof of financial dependence, here's a recent report https://assets.mozilla.net/annualreport/2021/mozilla-fdn-202...

In 2021 they got $500M "royalties" (this is their payment from Google) with only $75k revenue from all other sources, including $7.5k donations.
robbie-c
·7 miesięcy temu·discuss
I think it's just an AI-generated simplification, sucks that it made it to the front page. The subject matter is interesting, I would have loved to have read something written by an expert!
robbie-c
·8 miesięcy temu·discuss
I've wondered about this too. I live in the UK and have been idly daydreaming about my next startup, and it seems like Brexit, and therefore having a small market / uncooperative EU, is such a headwind for some of the things I'd like to do. Seems like many UK startups just pretend to be based in SF.

I think rejoin is going to be politically unpopular for a while, as there's no way we could rejoin the EU on anything like as good terms as we left on.