As the title states, I made a library to process and validate a massive JSON in Node.js where using JSON.parse() means saying hello to OOM.
It came from an unfortunate situation at work where we had no control over the data and can't change the format to something that's easy to stream. Therefore, processing it meant dealing with a high memory footprint in our pods, which I thought was a bit wasteful, since we only really needed some parts of the data and we didn't need to load the whole thing in memory. There are JSON streaming libraries available but it wasn't really to my taste (e.g. SAX-style, callbacks) and it doesn't flow how I like my code to look.
I left that job 6 months ago but the idea still lingered so after loads of mistakes with parsing, SIMD, binary encodings, FFI performance hits and general API design, I've finally arrived at something I'm proud of. I'll write a blog about the journey at some point since I think it's interesting.
It's at pre-1.0 at the moment and would really appreciate feedback. I've tried to really hard to minimize the API surface to make it simple. Please have a look.
Disclaimer:
I wrote this with the help of AI but made sure I was on the wheel before pushing code. Granted, there were times I just went LGTM like a Friday morning PR but those usually come back to bite you so I try not to.
There's a lot of tech stack jargon here that it doesn't really tell me what it actually does. How does this differ from me using an agent with Github's MCP to control my repo?
> source code open makes it auditable, if not by you, maybe the community
I think part of why this social engineering works so well is it takes advantage of that "many eyes" trust, where people are prone to delegating the responsibility of checking to the community and not do due diligence on themselves. I know I'm susceptible to it if I see a Github repo with more than 10k stars on it.
I'm not sure if Claude Code has integrated it in its system prompts or not since it's moving at breakneck speed, but one instruction I like putting on all of my projects is to "Prompt for technical decisions from user when choices are unsure". This would almost always trigger the prompting feature that Claude Code has for me when it's got some uncertainty about the instructions I gave it, giving me options or alternatives on how to approach the problem when planning or executing.
This way, it's got more of a chance in generating something that I wanted, rather than running off on it's own.
I noticed this effect as well when I work in a vibrant cafe. Someone else's productivity is infectious and it makes me more inspired to do the things I set out to do that day, than the intrinsic pressure of needing to do the task.
Long time lurker here.
As the title states, I made a library to process and validate a massive JSON in Node.js where using JSON.parse() means saying hello to OOM.
It came from an unfortunate situation at work where we had no control over the data and can't change the format to something that's easy to stream. Therefore, processing it meant dealing with a high memory footprint in our pods, which I thought was a bit wasteful, since we only really needed some parts of the data and we didn't need to load the whole thing in memory. There are JSON streaming libraries available but it wasn't really to my taste (e.g. SAX-style, callbacks) and it doesn't flow how I like my code to look.
I left that job 6 months ago but the idea still lingered so after loads of mistakes with parsing, SIMD, binary encodings, FFI performance hits and general API design, I've finally arrived at something I'm proud of. I'll write a blog about the journey at some point since I think it's interesting.
It's at pre-1.0 at the moment and would really appreciate feedback. I've tried to really hard to minimize the API surface to make it simple. Please have a look.
Disclaimer:
I wrote this with the help of AI but made sure I was on the wheel before pushing code. Granted, there were times I just went LGTM like a Friday morning PR but those usually come back to bite you so I try not to.