I listened to you on the latentspacepod the other day and tried this method myself.
1. Used GPT-4 to build functions that pulled in sample datasets from my BigQuery.
2. Used GPT-4 to insert them into a sqlite database.
3. Upload the zip file into Code Interpreter.
4. Now I'm in an environment to quickly self-iterate on GPT-4 generated SQL.
It felt incredibly powerful. Previously I was developing with the latency of 100Gb+ tables – now the sqlite db responds instantly, and I don't even need to write the code.
I’ve used this library for a number of projects and it’s a joy to use. I don’t think it’s an understatement to say it’s paradigm shifting – to the extent that once you have your environment set up, you are free to code, think, iterate, deploy and document your projects all at 99% of the speed of thought.
There seems to be a lot of discussion in here around the pitfalls of jupyter, and notebooks, and the poor coding practices of data scientists. If you haven’t read the article or used the software I’d like to highlight that all of these (legitimate) complaints are exactly what nbdev2 was created to address, and in my opinion very successfully solves.
The way it works is that everything runs off a master notebook, and then with one command: libraries are built, git diffs are magically fixed, tests are run, documentation is automatically created. It doesn’t fundamentally change your workflow in any way, it just abstracts and automates away all of these pain points.
There’s a reason that everyone uses jupyter notebooks. They are fun to use, they are great for exploring and developing ideas. And (minus the aforementioned git collaboration issues) they are great for sharing with others, which is a huge part of the wider data science ecosystem. We don’t need to recommend avoiding notebooks, and allege they are just for beginners. We need to use tooling which addresses some of these final issues with writing mature software. And I'd like to thank the authors of nbdev for this.
The people who look down their noses at notebooks can continue to do so – but what they will find is that nbdev quite effortlessly leap-frogs over these sneered complaints, and allows you to write better software more productively.
I think you’ve done a good job of summarizing the landscape – it matches my experience.
The progression seems to be the decline of the nerd-nerd monoculture in the professional tech environment. With some irony, the increase in diversity is from the non-neurodiverse. Product managers, engineering managers, vertically integrated teams.
I believe on the most part this creates healthier productivity for the business – but likely at the social expense of the neurodiverse. Their technical skills can be more readily exploited by the hustlers.
It felt incredibly powerful. Previously I was developing with the latency of 100Gb+ tables – now the sqlite db responds instantly, and I don't even need to write the code.