I thought this could be interesting - sharing how we iterated from a no-code product for non-technical users to a developer platform, and from proprietary to open-source.
For anyone who is currently facing similar challenges, this could be worth a read :)
Very much needed. Spoke with many of the people listed on the repo, can confirm that they are not only open-source but generally very kind and trying to help. Using this for our current fundraise, and hopefully using this for our next one, too :)
Annotation platforms use Excel. I once received 25 files of separate Excel spreadsheets from a labeling service for 10k texts (short texts about product titles, e.g. "Sauvignon blanc" -> "wine"). Had to merge them, which wasn't as easy as you'd might expect.
Also, I once labeled 5,000 texts during my master's degree via Excel. Was painful as hell.
My guess (some if this we already have, some we don't):
- automation: integration of heuristics (multiple columns that you can program via formulas and such)
- exploration: finding outliers or most similar records given some reference (e.g. "I want to label more rows that are about business news in some extent")
- monitoring
- labelmanagement [which we don't offer yet in the extent we'd like to]: merging and splitting labels etc.
generally anything that scales and "somewhat" guarantees the users to input valid labels.
But it definitely offers something that new tools don't: users are super familiar with it.
I think so too. Mostly that it is something open. I also believe that it will change the workflow a bit, and that DVC will play a major role in it for versioning your different data hypotheses. Let's see, exciting times ahead!
Hey Mark, I totally agree. We're focusing on NLP, but we're generally interested in what programming will develop into.
To exaggerate a bit, but I like that idea: With "regular programming" (not the best term, but I mean rule-based systems etc.), you used to develop via punch cards. You had to think multiple times before "compiling" something, right? I believe that we're currently in that phase regarding supervised learning development. If you have a labeling project, you need to plan this long in advance, ...
We're in love with VS Code, but we're missing something like this for AI. Our application tries to show how developers can program their training data, i.e. refine raw data into training data, and do so with many programmatic approaches. We're trying to show how something like this could look like (hence the title), and do so in NLP.
We've focused on JSON as the user-specified data model. So you can upload anything fitting into a JSON. We're using pandas to process the uploaded data, so spreadsheets or CSV-ish also work.
Hey, thanks! :)
What exactly do you mean with application? You can just pull the repository. As mentioned in the installation section, it is quite easy to start the app on your local host.
The most famous is arguably Snorkel, which started with an open-source library as a research project. We used that a lot ourselves, but the library by now is deprecated.
We aim to extend on that idea by providing something that comes as close to a programmable interface for data-centric tasks as possible, and do so via open-source.
There are lots of cool tools out there btw. in that area. Definitely worth to have a look at a landscape (haha idea for the next HN post incoming I guess :D)
A bit below it, you find a table with the links to all repositories. All of them are open-source. But thanks for the feedback, I'll try to make it a bit easier to understand! I appreciate that! :)
Hey, I'm Johannes - one of the maintainers of refinery. Thanks Jonathan for sharing!!
Would be super excited if you guys have any feedback. It's nowhere near perfect yet, but you can already use it to build some great data-centric use cases. Amongst others for sentiment analysis, conversational AI or finetuning of your embeddings (which you can check out here: https://github.com/code-kern-ai/refinery-sample-projects).
For anyone who is currently facing similar challenges, this could be worth a read :)