Yup agreed. We built an orchestration engine into Neosync for that reason. Can handles all of the reading/writing from DBs for you. Also can generate data from scratch (using LLMs or not).
Nice write up, mock data generation with LLMs is pretty tough. We spent time trying to do it across multiple tables and it always had issues. Whether you look at classical ML models like GANs or even LLMs, they struggle with producing a lot of data and respecting FKs, Constraints and other relationships.
Maybe some day, it gets better but for now, we've found that using a more traditional algorithmic approach is more consistent.
Transparency: founder of Neosync - open source data anonymization - github.com/nucleuscloud/neosync
Thanks for the question! Faker is useful but doesn't have a lot of features. For example, referential integrity, data orchestration or the ability to read/write to a db. So faker can work for simple API schemas but if you need something more robust for an entire database, then that's where we can help.
Thanks! Yeah we generally recommend not making your databases public and instead connecting to them using a bastion host. We support this at Neosync. Also, ideally, not connecting to a live DB and instead a snapshot or back up. A read replica could work as well but a snapshot is better.
The ideal experience is that you anonymize prod and sync it locally. Whether it's for testing or debugging, it's the only way to get representative data.
When you write mock data, you almost always write "happy path" data that usually just works. But prod data is messy and chaotic which is really hard to replicate manually.
This is actually exactly what we do at Neosync (https://github.com/nucleuscloud/neosync). We help you anonymize your prod data and then sync it across environments. You can also generate synthetic data as well. We take care of all of the orchestration. And Neosync is open source.
Excited to announce a new partnership between Neon (open source serverless postgres) and Neosync (open source data anonymization) to give developers the easiest way to create data branches with anonymized production data for better testing, debugging and developer experience.
Nice! appreciate you sharing it - would love to see the code at some point but looks like it's confidential.
I spent a lot of time building tokenization solutions at a previous startup so we'll definitely support tokenization at some point. There is a good use-case for it as well!
The ideal scenario is that you're able to augment your existing data with more data that looks just like it. The matter of statistical significance really depends on the use-case. For load testing, it's probably not as important as it is for something like feature testin/debugging/analytical queries.
Even if you know the distribution of the data (which imo can be fairly difficult) replicating that can also be tricky. If you know that a gender column is 30-70 male - female, how do you create 30% male names? How about the female names? Are they the same name or do you repeat names? Does it matter? In some cases it does and in others it doesn't.
What we've seen is that it's really use-case specific and there are some tools that can help but there isn't a complete tool set. That's what we're trying to build over time.
We're actually evaluating a clickhouse integration at the moment for a customer that we're working with so that might be coming in the future. Although today just PG and Mysql.
To answer your questions:
1. We don't support this quite yet although we're working on it for both anonymization and synthetic data. For anonymization, that typically means having deterministic anonymizers that output the same value for the same input (like a hash). For synthetic data that means using a model to be able maintain those same statistical characteristics. We'll have support for both of these within the next quarter. It also depends on what you want to anonymize. If the values that you want to anonymize wouldn't meaningfully change the distribution of the data (think like a name or an address and you're not doing any analytics or queries on those fields) then the statistical distribution of the data stays the same.
2. A few big differences. PG Anonymizer doesn't handle referential integrity, pretty much everything has to be defined in sql and it doesn't have a GUI and it doesn't have any orchestration across environments or databases. Neosync supports all of those.
Folks use the cloud service because they don't have the resource or time to deploy/run the OSS offering themselves. These are usually startups who are okay with us streaming their data and anonymizing it and sending it back to them. We're SOC2 type 2 compliant and usually got through a security review for these deployments. Conversely, they can also just run our managed version and keep all of their data on their infra while we host the control plane.
yeah the referential integrity and constraints part is usually the most complicated part and everyone does things differently which adds another layer of complexity on it
yeah good question, if you're doing any sort of analytical work, then you'll care about the statistical distribution of your data. If you're running queries or sharing data with third parties, then you want to maintain the same stats. If you're just building features then you might not care as much as if you were doign analytical work. But it could still be relevant if you're building metrics/dashboards/anything visual - you'll want to see that you can render your prod data correctly. So more so for analytical work but less so for normal, run of the mill dev work.
it's a combination of creating a random number of records for foreign keys i.e 1 customer - create between 2 and 5 transctions. Working on giving you control over that, and handling referential integrity with table constraints (foreign keys, unique constraints, etc.)
ML based approaches typically are not very good at this and struggle with handling things like referential integrity. So a more "procedural" or imperative way is slightly better. The ideal is a combination of both.
Thanks for the comment and hear you on the anonymization. What we see is that customers will go through and categorize what is PII and what is not and anonymize as needed. If not, they'll back fill with synthetic data. You can change the gender from male to something else, same with the city, etc.
It's really down to the use-case. If you're strictly doing development, then you'll probably want to use more synthetic data than anonymization. If you care about preserving the statistical characteristics of the data then you can use ML models like CTGAN to create net new data.
Definitely a balance between when do you anonymize vs. when do you create synthetic data.
We just launched our brand new AI Data Generation feature which allows you to use ayn LLM to generate synthetic data and insert that directly into a Postgres or Mysql database.
Simply connect your database, configure any LLM that you want to use (we support any model that is hosted on an endpoint, it can be local as well!), then provide a prompt and generate data.
We first sample 10 records to give you an idea of what the data looks like and you can iterate on the prompt and data until you're ready.
Neosync handles all of the infrastructure, prompt-chaining, formatting and orchestration of the data from the LLM to your database.
Here are some use-cases:
- If you're building a new app you can use Neosync to seed your database
- If you're working on a new service that has it's own database or schema, you can use Neosync to seed it with data.
- If you want additional data for fine-tuning or RAG, you can use Neosync to generate that data
What's next?
We can generate up to 1000 records right now and are working on supporting up to 10k in the next few weeks. Currently it works for a single table but we're working on making it work for an entire database including all of your constraints.
Here's a 5-min demo video:
https://github.com/nucleuscloud/neosync
(I'm one of the co-founders)