We have a small SaaS product (4 developers on the team) with many competitors in the space, including a variety of homegrown solutions. Our industry has existed for many decades, and so many of our new customers have records dating back tens of years or more. In our industry, it's vital the historic records be migrated to any new software solution, and so we are in a position of needing to migrate data for almost every new client that we onboard. And because these new clients are currently using a variety of competitor software products, or their own in-house system of Excel sheets and TXT files, we end up hand-migrating everyone, which is both error prone and time consuming.
We have an in-house system to help eliminate errors in the migrations, but it is still EXTREMELY time consuming to build the migration scripts and vet the data, and we still end up with errored data from time to time.
What I'm hoping to find is some program (Free, Paid, whatever) or framework (Any language is fine) that can help with the migration process. Ideally this is a program that we can point at a data source (Even if we first have to hand move things into a proper database engine) and then write queries or transformations to get that data into our required format. Even better if the utility can keep track of data that we didn't move and show us what tables/columns were not moved so that we can audit the migrations in order to verify that all required data is being moved.
With all of the SaaS products in the world, I'd be surprised if nothing exists in this space, but so far in my lunch break googling I've come up empty.
0 comments
—
Ask HN: Data Migration Apps / Frameworks? · HackerTrans
We have an in-house system to help eliminate errors in the migrations, but it is still EXTREMELY time consuming to build the migration scripts and vet the data, and we still end up with errored data from time to time.
What I'm hoping to find is some program (Free, Paid, whatever) or framework (Any language is fine) that can help with the migration process. Ideally this is a program that we can point at a data source (Even if we first have to hand move things into a proper database engine) and then write queries or transformations to get that data into our required format. Even better if the utility can keep track of data that we didn't move and show us what tables/columns were not moved so that we can audit the migrations in order to verify that all required data is being moved.
With all of the SaaS products in the world, I'd be surprised if nothing exists in this space, but so far in my lunch break googling I've come up empty.