My team of SWE's is at an inflection point with how we handle database backfills. Think new fields on existing records across environments being populated by some conditional logic. I am curious how other development teams handle their own backfill operations and if there is a "standard" or rule-of-thumb.
In past teams, repos comprised of common functionality executed by Jupyter Notebooks, or even loose scripts, would be used for us to review incoming backfills before the engineer would shell in and execute the script. This is better than no reviews of backfill operations but I would like to think there is a more streamlined way.
0 comments
—
Ask HN: How does your team handle data backfilling? · HackerTrans
In past teams, repos comprised of common functionality executed by Jupyter Notebooks, or even loose scripts, would be used for us to review incoming backfills before the engineer would shell in and execute the script. This is better than no reviews of backfill operations but I would like to think there is a more streamlined way.