How do you handle domain-specific transformation logic without hardcoding it?
Working on a data platform that needs to support multiple business domains (sales, EMIR regulatory, finance) from the same pipeline infrastructure. The problem we kept hitting: every time we added a new domain, we were either: a) Adding if/else blocks into the core transformation job b) Duplicating the entire pipeline We solved it with domain profiles + plugins — each domain has a JSON profile declaring sources, silver modules, gold builder config, and DQ packs. The core Glue job reads the profile and executes generically. New domain = new profile + plugin. Zero changes to core code. Curious what approaches others are using. Are you using dbt models per domain? Separate pipelines? Something else? What's worked at scale?
1 comments
Read up on Software design patterns: https://en.wikipedia.org/wiki/Software_design_pattern