As a platform team leader that caters to other engineers within the organisation, extensive documentation is what enabled me and my team to be productive.
I answer almost all questions posed to me with a simple have you read this (confluence link )
I saw my interviewers notes in our teams SharePoint.
Mixed emotions after knowing that i was almost not hired for being younger / having less experience than expected for the role.
7 years now in the same company and the same person has claimed that I hired him multiple times over the years.
For big data systems, CSV is a arguably worse format
Row wise storage doesn't provide compression benefits like columnar storage which can significantly reduce storage needs.
Data could contain the same delimiters ( comma, newline ) as used by the parsers and introduce errors in computations. binary formats( parquet/ORC) can eliminate this issue while providing other benefits as well.
A data quality framework like deequ should help catch errors in data before you introduce the same to downstream applications in ELT.(not infalliable )
For ELT processes, always filter first before other processing.
Plan your partitions according to expected querying patterns. If reports are run country wise , then country is a good partition key or if date wise then date is a good parition key.
Be aware of data skews , you might introduce skew during partitions ( ex : partitioning by country and a few countries have large number of records compared to others.
You should also see the pathetic copycat feature YouTube has created called the shorts. Facebook's attempt with short videos and reels is not much better either.
Why does every company feel the urge to copy this and also fail so badly at this
I answer almost all questions posed to me with a simple have you read this (confluence link )