We already have these in the industry. They're Staff+ Engineers and Architects. It's generally the norm to not be cranking out code at this level, but they make sure everyone is moving in the right direction, assisting managers, and mentoring juniors.
Could you go more into this? How did you go about the evaluation and what metrics were you using? How did you capture the 'developers just hated it' in a way that leadership understood?
I'm not disputing the outcome, just feel like it would be useful for others in a similar situation.
I'm assuming that you run Kafka within AWS. Much of the hardware requirements/suggestions I've seen for Kafka are all for non-virtualized environments. If you can get into it, could you share some details...
- What is the size of your Kafka cluster
- What instances types do you use?
- Do you use EBS or use ephemeral storage?
- How much do you over-provision to deal with instance loss?