Seems like a fundamental misunderstanding to me. Commands get validated via your aggregate, not events. Events are not allowed to be rejected. Ever. Because they already happened.
I feel like people are running into these problems because they want to pretend that a message broker is an event store. I could try to shovel a star schema into MongoDB too, but why would I want to?
Keeping data only in the latest schema is dangerous. We have no idea what data the business will find useful years down the line. By only having whatever is in the latest schema, you may have thrown valuable data away.
Best practices around this have already been established. Most if not all event stores - which Kafka is not - have a concept called 'position.' You save the position atomically along with whatever you did with the message. Then if you crash, you simply ask for all messages starting from that position. If you have a new service (or a new projection), your position is 0 so you get everything.
Indeed. These are two separate patterns. They can have great synergy when used together but it's not a requirement.
> it's hard to prevent invalid state transitions being submitted to an event sourcing datastore
Apply DDD, use code review, and use stream level ACLs.