Thanks all for your input.
It's a bit buffling that calendars are not a solved problem in 2021.
Going down the path of vdirsyncer + davmail with some self written filters to only show blockers (stripped everything else out) seems to be the most sensible way.