Here's a thought experiment, similar to the one-electron universe thought experiment: there exists one particle, the "consciousness particle," which randomly jumps around throughout 4D space. Whenever it appears at a 4D position, consciousness is experienced as computed by integrated information theory (or a similar model) [1].
When you experience a moment of consciousness at a specific 4D position, you experience all the memories of the "past" that are in your brain. Thus, the consciousness particle interpretation of consciousness is compatible with your direct perception of living a 3D life along a time dimension.
Yes, it is outrageous to claim free will is an illusion. As you point out, it contradicts a huge volume of evidence (our perception). However, a significant proportion of physicists believe free will is an illusion. What possibly could have persuaded such a large number of Earth's greatest skeptics?
When you experience a moment of consciousness at a specific 4D position, you experience all the memories of the "past" that are in your brain. Thus, the consciousness particle interpretation of consciousness is compatible with your direct perception of living a 3D life along a time dimension.
[1] https://en.wikipedia.org/wiki/Integrated_information_theory