1. Often if you move house they will let you transfer your mortgage (actually take out a similar product and avoid the refi costs).
2. If you win the lottery, or otherwise are in a situation to prepay your whole mortgage? Well you might care much less about the 5% early fees in that case.
3. You can prepay usually 10% a year, at their discretion, without any early fees. If you just become a fair bit richer than you expected, prepaying at this level will reduce the mortgage quite quickly.
Note that the upper quartile having $800k includes $500k which isn't in their primary residence. Older people tend to own cheaper houses than you might think, since they are more likely to live in more rural areas.
If you're 70 and have a house you might be able to live quite a few years on half a million. In practice, you have a decent chance of dying before you spend all your capital.