There exist pauseless GCs, Apple's objection to GC is that GCs need more RAM, and that would reduce their profit margin on iPhones. A fair reason if you ask me.
You've hit the nail on the head, but I take the opposite conclusion.
Swift is designed around premature optimization. In my experience, improving programmer productivity is more important. The more time spent with compiler enforced busywork is less time in Instruments. The same goes for compile times. If you want people to write fast code then help them to iterate quickly.