There are tricks to make it faster. If you find yourself working on the project again, ask for advice on IRC. 2-3 minutes is longer than it should take in most cases. (As long as you aren't touching widely included headers files, that is.)
It's a balance, but on many projects the overhead of the headers themselves is so large that concatenating a few .cpp files together doesn't increase incremental compilation time significantly over simply building each .cpp file in isolation.