Amen. The existing spam call blockers are all based on blacklists, and it's a losing battle against spammers who can simply keep spoofing new numbers.
I'm working on an app (iphone & android) that solves this the way you suggest, via a whitelist of contacts, with an optional way for non-whitelisted, non-robocallers to still get through.
If you're interested in early access, email me at russell.davis [at] gmail [dot] [com].