"Number: The Language of Science: A Critical Survey Written for the Cultured Non-Mathematician" really opened the field up to me and did away with some of the misconceptions I had incurred in school.
I'm currently learning group theory, matrices, and graph theory.