I had almost the exact same experience. I was very into Buddhism for a time, to the point of seriously considering becoming a monk. I ultimately left the faith because I realized a) I was using it as an excuse to avoid facing my demons and b) rebirth underpins the entire philosophy. If you don't believe in rebirth, the whole thing falls apart.