I think the short answer to your question is that it is difficult but it can be done.
The biggest problem I have is finding large enough blocks of time (> 30 mins) to concentrate on the task in hand. The only two blocks that I've managed to secure are early in the mornings before the kids get up and on my commute to work. I use the return journey of my commute to plan what I'm going to do the next day. Although this strategy only provides me with a few hours a day, I find that this is sufficient to give my project some sort of momentum.
I tend to avoid evening development as the quality of the work I produce isn't optimal and it would eat into the time that I want to spend with my very supportive wife and children.