So curiosity is the primary motivation. You don't have to be first to market. That makes it easier, you can focus on making it better. Or just launch it now, you will get a different experience (e.g. Kickstarter or something). Breaking even and making money takes a long while, so you have to be able to fix the day job first.
Have you tried a lunch and learn to build up a case? Teaching people something new and answering questions is a different type of challenge and allows you to build some support to fix things that seem to be set in stone. Is there any way to do your day to day a different way? What about trying a new tool to record testing results?
I can relate, it is hard to balance the loathing/apathy, ambition, and family. Family comes first, so maybe the hours spent on the side project needs to be re-evaluated. If you have a relaxed environment, maybe you can layer your work and side project. Maybe you might find a coworker who also can help out. I would recommend talking to your manager to see what options are available (less hours, more pay, more challenging work). I like to believe good managers exist who would be able to help (somewhere, maybe just not at your place).
I try to make challenges at work by looking at things another way (e.g. switch a programming language or ask for another opinion). That will only increase stress though, so be careful because you might be in a good place if the work is easy and expectations are low.
What is your goal for the side project? Money, experience, or curiosity? Can you find another way to either get more money or more experience that won't involve you spending an extra 5 hours?
I can let you know that in my case I spent a small time looking around and evaulating options ultimately finding a slightly better job. Be optimistic.
Have you tried a lunch and learn to build up a case? Teaching people something new and answering questions is a different type of challenge and allows you to build some support to fix things that seem to be set in stone. Is there any way to do your day to day a different way? What about trying a new tool to record testing results?