Keep in mind that this is just step one of many we'd like to take with this plugin. It's not that we don't care about statistical significance - it just didn't make it into this version, so we'll work to get that in there soon.
We will also be adding more in the FAQ tab that talks about these sorts of issues where users could possibly make decisions too soon instead of letting the tests gather more data.
Yes, visitors get a cookie that has the value of the page they should see on return visits. For instance, if a visitor ends up on variation page #2, they will see variation page #2 from now on. Or at least until they reach the conversion page or the cookie expires in 30 days.
Trying to get your startup going on the side while maintaining your current job is certainly an attractive option for many people. But the risk with that approach is that the startup never gets launched and off the ground. They tend to stall and stall and stall, all because they have that big safety net of their current job and salary. It can work, but not always.
Fair enough, but you can fail in something other than a bootstrapped startup and learn from it, all without risking your own personal stress, finances, etc.
I mentioned it in another comment, and maybe it's not clear from my post, but I'm married (14 years) and have 3 kids (ages 8, 10, and 12). Being bootstrapped with a family like that probably requires more adjustments than for those who aren't married and don't have kids.
But you bring up a good point about prepping for those changes before you get into it. If you have the notion to start down that path, then making those adjustments ahead of time will help ease some of the "pain".
Not sure if this was meant for me or not, but I hope I didn't come off as complaining in that post. I just wanted to make sure people thought about the personal side of things before diving into it. People tend to overlook the spouse's point of view, and I wanted to make sure I pointed that out.
We will also be adding more in the FAQ tab that talks about these sorts of issues where users could possibly make decisions too soon instead of letting the tests gather more data.