Do you know what the monthly minimum for Braintree processing was? The last quote I got from them was $200/month if you're processing under $1mm/year. I've also heard of a couple recent cases in which it was negotiated down to $100 though.
PayPal is by far the most cost-effective way (in terms of % rates) to get up and running - compared to the time and costs involved in setting up a merchant account/payment processor.
As Aaroneous mentioned- Paypal Website Payments Pro allows you to process paypal payments and credit card charges invisibly in the background (there is no redirect to Paypal).
The processing fees/cost economics of establishing a merchant account/processor definitely make sense however once you're processing several $k/month.
There are very, very few markets that it is truly winner take all. Ex. even Airbus/Boeing, Microsoft/Linux/Apple, etc
For most part, I find that ideas are essentially worth the paper that they're scribbled on. It almost always comes down to execution. Go for it! You'll never know until you try, otherwise you'll always be saddled with the shoulda, coulda, wouldas(particularly if that startup competitor ends up exiting in a big way down the road). I learned my lesson the hard way.
Remember, sometimes its better to be a fast follower (and faster at reiterations) than a first mover.
Braintree is awesome with Rails apps (and particularly rock solid if you needed to implement recurring charges). The only issue is you would need to process $1+ mm through your site, otherwise, there'd be an additional monthly charge. Highly recommended however.
Also, I'd assume that you're integrating the APIs through ActiveMerchant? I'd definitely use that regardless of which reseller/merchant provider you choose.
Advertising and premium accounts need not be exclusive. You could try both with a freemium model (see Chris Anderson's recent WSJ piece). Place ads on the free version and take them away in the premium (as well as adding other features of course)