There are patterns (prerendering, streaming (https://jakearchibald.com/2016/streams-ftw/#streaming-result...), etc.) other than simply breaking up the JS app bundle into smaller pieces that can alleviate the traditional "SPA concerns" — hopefully you can just reduce its size overall.
For other thoughts and ideas regarding SSR pros and cons, checkout this presentation given at Polymer Summit (http://www.youtube.com/watch?v=wYGoJ8R3nnM&t=5m35s).