Because spammers aren't interested in SEO best practice, if they were, they wouldn't be spam posting links in comment sections. The fact the links are nofollowed likely doesn't really matter to them or like you said, they probably don't even know what it means. The people buying these kinds of spam links are probably lied to about their worth etc
Most likely it's bots doing this kind of posting anyway, looking for common comment plugins in Wordpress or whatever and mass commenting this rubbish.
I'm not sure about this - the part of nofollows being bullshit - they have an observable effect.
Links to other sites are a key signal for Google that a site is worth showing. If all links are equal - that is, without any directives like nofollow or ugc - then it's much harder for Google to do this properly due to the existance of comment sections, user posts etc.
So observing nofollow directives on links is in Google's best interest since it's a direct signal from the site that "I don't fully trust this link, so don't think my linking to it is an endorsement". It helps search engines understand what links that particular site is linking to out of endorsment, and what links are there for other reasons such as comments etc.
I've run tests where every internal link to one specific page on a site is nofollowed. The result is that every page ends up indexed except the nofollowed page.
Site owners nofollowing every external link is just a lack of knowledge stemming from a time when it was "best practice" to keep all of your "link juice" internal, which I think you're hinting at with your mention of pagerank which is an equally outdated concept.
Agreed that spammers probably do it regardless of nofollows because one or two people might click the links though.
I don't quite understand what you mean when you say "sending users to those websites is not profitable for google" - referring to Wikipedia, Stack, Reddit etc...
Organic search results only ever profit Google if they answer the user's question, because it makes that user more likely to use Google again and click an ad in the future. If a user searches something and the top result doesn't answer their question, then Google hasn't done it's job and the user is less likely to use Google again and therefore less likely to click an ad in the future.
Surely the most profitable option for Google is to answer people's questions quickly and easily, whether you think these questions are answered by Wikipedia instead of a business site targeted to answering that question is a different story.
I just don't understand how Google's goal of "answering people's questions" is somehow opposite to "providing quality" for users. I think for most of HN, they have a very narrow view that "quality" means antiquated wiki-type sites.