Recommender systems moved from explicit feedback (like ratings) to implicit feedback precisely because users are less likely to actually rate stuff and also because ratings are subjective; by which I mean your interpretation of 3 stars(good) may not agree with mine(average). I have watched tons of movies/shows on netflix or videos on YT for that matter but have not rated a single video.
To address the other part of your suggestion i.e collapse ratings/feedback across media like movies,books,etc. usually it is very difficult to have a dataset that spans multiple media across the same set of users. Even if it is present it would be too sparse (more sparse than usual for a site like YT with a continuously changing content library) to actually help. Though I agree that if anybody can get the recommender right, it is Google with the sheer amount of info it has on each user.