It sounds like you're already approaching this idea on your own, but if you haven't checked it out already you should have a look at Digital Minimalism [1]. It's a really well thought out analysis of exactly what you're describing. In theory, a few hours here and there shouldn't be an issue. However, the main problem with modern media (social and otherwise) is that it is very insidious. It's not just about the time spent indulging, but also what that indulgence does to your mental state throughout the rest of the day.
[1] https://www.calnewport.com/books/digital-minimalism/