Agreed. I have always thought about refactoring as developer responsibility. If it needs to be done do it while working on real feature and update deadlines accordingly. That way it is way easier to justify it because you talk about it only with technical people. In long run it makes code base way better. This results in easier maintenance and faster development of new features.