I do not agree. It's not because my door does not have locks that you can feel free to enter. And vice-versa, it's not because there are locks on a door, that you don't have the right to try to enter...
While he made a successful comeback (however far from his first success), saying he toiled in misery for years is a little bit exagerated. Indeed, he was still the owner of a house (and probably of other goods). And this house was big enough to have a value probably equal to a today million dollars. That's not what I call to be toiled in misery.