You have every reason to take pride in your work and how far you've come, but you're missing the point. "Slave labor" is of course an exaggeration (and an exaggeration that shouldn't be made lightly) but the point is you didn't have as much freedom in that job as your Western colleagues.
You had to work harder than the Swedes because the alternative was worse for you. It sounds like you're happy for your time at that company, but a lot of H1B workers find themselves in unpleasant work conditions that they don't have the same freedom to leave as their American counterparts. Their employers know they can subject them to a worse environment because they have more riding on it, whereas an American worker can simply leave and find a better job.
That's the point. It's more like indentured servitude than slavery in that regard anyway. The problem isn't that it doesn't make anyone's lives better in the end, the problem is that it's of concerning ethics on the part of the employer.