There are more efficient ways to achieve the same and get a bunch of other positive side effects in one go: Land Value Taxation. Henry George proposed it already in 1879 in his book "Progress and Poverty".
Only that doesn’t happen in real life. People on the brink of poverty don’t inherit property. And if they do, they can always sell it and put that money to better use.
You most certainly should be paying the equivalent of 10 households. You deny 9 others the ability to use the land.
If you have real use for the land, paying the lvt won’t be an issue. If you can’t use it efficiently, an lvt will incentivise you to sell what you don’t use.