I suppose it all depends on the abstractions you make and how well those abstractions hold in the real world. Humans of course make abstractions but many of these are done subconsciously.
Simulating brick laying might be able to be done in a controlled environment, is it possible to make it low cost enough and accurate enough for all general purpose brick laying situations? Probably, given enough investment we could get closer. Is ironing out all the nuances cost effective? I don't know.
We can definitely simulate a driving environment but I given the recent struggles of self driving cars I don't think I would say that we are at the point where we've solved the problem of actually driving them in everyday situations.
I think it is different from people saying we couldn't automate chess or go. The difference is between a purely data domain, chess or go, where it can both be translated 1-1 into a computer simulation of the game AND the inputs are data.
I realise that you could define everything as data - laying a brick, you take the inputs of where to position the brick, etc. However I think we can make the distinction between chess where the data is "Pawn is on e4" and the much greater complexity of the real world where we are dealing with billions of atoms. Perhaps not everyone agrees with me.