Nevertheless, if you block all the ~850nm IR light from reaching the lens, then there cannot be any retro reflection at this wavelength, meaning the ToF cannot see anything.
IR cut filters used in smartphone-style cameras are typically reflective.
Installing an absorptive IR cut filter on top of the lens would decrease the amount of reflected light, and might hinder your approach. Those are pretty cheap to buy, so you could try it out pretty easily.