The "infinite depth" seems to be a matter of definition. It's practically infinite if you include feedback loops via learning.
If you exclude learning, then it's far from "infinite". Activations linger for up to 15-30 seconds, so at oscillations of around 30 Hz that would result in about 450-900 loops (times an unknown small multiplier for the actual number of layers). But the brain presumably only backprops/optimizes a few layers at a time and not much "through" time.