Might be wrong about what Aperocky is alluding to but there is an entirely theoretical edge case. The time to the next difficulty adjustment is based on the current speed of mining, and the possible change in difficulty is capped. With enough minors leaving it will drop the speed of mining/network speed/ and push out the expected time to the next difficulty adjustment.
I can't think of any realistic way this can occur given the miners that stay will (personally) be producing blocks as often, the increase in time being balanced out by being a larger proportionate of the mining rate. They don't care if they get 1% of the blocks, which average about 20 mins per block or 5% of the blocks that average 100mins per block.