I would have thought the opposite would be true, hunter gatherers are generally well fed from a variety of food sources, would have lived in low densities and would have been highly mobile, so in the event one food source become scarce they could move elsewhere, or switch foods. Early agriculturalists would have been stuck in once place reliant on a single, or few, food source/s and therefore vulnerable to failed harvests for a variety of reasons, droughts, natural disasters, theft (of stores, if they are lucky enough to farm food that can be stored) etc.