Coming from the middle class, I've never met anybody talented coming from a shitty school who didn't also get filtered into a gifted program at a good school. Maybe your school system just wasn't competent at filtering people?
Those that I know who have made bad life decisions (kids too early, dropping out of college, leaving extremely supportive parents) could totally have been expected to based off looking at GPA/SAT signals.