Sometimes you just need a plain old "for" over your pandas dataframe, to do a more complicated aggregation (variable window size, conditionals, ...)
You can try twisting numpy/pandas into a solution, but it might be much simpler to just write a dumb numba function which will be easier to maintain since the logic will be very clear.