This is interesting! Have you considered leveraging Google OR Tools[1] for route optimization? At a previous hyper-local eCommerce startup I worked for, we used it to solve similar problems. Although the setup and integration is not super easy, but the results far outweighed the effort.
1 - https://github.com/google/or-tools