RLlib maintainer here -- We've been in the process of making many API changes over the past couple months to make it easy to modify or implement custom algorithms. The full set of changes and updated docs will be released along with ray 2.0 in August!