GBM already IS a userspace layer than abstracts over driver-specific kernel interfaces. That's the whole point.
You also seem to have missed the whole point of wayland's design. There is no separation between "window managers" and "display servers"; there are just "compositors". So naturally it is something that needs to be dealt with.
wlc, libweston, wlroots (for which I am an author, responsible for the DRM/GBM code), etc., are designed to make it easier to write compositors and allow for more code reuse.
So yes, someone using one of these libraries wouldn't need to deal with these details, but we're actually writing such a library.
You also seem to have missed the whole point of wayland's design. There is no separation between "window managers" and "display servers"; there are just "compositors". So naturally it is something that needs to be dealt with. wlc, libweston, wlroots (for which I am an author, responsible for the DRM/GBM code), etc., are designed to make it easier to write compositors and allow for more code reuse. So yes, someone using one of these libraries wouldn't need to deal with these details, but we're actually writing such a library.