gst = f"rtspsrc location={video_url} latency=0 buffer-mode=auto ! decodebin ! videoconvert ! appsink max-buffers=1 drop=true"
self.video = cv2.VideoCapture(gst, cv2.CAP_GSTREAMER)
When testing, I also found out that the codec and image settings of the camera matter. With a h264 stream, the images came in batches of a number of frames, whereas MJPEG provided a more constant image stream with lower latency. Lastly, disabling 3D noise reduction also removed some delay.
[0]: https://bower.sh/you-might-not-need-tmux