There's nothing bad about SFU, particularly the version you wrote, which forms the basis of Livekit. It would be my first choice for supporting larger groups in Briefing anyway. If the traffic is E2EE, it doesn't matter if an SFU is involved. The critical part is the signalling, in my opinion. This is where the initial communication is established. In the current version of my app, whose source code is yet to be published, this can happen via an untrusted server.
Inspired by this tool I wrote something that fits macOS better. It uses the native sandbox-exec from Apple and can wrap other apps as well, like VSCode in which you usually run AI stuff. https://github.com/holtwick/bx-mac