Using OpenBSD Relayd(8) as an Application Layer Gateway(tumfatig.net)
tumfatig.net
Using OpenBSD Relayd(8) as an Application Layer Gateway
https://www.tumfatig.net/2023/using-openbsd-relayd8-as-an-application-layer-gateway/
26 comments
In addition to the excellent man pages, I enjoyed Michael W Lucas' "Httpd and Relayd Mastery" [1].
[1]: https://www.tiltedwindmillpress.com/product/httpd-and-relayd...
[1]: https://www.tiltedwindmillpress.com/product/httpd-and-relayd...
Seconded. Nice, concise volume that helped me setup hosting for multiple sites on a single VM with different requirements (some reverse proxy, some CGI, some static).
Really looking forward to the 3rd edition of Absolute OpenBSD.
Really looking forward to the 3rd edition of Absolute OpenBSD.
> Side note, I haven't heard failover used in the context of kicking unhealthy backends before …
In OpenBSD “Failover” also appears in “Trunking a Wireless Adapter”:
https://www.openbsd.org/faq/faq6.html#Wireless
“The trunk is set up in failover mode, so either interface can be used. If both are available, it will prefer the bge0 port, since that is the first one added to the trunk device.“
In OpenBSD “Failover” also appears in “Trunking a Wireless Adapter”:
https://www.openbsd.org/faq/faq6.html#Wireless
“The trunk is set up in failover mode, so either interface can be used. If both are available, it will prefer the bge0 port, since that is the first one added to the trunk device.“
Is it for relaying any tcp/udp or just http? At first I thought it was the former but it understands http headers. Does it support any other protocols?
My understanding is that it started life as a fork(note: by same author) of openbsd's httpd. The author wanted to add reverse proxy capability to httpd and decided that the problem domain was complex enough that it would hurt httpd's simplicity and warranted it's own program.
So it's heart is a reverse http proxy. generalized to be a generic application layer proxy. and then integrated with the pf firewall to be a ip layer forwarder. note the distinction in the manual between redirections(ip forwarder) and relays(application proxy)
According to the manual it has application specific relays for http and dns. for other than that you have to use the generic tcp relay. if you are redirecting, it does not look at the internals of the packet at all.
So it's heart is a reverse http proxy. generalized to be a generic application layer proxy. and then integrated with the pf firewall to be a ip layer forwarder. note the distinction in the manual between redirections(ip forwarder) and relays(application proxy)
According to the manual it has application specific relays for http and dns. for other than that you have to use the generic tcp relay. if you are redirecting, it does not look at the internals of the packet at all.
No, you got that backwards.
https://man.openbsd.org/httpd#HISTORY
> The httpd program first appeared in OpenBSD 5.6. httpd is based on relayd(8).
https://man.openbsd.org/httpd#HISTORY
> The httpd program first appeared in OpenBSD 5.6. httpd is based on relayd(8).
Relayd is really great! Especially in combination with OpenBSD and CARP.
The original author of relayd also made a Rust prototype: https://github.com/reyk/relayd-rs
One thing to note about relayd is that it only resolves names (DNS) at startup. Meaning if the name of a host changes you will need to reload relayd.
The original author of relayd also made a Rust prototype: https://github.com/reyk/relayd-rs
One thing to note about relayd is that it only resolves names (DNS) at startup. Meaning if the name of a host changes you will need to reload relayd.
I have moved my homelab applications from Nginx Proxy Manager to relayd on my OpenBSD router, and honestly, it works very well for these small loads. It does filtering, virtual hosts and TLS, all controlled from a single file and consuming negligible amounts of memory. Scaling is rumored to be a wall most OpenBSD products hit at some point, but for my use case, it's excellent.
> Scaling is rumored to be a wall most OpenBSD products hit at some point
I suspect the vast majority of OpenBSD users never hit a "scaling wall". What do you mean by product?
In a previous life I installed two OpenBSD routers with a 10GbE card each. With carp(4) they could push nearly 20Gbps total. This was back in 2015 or so.
"Scaling" would just be matter of adding more nodes, or replace with 40GbE cards (which would require a more powerful CPU).
Perhaps running a high traffic relayd on them would have reduced the throughput. But that smells like poor architecture: let routers be routers and load balancers be load balancers :)
I suspect the vast majority of OpenBSD users never hit a "scaling wall". What do you mean by product?
In a previous life I installed two OpenBSD routers with a 10GbE card each. With carp(4) they could push nearly 20Gbps total. This was back in 2015 or so.
"Scaling" would just be matter of adding more nodes, or replace with 40GbE cards (which would require a more powerful CPU).
Perhaps running a high traffic relayd on them would have reduced the throughput. But that smells like poor architecture: let routers be routers and load balancers be load balancers :)
OpenBSD is many things but performance isn't something I rely on it for.
I'm interested to understand whether it's thread, process or singlethread event loop based. And maybe other architectural nuance that set relayd apart for example with nginx.
In case no one familiar with the internals shows up, here are some pointers if you want to dig: the configuration man page[0] has a prefork option, and nothing thread-related:
[0]: https://man.openbsd.org/relayd.conf.5
[1]: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/relayd...
[2]: https://man.openbsd.org/relayd.8
prefork number
When using relays, run the specified number of processes to handle relayed
connections. This increases the performance and prevents delays when connecting
to a relay. relayd(8) runs 3 relay processes by default and every process
will handle all configured relays.
[0]: https://man.openbsd.org/relayd.conf.5
[1]: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/relayd...
[2]: https://man.openbsd.org/relayd.8
Unless I am mistaken it's "credentialless" forwarding. I couldn't see any directive which would make the relay check user, its the IP routing mechanism to make something "inside" visible "outside" with loadshare, failover, test-before-connect, but not "gate keep"
an ssh -D SOCKS5 tunnel requires you to present credentials on the jump host. This daemon bypasses that need. (happy to be corrected. I'd love to be wrong)
an ssh -D SOCKS5 tunnel requires you to present credentials on the jump host. This daemon bypasses that need. (happy to be corrected. I'd love to be wrong)
I am not exactly sure what you want(Auth in the relayd process?) however you do mention ssh so I present you authpf.
http://man.openbsd.org/authpf
Update: I think strictly speaking auth is outside the scope of relayd, but looking at the docs, I bet I could wire something together around having specific cookies or other headers.
http://man.openbsd.org/authpf
Update: I think strictly speaking auth is outside the scope of relayd, but looking at the docs, I bet I could wire something together around having specific cookies or other headers.
match request type cookie "ident" value "hashed password"
match request header "Authorization" value "Basic dXNlcm5hbWU6dGhpcyBpcyBzZWNyZXQK"
Not sure about now, but at least a couple of years ago it didn't support SNI either.
What I love about openBSD documentation is that if it's not there in the docs, one can be sure that it doesn't exist.
What I love about openBSD documentation is that if it's not there in the docs, one can be sure that it doesn't exist.
SNI support was added with "tls keypair" option in OpenBSD 6.6: https://marc.info/?l=openbsd-cvs&m=155931636824866&w=2
I stand corrected.
It was just about time, considering there was a jestful tricking the developer of relayd to reprogram it to httpd. IIRC he got epic tricked into it! In the sense he didn't even know he was making a webserver until he actually made it. This may not be a very true version, but I love to believe so.
The camaraderie and passion in the project is mind boggling.
It was just about time, considering there was a jestful tricking the developer of relayd to reprogram it to httpd. IIRC he got epic tricked into it! In the sense he didn't even know he was making a webserver until he actually made it. This may not be a very true version, but I love to believe so.
The camaraderie and passion in the project is mind boggling.
Yes, after "its rock solid and they fix bugs, real bugs, especially security bugs FAST" the docs are consistent. It's amazing.
Control over deleting and adding HTTP headers is a nice feature.
Note to self: Compare memory usage of using stunnel with large stunnel.conf containing many backends versus relayd with large table containing many backends.
From relayd.conf manpage:
"TLS client and server
When combining both modes, TLS server and client, relayd(8) can filter TLS connections as a man-in-the-middle. This combined mode is also called "TLS inspection". The configuration requires additional X.509 certificate settings; see the ca key description in the PROTOCOLS section for more details."
Check if it's possible to disable sending SNI.
From relayd.conf manpage:
"TLS client and server
When combining both modes, TLS server and client, relayd(8) can filter TLS connections as a man-in-the-middle. This combined mode is also called "TLS inspection". The configuration requires additional X.509 certificate settings; see the ca key description in the PROTOCOLS section for more details."
Check if it's possible to disable sending SNI.
I used to use relayd to add TLS support to suckless' ii, something close to stunnel.
Looks like it’s been ported to FreeBSD also.
https://man.openbsd.org/man8/relayd.8
https://man.openbsd.org/man8/relayctl.8
https://man.openbsd.org/man5/relayd.conf.5
Side note, I haven't heard failover used in the context of kicking unhealthy backends before. Failover is typically something that would be implemented with carp(4) (aka VRRP, but with support for active/active setups) on two different machines running relayd.
https://man.openbsd.org/man4/carp.4
Nothing beats the sheer elegance of OpenBSD for networking tasks. :-)