Dora: A open-source Rust DHCP Server(github.com)
github.com
Dora: A open-source Rust DHCP Server
https://github.com/bluecatengineering/dora
39 comments
Did you try Kea[0]? There are a variety of ways to ensure the same MAC addresses get the same IPs even after the initial lease expires.
I'm not surprised you had issues with dnsmasq - I wouldn't use it for anything other than a run of the mill /24 (or maybe a few of them on VLANs). Don't get me wrong - it's perfect for what it was originally intended for and I use it in at least half a dozen places. Just not for an application and network like yours.
[0] - https://www.isc.org/kea/
I'm not surprised you had issues with dnsmasq - I wouldn't use it for anything other than a run of the mill /24 (or maybe a few of them on VLANs). Don't get me wrong - it's perfect for what it was originally intended for and I use it in at least half a dozen places. Just not for an application and network like yours.
[0] - https://www.isc.org/kea/
Kea is what we ended up with. It has a few feature differences from dnsmaq that we had to work around a bit. Not the end of the world, but not fun when you're under pressure.
Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over.
The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either.
Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).
Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over.
The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either.
Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).
Frankly I'm AMAZED you made it to 9k leases with dnsmasq. Not only from a performance standpoint but the overall logging, lack of instrumentation, limits in configuration, etc. As you say (from reviewing the source) it's really not intended for anything other than small home/SMB networks, memory and CPU constrained devices, etc. It's perfect for that and it's my go-to in those scenarios. A 10k host network is a completely different animal and from the down-votes my comment received it doesn't seem like many here on HN have been in scenarios like these either.
Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well...
I don't envy your situation in the slightest!
Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well...
I don't envy your situation in the slightest!
Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
Both myself and my super talented devops coworker hadn't ever run with that many individual machines in a single location. 4-5k was both our maxes. It worked fine before, we didn't expect it to fall over.
dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location.
HN has all sorts of people, I never try to quantify it. We don't know what we don't know.
dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location.
HN has all sorts of people, I never try to quantify it. We don't know what we don't know.
[deleted]
On IPv6 this would be a solved issue. With SLAAC the router just broadcasts Router Advertisements and all the machines just automatically configure their addresses, which are fully predictable based on their MAC address (if you have privacy extensions disabled). The router has no state to keep and as such the network can be scaled up "infinitely". Just pick a random /64 prefix and set up radvd (or systemd-networkd) to send the router advertisements.
Sure. The thing is that we also have a physical location based naming system which is somewhat tied to the IPv4 address and we use that system across multiple data centers. We also have a single daemon running on each box that helps with the naming. It would complicate things (ie: more code to write) to have one dc (where we had these issues), on IPv6. We hadn't had issues before, so it never really made sense to use IPv6 other than for the novelty of it. IPv4 works surprisingly well and the issue really wasn't at that layer. Who knows too.. maybe radvd or systemd-networkd also have other issues we don't know about.
It would be great to have a better dnsmasq with sane config file, better logging, a simple statistics page and some more.
[deleted]
We've done some basic benchmarking at +5k leases/second.
If you want to go down the rabbit hole with us of looking at using Dora and giving us feedback I'd be glad to chat further cdarby[at]bluecatnetworks[dot]com
If you want to go down the rabbit hole with us of looking at using Dora and giving us feedback I'd be glad to chat further cdarby[at]bluecatnetworks[dot]com
Thanks. I've saved your project and if we are ever in the mood to switch, we will definitely give you a try first. Keep up the good work, you're solving a real problem.
Hey, I'm from BlueCat (though my opinions are my own). Will lurk the comments if there are any questions around the project or general DHCP.
A couple of notes:
If you can, add the support for CIDR style netmask (/8, /24 etc). Nothing wrong without it, but when you hop between them constantly mistakes happen.
In the example.yaml you state what option 3 is required - but it's not. It is absolutely a valid way to run some network without providing a default router.[0] If this is somehow is really a requirement anywhere than that requirement should be removed.
One of the things what I often use in MS DHCP is the server wide options, so some basic option values are configured on the server level and I add/override the needed ones on the scope level.
PS while the name is amusing, it is a possible IP conflict there, especially since you clearly state you know about Dora.
[0] And you can still provide the routign info through the option 121. Even for 0/0
If you can, add the support for CIDR style netmask (/8, /24 etc). Nothing wrong without it, but when you hop between them constantly mistakes happen.
In the example.yaml you state what option 3 is required - but it's not. It is absolutely a valid way to run some network without providing a default router.[0] If this is somehow is really a requirement anywhere than that requirement should be removed.
One of the things what I often use in MS DHCP is the server wide options, so some basic option values are configured on the server level and I add/override the needed ones on the scope level.
PS while the name is amusing, it is a possible IP conflict there, especially since you clearly state you know about Dora.
[0] And you can still provide the routign info through the option 121. Even for 0/0
> One of the things what I often use in MS DHCP is the server wide options
The config format was created primarily to be easy to generate & consume programmatically, so the format is fully 'flattened' with no real abstraction. But this may be something we add later. It's also why the option format is the way it is. I'm sure we could make some improvements there to make hand writing easier.
I appreciate your other suggestions, thanks for taking the time to look through things. I will have a look at both the router option and the CIDR input.
The config format was created primarily to be easy to generate & consume programmatically, so the format is fully 'flattened' with no real abstraction. But this may be something we add later. It's also why the option format is the way it is. I'm sure we could make some improvements there to make hand writing easier.
I appreciate your other suggestions, thanks for taking the time to look through things. I will have a look at both the router option and the CIDR input.
Looks cool, what was the motivation for writing another DHCP server - just for learning Rust or something else?
FYI /u/leshow is the core contributor of this project.
We've been doing Rust before this. We've rewritten some of our core components over to Rust. Discussed a bit in this post: https://medium.techatscale.com/open-heart-surgery-from-java-...
In terms of writing a DHCP server there was research looking through Github, and the conclusion was that there is a gap in the DHCP space. DHCPd was approaching EOL (and now is EOL). Others in the DHCP industry were calling for options, and it looked like Dora could potentially fill some of those asks: https://ns1.com/blog/isc-dhcp-is-eol-what-will-replace-it
We've been doing Rust before this. We've rewritten some of our core components over to Rust. Discussed a bit in this post: https://medium.techatscale.com/open-heart-surgery-from-java-...
In terms of writing a DHCP server there was research looking through Github, and the conclusion was that there is a gap in the DHCP space. DHCPd was approaching EOL (and now is EOL). Others in the DHCP industry were calling for options, and it looked like Dora could potentially fill some of those asks: https://ns1.com/blog/isc-dhcp-is-eol-what-will-replace-it
I dunno about "another". After some issues with dnsmasq I went looking and there aren't actually a lot of open source DHCP servers. There's like dhcpd, dnsmasq and kea.
(I actually contemplated writing my own dhcp server in Rust while debugging dnsmasq but felt like it's one of those real PITA kinds of projects. Managed to workaround whatever the dnsmasq problem was after a few hours.)
(I actually contemplated writing my own dhcp server in Rust while debugging dnsmasq but felt like it's one of those real PITA kinds of projects. Managed to workaround whatever the dnsmasq problem was after a few hours.)
Depending on your use case CoreDNS may also be an option.
is there a plugin to serve DHCP I've missed?
was currently writing a rust cli that's called by an incantation of dnsmasq --script my_bin but it's a nightmare to debug.
I didn't look at the source code. I like dnsmasq for its flexibility. I understand it. It's great. But the learning curve is very steep. Which might be because of the options I wanted in a certain way.
I'll look at this and see if it's more consistent.
Also, if anyone knows a good device to use at home with router and switch (>4 ports) and ap that runs open source firmware... let me know. I don't like SRM on the Synology, and UniFi has gone the Apple way.
I'll look at this and see if it's more consistent.
Also, if anyone knows a good device to use at home with router and switch (>4 ports) and ap that runs open source firmware... let me know. I don't like SRM on the Synology, and UniFi has gone the Apple way.
I use pfSense on a small firewall appliance that has 6 intel ethernet ports and then a AP from TP-Link that is only an AP (no dual function).
Several people have recommended GL iNet routers (OpenWRT) but I haven't tried their gear myself yet.
I looked into it some time again and the shipping to Canada was at a significant premium, like half the cost of the device itself, and the models which were available on Amazon was a small sliver of their product range :(
Hard to go wrong with opnSense, it's open and very powerful and configurable. I run multiple VLANs, wireguard and OpenVPN, DHCP with some static IP mappings, dynamic DNS for my on-prem services, LACP for link aggregation to my core switch, multiple ISP connections with policy based routing (it's not as good as SD-WAN here but it's the best I've found that's open). Worth checking out, at the least.
Whats the motivation behind this and how does it compete with ISC DHCPd and Kea? I am operating a mid sized infrastructure using ISC DHCPd and plan on the long run to migrate to Kea for its API, Stork interface and convenience tools.
ISC DHCPD is end of life [1] and unlike Kea this server is written in a memory safe language (Rust). So from a security perspective alone that's enough motivation to start a project.
[1] https://www.isc.org/dhcp/
[1] https://www.isc.org/dhcp/
I touch on this below: https://news.ycombinator.com/item?id=33895978
Wouldn't say Dora is competing. The project is in the early days and isn't feature parity to either. Some in the industry have voiced gaps & concerns that I reference in my comment.
Wouldn't say Dora is competing. The project is in the early days and isn't feature parity to either. Some in the industry have voiced gaps & concerns that I reference in my comment.
Something that is sorely missing from the FOSS world is a DHCPv6 server that supports RADIUS. The only one I am aware of that is considered production ready is Kea ($$$). FreeRADIUS has one in development but I do not know when it will be complete.
nice, there's definitely a space for some stateless DHCP server backed by netbox or something with a local cache for speed. selfishly hope to see it move. That way or at least facilitate.
my gut says DHCP<>netbox<>coredns backed by vault for x509 could be pretty sweet
my gut says DHCP<>netbox<>coredns backed by vault for x509 could be pretty sweet
dora is for:
Discover: first request broacasted by client in order to get lease (@ip and others configurations)
Offer: response bradcasted by server with the lease (@ip and others configurations)
Request: broacasted by the client in order to confirm the usage of this lease(from this server, others server can release other lease)
Ack: ack from server
Discover: first request broacasted by client in order to get lease (@ip and others configurations)
Offer: response bradcasted by server with the lease (@ip and others configurations)
Request: broacasted by the client in order to confirm the usage of this lease(from this server, others server can release other lease)
Ack: ack from server
Yup. Although, to be pedantic, the offer/ack is often unicast. Here's the relevant bit from the RFC.
> If the 'giaddr' field in a DHCP message from a client is non-zero, the server sends any return messages to the 'DHCP server' port on the BOOTP relay agent whose address appears in 'giaddr'. If the 'giaddr' field is zero and the 'ciaddr' field is nonzero, then the server unicasts DHCPOFFER and DHCPACK messages to the address in 'ciaddr'. If 'giaddr' is zero and 'ciaddr' is zero, and the broadcast bit is set, then the server broadcasts DHCPOFFER and DHCPACK messages to 0xffffffff. If the broadcast bit is not set and 'giaddr' is zero and 'ciaddr' is zero, then the server unicasts DHCPOFFER and DHCPACK messages to the client's hardware address and 'yiaddr' address. In all cases, when 'giaddr' is zero, the server broadcasts any DHCPNAK messages to 0xffffffff.
> If the 'giaddr' field in a DHCP message from a client is non-zero, the server sends any return messages to the 'DHCP server' port on the BOOTP relay agent whose address appears in 'giaddr'. If the 'giaddr' field is zero and the 'ciaddr' field is nonzero, then the server unicasts DHCPOFFER and DHCPACK messages to the address in 'ciaddr'. If 'giaddr' is zero and 'ciaddr' is zero, and the broadcast bit is set, then the server broadcasts DHCPOFFER and DHCPACK messages to 0xffffffff. If the broadcast bit is not set and 'giaddr' is zero and 'ciaddr' is zero, then the server unicasts DHCPOFFER and DHCPACK messages to the client's hardware address and 'yiaddr' address. In all cases, when 'giaddr' is zero, the server broadcasts any DHCPNAK messages to 0xffffffff.
Switched to another DHCP server and the storms went away, but I am not entirely happy with it because of the way it handles leases, it doesn't guarantee that each MAC keeps the same IP over time.
Now, I'm always on the lookout for a good DHCP server... tried another one that was golang, but it wasn't working on my network for some weird spec reason, I didn't have time to debug it and the authors didn't seem that interested in helping out.