Open-source network emulators and simulators(brianlinkletter.com)
brianlinkletter.com
Open-source network emulators and simulators
https://www.brianlinkletter.com/2023/02/network-emulators-and-network-simulators-2023/
54 comments
Does anyone know if any of these tools support emulating wifi radios such that a person could test, say, hostapd configurations, independently or in the context of a network to make sure all the pieces are working together? (That is, without ever connecting anything to a live wireless card)
Brian Linkletter's website is one of the best portal if you want to learn network simulation and emulation.
From the provided list, I think the most promising and realistic open source network simulator and emulator is Mininet due to its lightweight approach capable of running off-the-shelf Linux OS and software. In addition it has thriving eco-system that made it unique, accessible and prime for wide adoption for examples:
a) It's currently actively being used to simulate and emulate smart city project by its original author to improve urban networking [1],[2].
b) It has open learning materials resources from open source book (Mininet-WiFi) and open lab instructions manual (Mininet-SDN) [3],[4].
c) Its lightweight feature enabling it to simulate very large number of nodes using the nested containers capability as demonstrated by extensions created by third party simulation tools namely NestedNet and ComNetsEmu [5],[6],[7],[8].
[1] SigComm 2022 Cosmos Testbed Tutorial:
https://wiki.cosmos-lab.org/wiki/Workshops/SigComm2022
[2] Reconfigurable topology testbeds: A new approach to optical system experiments:
https://www.sciencedirect.com/science/article/pii/S106852002...
[3] Mininet Wi-Fi book:
https://github.com/ramonfontes/mn-wifi-ebook
[4] SDN Labs using Mininet manual:
http://ce.sc.edu/cyberinfra/workshops/Material/SDN/SDN_Labs....
[5] Network Virtualization and Emulation using Docker, OpenvSwitch and Mininet-based Link Emulation:
https://scholarworks.umass.edu/masters_theses_2/985/
[6] NestedNet: A Container-based Prototyping Tool for Hierarchical Software Defined Networks:
https://ieeexplore.ieee.org/document/9244858
[7] ComNetsEmu: A Lightweight Emulator:
https://www.sciencedirect.com/science/article/abs/pii/B97801...
[8] A virtual emulator/testbed designed for the book: Computing in Communication Networks: From Theory to Practice (2020):
https://git.comnets.net/public-repo/comnetsemu
From the provided list, I think the most promising and realistic open source network simulator and emulator is Mininet due to its lightweight approach capable of running off-the-shelf Linux OS and software. In addition it has thriving eco-system that made it unique, accessible and prime for wide adoption for examples:
a) It's currently actively being used to simulate and emulate smart city project by its original author to improve urban networking [1],[2].
b) It has open learning materials resources from open source book (Mininet-WiFi) and open lab instructions manual (Mininet-SDN) [3],[4].
c) Its lightweight feature enabling it to simulate very large number of nodes using the nested containers capability as demonstrated by extensions created by third party simulation tools namely NestedNet and ComNetsEmu [5],[6],[7],[8].
[1] SigComm 2022 Cosmos Testbed Tutorial:
https://wiki.cosmos-lab.org/wiki/Workshops/SigComm2022
[2] Reconfigurable topology testbeds: A new approach to optical system experiments:
https://www.sciencedirect.com/science/article/pii/S106852002...
[3] Mininet Wi-Fi book:
https://github.com/ramonfontes/mn-wifi-ebook
[4] SDN Labs using Mininet manual:
http://ce.sc.edu/cyberinfra/workshops/Material/SDN/SDN_Labs....
[5] Network Virtualization and Emulation using Docker, OpenvSwitch and Mininet-based Link Emulation:
https://scholarworks.umass.edu/masters_theses_2/985/
[6] NestedNet: A Container-based Prototyping Tool for Hierarchical Software Defined Networks:
https://ieeexplore.ieee.org/document/9244858
[7] ComNetsEmu: A Lightweight Emulator:
https://www.sciencedirect.com/science/article/abs/pii/B97801...
[8] A virtual emulator/testbed designed for the book: Computing in Communication Networks: From Theory to Practice (2020):
https://git.comnets.net/public-repo/comnetsemu
The most important and most useful thing is the hardest to obtain: latency simulators.
I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.
This is a critical aspect of application performance that all developers just ignore because it's such a PITA to simulate.
Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!
Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.
I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.
This is a critical aspect of application performance that all developers just ignore because it's such a PITA to simulate.
Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!
Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.
Linux's tc-netem [0] is designed for this:
"The netem queue discipline provides Network Emulation functionality for testing protocols by emulating the properties of real-world networks.
The queue discipline provides one or more network impairments to packets such as: delay, loss, duplication, and packet corruption."
[0] https://man7.org/linux/man-pages/man8/tc-netem.8.html
"The netem queue discipline provides Network Emulation functionality for testing protocols by emulating the properties of real-world networks.
The queue discipline provides one or more network impairments to packets such as: delay, loss, duplication, and packet corruption."
[0] https://man7.org/linux/man-pages/man8/tc-netem.8.html
tc-netem is too complicated to set up and its performance profile is very different to real networks.
I mean this in the sense that it can’t even emulate a “long cable” properly.
I’ve tried both the Linux and BSD tools mentioned by others here and they’re all very old, weird, and difficult.
I mean this in the sense that it can’t even emulate a “long cable” properly.
I’ve tried both the Linux and BSD tools mentioned by others here and they’re all very old, weird, and difficult.
IPFW dummy net in FreeBSD was amazing for this. I haven't used it in 20 years. On my home DSL line in the 90s I would get so many hack attempts that it was becoming a nuisance, so I filtered ICMP through a dummy net that both delayed and dropped 10% of the packets. Hack attempts went to zero in 3 days.
Luigi Rizzo is a top notch hacker.
https://github.com/luigirizzo/dummynet
Luigi Rizzo is a top notch hacker.
https://github.com/luigirizzo/dummynet
[deleted]
I’ve noticed that Hacker News is a lot faster than other sites and apps on a bad connection probably because it’s just basic HTML
Basic HTML isn’t static HTML. The comments and posts for Hacker News still needs to be stored and loaded from a database.
That doesn’t happen over my mobile data connection so I don’t see how that’s relevant to my point
Afaik hacker news pages are stored on the file system.
Well, kind of, at least considering the last public version of Arc, that HN uses (found here: http://arclanguage.org/)
It seems to be storing stuff directly on disk, on the same host that the software itself runs on.
So you're right if you consider the filesystem a sort of database, but otherwise no :)
It seems to be storing stuff directly on disk, on the same host that the software itself runs on.
So you're right if you consider the filesystem a sort of database, but otherwise no :)
It was announced in 2008! 15% into the 100 year language.
Not sure their tech stack (I'm sure there's a post on it somewhere) but there was a forum software way back that would compile replies and such into static html pages when the submit button was hit, on the assumption that they would be read a lot more times than written to. Worked pretty well from what I recall.
Great idea in theory until you want to change the presentation and now have to recompile 3̵7̵8̵4̵3̵1̵0̵5 (37843314) items :)
True, but it was split into pages so at most 20 items :)
That doesn't matter for the client through. As long as the data can come in a single small file, for example the <3KB HTML file for the tab I'm writing this on, what the server is doing doesn't matter.
Because not every letter is a 500 line, 30 dependency component, in other words
I suppose any proxy can be used in combination with https://www.linux.org/docs/man8/tc-netem.html but it is a bit tedious.
Perhaps one of the linked projects can help with it?
Perhaps one of the linked projects can help with it?
There is clumsy which simulates bad network http://jagt.github.io/clumsy/
And comcast: https://github.com/tylertreat/comcast
This looks great! Besides that it seems to be only for Windows and requiring a GUI to configure the values...
Anyone know of something similar for Linux that can be configured via config/env vars/cli?
Anyone know of something similar for Linux that can be configured via config/env vars/cli?
there's labrea which is pretty dusty: https://github.com/dustin/labrea
you could probably achieve something similar with bpf today.
you could probably achieve something similar with bpf today.
Your proxy adding latency is a very good idea. I'm wondering what it would be wrote in. Go, so you can integrate it easily on a k8s as a side container or else ? Rust, so it can be performant and at low level ? Nim, so it can be easy to write and performant ?
I think that can't be that difficult to write, but maybe I'm wrong.
I will look into it but I don't promise anything. I think it is a good pet/side project.
> I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.
About 15 years ago I needed to emulate a network over a satellite link. We had limited amount of time on the dish and it was a fairly costly affair. We had a small rack of hardware together with a bunch of measuring instruments that would be in the field for data acquisition. It would be sending back data home where it would be processed and then sent back into the field. The (limited) bandwidth reserved and the inherent latency on the link gave us some interesting issues to deal with, but it required a few iterations to get things working smoothly.
The rack had a Linux box which acted as a router (among other things), and while it was in the office we'd just hook it up via ethernet. So I used tc[0] on there to introduce a fixed latency on transmission and cap the outgoing bandwith to whatever was available on the link. I did the same on the homestation for outgoing traffic but there I just used an old box with two ethernet ports and set it up as a bridge.
For dropping packets I used iptables, and some other things.
It requires some familiarity with the LARTC[1], which isn't the most readable document, and how things work in Linux. It gave us exactly what we needed without having to pay for time on the dish for testing.
It's been over a decade and the details are extremely vague, but I'm sure that if you want to you could mark certain packets with netfilter and then delay those packets somehow, rather than just delay the entire network device. I remember that with iptables we used to mark packets from SSH in the mangle table and then use that mark with tc to give traffic marked priority on the outgoing device.
It's not exactly a proxy, but back in the day it worked for that specific use case.
[0] : https://linux.die.net/man/8/tc
[1] : https://tldp.org/HOWTO/Adv-Routing-HOWTO/
About 15 years ago I needed to emulate a network over a satellite link. We had limited amount of time on the dish and it was a fairly costly affair. We had a small rack of hardware together with a bunch of measuring instruments that would be in the field for data acquisition. It would be sending back data home where it would be processed and then sent back into the field. The (limited) bandwidth reserved and the inherent latency on the link gave us some interesting issues to deal with, but it required a few iterations to get things working smoothly.
The rack had a Linux box which acted as a router (among other things), and while it was in the office we'd just hook it up via ethernet. So I used tc[0] on there to introduce a fixed latency on transmission and cap the outgoing bandwith to whatever was available on the link. I did the same on the homestation for outgoing traffic but there I just used an old box with two ethernet ports and set it up as a bridge.
For dropping packets I used iptables, and some other things.
It requires some familiarity with the LARTC[1], which isn't the most readable document, and how things work in Linux. It gave us exactly what we needed without having to pay for time on the dish for testing.
It's been over a decade and the details are extremely vague, but I'm sure that if you want to you could mark certain packets with netfilter and then delay those packets somehow, rather than just delay the entire network device. I remember that with iptables we used to mark packets from SSH in the mangle table and then use that mark with tc to give traffic marked priority on the outgoing device.
It's not exactly a proxy, but back in the day it worked for that specific use case.
[0] : https://linux.die.net/man/8/tc
[1] : https://tldp.org/HOWTO/Adv-Routing-HOWTO/
tc is the way to go if the options are sufficient. You can bridge two interfaces (also virtual ones) in a VM and set qdiscs. Also works asynchronously, e.g., different rules for up/downlink. Instead of a VM setup one could use a cheap linux/openWrt router or an SBC-type board.
Tc is always a pain to search for, I add 'netem' as a recommendation for keywords and also the IFB trick to put a qdisc on an input stream.
For web apps, chrome has a throttling tool in the dev tools. Not sure how sophisticated it is, but I've used occasionally to recreate bugs only noticed by clients on slower connections.
Huh? FreeBSD had this 15 years ago already (if not more). Thingie is called DUMMYNET. Imunes utilizes netgraph thingie that can be connected thro pipes (DUMMYNET) to emulate whatever delay, bandwidth and BER you want. Very handy thing
to check out how your stuff will behave on dialup for example.
This is my BGP troll lab for some testing ;)
http://ds-1.ovh.uu3.net/~borg/pics/bgp-lab.png
This is my BGP troll lab for some testing ;)
http://ds-1.ovh.uu3.net/~borg/pics/bgp-lab.png
Does 'tc' fit your need? its a simple solution to that sort of need:
https://man7.org/linux/man-pages/man8/tc.8.html
On Cloud you could look at the AWS Fault Injection Simulator. This example is in the context of EKS but you could use it for EC2.
"AWS Fault Injection Simulator supports chaos engineering experiments on Amazon EKS Pods" - https://aws.amazon.com/blogs/containers/aws-fault-injection-...
"AWS Fault Injection Simulator supports chaos engineering experiments on Amazon EKS Pods" - https://aws.amazon.com/blogs/containers/aws-fault-injection-...
I use this to simulate delays between various local services:
https://github.com/Shopify/toxiproxy
If you have Docker all you need is a few terminal commands
https://github.com/Shopify/toxiproxy
If you have Docker all you need is a few terminal commands
> I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.
Built-in into linux kernel, althought it's per interface unless you get dirty with traffic shaping tools.
Also you probably dont want proxy as support for proxy servers can be pretty iffy across software stack.
> Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!
Reminds me of a time when COVID hit and the suddenly-remote developers were suprised the time to start of their application went from 5 to 30 minutes (they connected their app to DB server on dev environment to not carry big-ass database with them)...
> Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.
And that reminds me of this time frontend developers loading 700MB of data to load the site and not noticing that, but sending ticket to ops "server is a bit slow, could you give it more cores?". Said developers were working in office that had direct connection to data center 500m away with sub-ms latency and plenty of bandwidth.
I had many times pondered traffic shaping frontend devs connection to like 10Mbit after that... maybe 56kbit for repeat offenders.
Built-in into linux kernel, althought it's per interface unless you get dirty with traffic shaping tools.
Also you probably dont want proxy as support for proxy servers can be pretty iffy across software stack.
> Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!
Reminds me of a time when COVID hit and the suddenly-remote developers were suprised the time to start of their application went from 5 to 30 minutes (they connected their app to DB server on dev environment to not carry big-ass database with them)...
> Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.
And that reminds me of this time frontend developers loading 700MB of data to load the site and not noticing that, but sending ticket to ops "server is a bit slow, could you give it more cores?". Said developers were working in office that had direct connection to data center 500m away with sub-ms latency and plenty of bandwidth.
I had many times pondered traffic shaping frontend devs connection to like 10Mbit after that... maybe 56kbit for repeat offenders.
[deleted]
Nice sum up of simulators and emulators.
My personal favorite is IMUNES. Works very nicely as VM and its lightweight. You can run 100s of instances without problem.
My personal favorite is IMUNES. Works very nicely as VM and its lightweight. You can run 100s of instances without problem.
Really nice list. Back in the day when I started out and did a bit of CCNA material I used the proprietary Cisco Packet Tracer a lot. Ultimatively I did not follow the networking route for various reasons, but if I would have, then GNS3 would have been my simulator of choice. You can load the original Cisco firmware into it (if you can get hold of it) and really study in depth.
I used to use a network Emulator+Physics/Traffic Simulator called NCTUns to research vehicle networking back in 2010.
I remember it was pretty cool. If I recall correctly, I could run several instances of all kinds of networking gear, configure wireless interfaces, place physical barriers, roads, etc... then I could see how packets and routing tables would behave while the emulated instances of routers/switches/APs moved around the physical world.
Pausing the would, clicking on a little car, and serialing into a particular router that was running Cisco IOS was always pretty mind blowing to me though.
They had all kinds of custom protocols that could route network traffic depending on which direction the traffic was flowing: you didn't want packets to get stuck in loops, hopping back and forth between oncoming traffic. Especially in sparse environments.
I don't know what happened to it, it looks like it got spun off into a commercial company:
http://nsl.cs.nctu.edu.tw/NSL/nctuns.html
I remember it was pretty cool. If I recall correctly, I could run several instances of all kinds of networking gear, configure wireless interfaces, place physical barriers, roads, etc... then I could see how packets and routing tables would behave while the emulated instances of routers/switches/APs moved around the physical world.
Pausing the would, clicking on a little car, and serialing into a particular router that was running Cisco IOS was always pretty mind blowing to me though.
They had all kinds of custom protocols that could route network traffic depending on which direction the traffic was flowing: you didn't want packets to get stuck in loops, hopping back and forth between oncoming traffic. Especially in sparse environments.
I don't know what happened to it, it looks like it got spun off into a commercial company:
http://nsl.cs.nctu.edu.tw/NSL/nctuns.html
Dumb question, has anyone tried LLMs for network design? I got a CCNA few years ago and I remember being surprised that most of that stuff wasn't automated already. I feel like most things that CCNA or even CCNP require can be done by a LLM if it's trained on enough examples.
[deleted]
If you're using a tool that drives headless chrome, there's also programmatic access to Network.emulateNetworkConditions... the same knobs used for network throttling in the Chrome dev tools network tab.
https://chromedevtools.github.io/devtools-protocol/tot/Netwo...
https://chromedevtools.github.io/devtools-protocol/tot/Netwo...
Does anyone have recommendations if you want to simulate poor/unreliable/slow networks in order to ensure a client application handles these scenarios? Would a network emulator be the appropriate tool for this?
You can configure the Linux network stack to include packet loss and latency using tc.
Not sure about other os.
https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropp...
Not sure about other os.
https://www.pico.net/kb/how-can-i-simulate-delayed-and-dropp...
Look at using pf on OpenBSD. Specifically, look at the probability and set delay options: https://man.openbsd.org/pf.conf
Couple this with inserting pf as a transparent firewall and you can make a good hop become all sorts of crappy. And you can do this on very low end hardware with just a pair of NICs.
Couple this with inserting pf as a transparent firewall and you can make a good hop become all sorts of crappy. And you can do this on very low end hardware with just a pair of NICs.
FreeBSD with dummynet. Has always worked great for testing different network conditions.
+1 for GNS3. I used it to mimic my home network (MikroTik gear) and test out VLAN configs. Great tool, quick to learn, easy to use, works flawlessly on Linux!
eve-NG is still the easiest to use IMHO, me and many of my coworkers used it to prepare for our CCNA/CCNP/CCIE certifications.
I would swap it in an instant for an equally user-friendly open-source one though, if there was anything comparable, as some of the features lacking in eve-NG Community were really annoying to deal with (e.g. no connection to a real, physical network and/or the internet)
I would swap it in an instant for an equally user-friendly open-source one though, if there was anything comparable, as some of the features lacking in eve-NG Community were really annoying to deal with (e.g. no connection to a real, physical network and/or the internet)
Thank you for this list. I started my career with Opnet (Riverbed purchased them in 2013) and IIRC the license cost mid five figures per year. This list tipped me to OmNet++ which may provide a similar DES environment. I look forward to checking it out.
CORE has been updated twice since the list was created. It’s actively maintained. CORE emulates mobile, changing networks like IoT devices, phones, or network-connected vehicles.
GitHub - https://github.com/coreemu/core
Community Discord - https://discord.gg/AKd7kmP
GitHub - https://github.com/coreemu/core
Community Discord - https://discord.gg/AKd7kmP
I also wrote a now somewhat out of date cautionary note about all the pitfalls you can land into here: https://www.bufferbloat.net/projects/codel/wiki/Best_practic...
And over here, while we convinced systemd to adopt fq_codel by default, I cautioned against drawing conclusions from within the box simulations verses the general internet's behaviors: https://github.com/systemd/systemd/issues/9725#issuecomment-...
I am perversely glad measuring this stuff irks so many other people besides me!