Making 1000 HTTP requests per second in Python(blog.jonlu.ca)
blog.jonlu.ca
Making 1000 HTTP requests per second in Python
https://blog.jonlu.ca/posts/async-python-http?ref=hn
27 comments
Or switch to a language more suited for the job and you can add a „k“ to that 1000
Exactly - 1000 req/s on modern hardware sounds a lot like "Look ma, I was able to slow it down!"
What language would you use for web scraping?
Go is sort of made for that, as is NodeJS. Rust and Java are other languages I assume would make short work of a task like this.
In terms of NodeJS vs Python, specifically for web scraping, would you choose NodeJS? If so, why?
I'm more familiar with NodeJS but I'm working with a team that is leaning towards using Python for web scraping, so that's why I'm asking. They said spinning up multiple processes in Python is easier so at scale it will work better.
I know you can use the Cluster module to have child processes in NodeJS, but in my experience it's a bit of a pain to use, although it's not always required to use anyway, at least when only using NodeJS as a web server (as long as you have multiple NodeJS instances, in case one goes down). Web scraping is a bit different though.
Curious if you have any thoughts on this.
I'm more familiar with NodeJS but I'm working with a team that is leaning towards using Python for web scraping, so that's why I'm asking. They said spinning up multiple processes in Python is easier so at scale it will work better.
I know you can use the Cluster module to have child processes in NodeJS, but in my experience it's a bit of a pain to use, although it's not always required to use anyway, at least when only using NodeJS as a web server (as long as you have multiple NodeJS instances, in case one goes down). Web scraping is a bit different though.
Curious if you have any thoughts on this.
NodeJS (V8) is faster than CPython, and NodeJS was built for this precise use case. I wouldn't use it over Python personally though, I don't use NodeJS and would sooner reach for Rust if performance mattered.
Concurrency is an important limitation as you've noticed, but it's already a problem for CPython. You would be able to squeeze out more req/s from NodeJS than CPython, up to a point where you would need to bring in something extra to scale to all the cores of one machine (multiprocessing in Python, something like Cluster in NodeJS) which you wouldn't need in Go/Rust/Java.
Then of course scaling further, you would need a system to run jobs across machines, and your choice of Go over Python wouldn't necessarily matter so much. The difference in performance wouldn't limit what you can do, it would just change what you pay for compute. If your compute costs more but your devs can implement features faster, performance is usually unimportant.
Concurrency is an important limitation as you've noticed, but it's already a problem for CPython. You would be able to squeeze out more req/s from NodeJS than CPython, up to a point where you would need to bring in something extra to scale to all the cores of one machine (multiprocessing in Python, something like Cluster in NodeJS) which you wouldn't need in Go/Rust/Java.
Then of course scaling further, you would need a system to run jobs across machines, and your choice of Go over Python wouldn't necessarily matter so much. The difference in performance wouldn't limit what you can do, it would just change what you pay for compute. If your compute costs more but your devs can implement features faster, performance is usually unimportant.
I worked on a project which required some medium scale web scraping (less than 100 million pages), and went with node primarily because of puppeteer.
The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.
Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)
The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.
Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)
> Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled
I wonder if these kind of processes are cheaper on Lambda.
I wonder if these kind of processes are cheaper on Lambda.
[deleted]
I’ve never done web scraping professionally, but I can relay that any time I’ve tried to use Python for anything involving concurrency, except in the most trivial cases, it has been pure pain.
I can see Go hitting large numbers, but would Node.js really be comparable in throughput?
I’ve heard V8 is fairly optimised, but single thread (regardless of async) is a fairly big hit over Go’s green threads right?
I’ve heard V8 is fairly optimised, but single thread (regardless of async) is a fairly big hit over Go’s green threads right?
It probably wouldn't beat Go, Rust, or Java. It would certainly beat Python though.
Elixir. Parallelism is 99% free, it's super easy to make almost any task parallel. And it's one of the fastest dynamic languages too, surpassing Node.JS in places.
I prefer Erlang for serious networking-related code. Go for less serious ones.
I wonder where the time is going. I suspect that something like atop would show that the CPU is idle for much of the time, which suggests that the problem is not Python or even the GIL.
If that's true, there's something ineffective about the way that the requests are being made/handled.
AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.)
There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining.
It might also be useful to use something like epoll to wait for the sockets.
If that's true, there's something ineffective about the way that the requests are being made/handled.
AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.)
There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining.
It might also be useful to use something like epoll to wait for the sockets.
The author very much did try to do that, doing requests with async APIs is often the most performant way to do them because under the hood they use IO async primitives like epoll/kqueue/iocp depending on the environment they run on.
If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time.
TL;DR: network I/O is fast, json is a really slow format to parse.
If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time.
TL;DR: network I/O is fast, json is a really slow format to parse.
Json decoding will show up as CPU usage.
Epoll to decide which socket to read is just one thing - efficiently using sockets also matters, hence my suggestion of communication pipelining.
Epoll to decide which socket to read is just one thing - efficiently using sockets also matters, hence my suggestion of communication pipelining.
If anyone is wondering why the threaded implementation was slower
> pool = ThreadPool(40)
> pool = ThreadPool(40)
Many people still don't understand you can run millions of threads on Linux on a beefy server. The bottleneck with python will always be the CPU doing something with those requests.
To be fair they did try multiple concurrency factors for asyncio and found that 60 was fastest, so this is not too far. Hopefully.
How is python doing a thread pool with the GIL?
> How is python doing a thread pool with the GIL?
All but one of the runnable threads are waiting on the GIL. The "not runnable" threads are waiting on some lock or a blocking system call.
All but one of the runnable threads are waiting on the GIL. The "not runnable" threads are waiting on some lock or a blocking system call.
So it's a single thread event loop? which means 1 cpu utilization?
I get a lot of people have gotten mileage out of event loop scaling over the last 10-15 years (that's the trick of nginx, node.js, etc, right?), but even raspberry pis are quad core?
I get a lot of people have gotten mileage out of event loop scaling over the last 10-15 years (that's the trick of nginx, node.js, etc, right?), but even raspberry pis are quad core?
Some C extensions run even when they don't have the lock. Also, the OS gets to run.
If you have a lot of parallel computation, Python may not be a great choice. However, if you're managing a lot of IO with not much computation, Python is perfectly reasonable.
If you have a lot of parallel computation, Python may not be a great choice. However, if you're managing a lot of IO with not much computation, Python is perfectly reasonable.
Takes 1.8 seconds here. But will obviously depend on the network situation and server. And there probably is something faster than wget.