Improving Legacy Code: Using Task Queue to Speed Up a Crawler in an ETL Process(joaojunior.org)
joaojunior.org
Improving Legacy Code: Using Task Queue to Speed Up a Crawler in an ETL Process
https://joaojunior.org/posts/using-task-queue-to-speed-up-a-crawler-in-an-etl-process/
1 comments
I described how I improved an ETL process to be more than 4x faster using the same resources. While the old architecture used Python threads, the new one used task queue architecture to be more reliable and scalable. So, I will explain how we can improve a legacy code and speed it up by only modifying the architecture to run the code.