This is one of the many cases for LLMs that I ask for the intermediate work, e.g. a script that generates random numbers, instead of asking to do the work itself.
I attempted to scrape a one page grid with 800 items and also ended up asking for the Javascript look with document query selector instead of the result as I was hitting all sort of limits, context, or the LLM would do the wrong capture, print it out and get worse responses on next prompt.
We are using a service that abstracts redis from us and requires to be treated like a critical dependency, think RDS, Aurora, Postgres, if they are down the whole site is down. Every job push is a call to this service. Upgrading the service = downtime.
For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane:
* job push: API writes to Redis
* job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
It's on the landing page, there is not even a standalone mode yet, so you can't use it with node.js you need to use with Next.js. All to say, it's just early alpha preview, I would wait the project to mature before considering using it anywhere.
I attempted to scrape a one page grid with 800 items and also ended up asking for the Javascript look with document query selector instead of the result as I was hitting all sort of limits, context, or the LLM would do the wrong capture, print it out and get worse responses on next prompt.