Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as:
- Checking the status of a task (queued, pending, failed, cancelled, completed)
- Cancelling a queued task (or pending task if the execution environment supports it)
- Re-prioritizing queued tasks
- Searching for tasks based off an attribute (e.g. tag)
I believe the problem is the lack of proper dependency indexing at PyPI. The SAT solvers used by poetry or pdm or uv often have to download multiple versions of the same dependencies to find a solution.
In a trading competition, the best strategy is the riskiest strategy. This maximizes likelihood of making a lot of money ... and losing a lot of money. But in a competition, the left tail doesn't matter. Second place and last place are identical.
Ideally, a trading competition should penalize risky investments. But this is hard to do retrospectively, especially when evaluating algorithms.
- Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag)
You really do need a database for this.