I built a Python job queue that stores its entire state in one JSON file on S3, GCS, or your filesystem. No Redis, no RabbitMQ, no Postgres. Just object storage.
How it works
Read the JSON file, modify it in memory, write it back with compare-and-set. If two workers race, one wins. The loser retries with fresh state. No locks needed.
How it works
Read the JSON file, modify it in memory, write it back with compare-and-set. If two workers race, one wins. The loser retries with fresh state. No locks needed.
Inspired by; https://turbopuffer.com/blog/object-storage-queue