There are a few mentions of Oban [1] here. Most people don't realise that Oban in fact uses SKIP LOCKED [2] as well.
Oban's been great, especially if you pay for Web UI and Pro for the extra features [3]
The main issue we've noticed though is that due to its simple fetching mechanism using locks, jobs aren't distributed evenly across your workers due to the greedy `SELECT...LIMIT X` [2]
If you have long running and/or resource intensive jobs, this can be problematic. Lets say you have 3 workers with a local limit of 10 per node. If there are only 10 jobs in the queue, the first node to fetch available jobs will grab and lock all 10, with the other 2 nodes sitting idle.
From my understanding, `work_mem` is the maximum available memory per operation and not just per connection. If you have a stored procedure with loops and/or many nested operations, that can quickly get quite big.
One trick worth noting, is that you can override the working memory at the transaction level. If you have a query you know needs more memory (e.g doing a distinct or plain sorting on a large table), within a transaction you can do:
`set local work_mem = '50MB'`
That will override the setting for operations inside this transaction only.
> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.
This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.
The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being a single node, and global across the cluster.
Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.
Adding my vote here. Throughout my career I've gone the C#.NET -> RoR -> Clojure -> Elixir/Phoenix route. After spending a year with Elixir, iterating rapidly on a production system, I'd default to it for any MVP or serious project going forward.
LiveView is also very close to becoming default over React for interactive Frontends.
A lot of people are in the right place at the right time, but very few are equipped to realise it and take advantage of it.
At the beach, waves will hit everyone in their path (right place, right time) but only those that notice it and already have momentum in the direction it's heading are able to ride it. If they have experience, and the tools (a board), they can ride it for longer without crashing.
On the tools: I have gone the route of using the libraries (e.g Kaffy for Phoenix, ActiveAdmin for Rails) and later building a central shared backoffice from scratch that any team can contribute to. In the former you are always wasting time fighting the library to get what you want, in the latter it has the advantage of being language/framework agnostic but it's too costly for smaller teams.
More recently i've used tools such as Retool (there are many alternatives), it has been ideal and allows for non engineers to build there own tools with a little bit of SQL knowledge.
Alpine.js is seeing an increase in utility as it pairs nicely with htmx[0], Phoenix LiveView[1] and the likes for lightweight interaction that doesn't require the server.