GoodJob – a Postgres-based ActiveJob back end for Ruby on Rails(island94.org)
island94.org
GoodJob – a Postgres-based ActiveJob back end for Ruby on Rails
https://island94.org/2020/07/introducing-goodjob-1-0
95 comments
Rails was always great, JS just paid a lot more...
It felt great to get back into it having people pay to migrate their Rails apps to React and Node.
It felt great to get back into it having people pay to migrate their Rails apps to React and Node.
What's great about Rails ? Serious question from someone who landed into a mature Rails app and is now decidedly running away from that ecosystem after a year working in it.
Rails looks like an early take on MVC that was setup around fast project creation and encourages unmaintainable code - concerns are straight out retarded (a design pattern that encourages breaking encapsulation and makes dependencies impossible to trace) and fat models encourage coupling things that don't need to be coupled.
Then there's Ruby which in my opinion goes against Zen of Python which I'm a fan of - aliases all over the place for same thing, pointless name shortening and stuff to sacrifice readability, rails breaking conventions of standard library...
In the context of when it got popular I understand why it got the hype - replacing the PHP SQL in views garbage and Java 6 verbosity with XML configure everything, and the horrible ASP monstrosity.
These days I'd take Python as a more popular language, that I would say is better designed and has far better Windows support.
NPM/TypeScript and structural static typing is getting really good as well but is not as mature - but code sharing between client and server is a real thing.
Go emerged since the time of Rails hype.
.NET and JVM languages and frameworks got a lot better and scale far better with big code bases.
I don't really see where Rails is great compared to alternatives, it feels like legacy at this point. I'm not saying you can't write good maintainable apps in Rails either - I'm just saying other frameworks and languages will make it easier IMO.
Rails looks like an early take on MVC that was setup around fast project creation and encourages unmaintainable code - concerns are straight out retarded (a design pattern that encourages breaking encapsulation and makes dependencies impossible to trace) and fat models encourage coupling things that don't need to be coupled.
Then there's Ruby which in my opinion goes against Zen of Python which I'm a fan of - aliases all over the place for same thing, pointless name shortening and stuff to sacrifice readability, rails breaking conventions of standard library...
In the context of when it got popular I understand why it got the hype - replacing the PHP SQL in views garbage and Java 6 verbosity with XML configure everything, and the horrible ASP monstrosity.
These days I'd take Python as a more popular language, that I would say is better designed and has far better Windows support.
NPM/TypeScript and structural static typing is getting really good as well but is not as mature - but code sharing between client and server is a real thing.
Go emerged since the time of Rails hype.
.NET and JVM languages and frameworks got a lot better and scale far better with big code bases.
I don't really see where Rails is great compared to alternatives, it feels like legacy at this point. I'm not saying you can't write good maintainable apps in Rails either - I'm just saying other frameworks and languages will make it easier IMO.
> What's great about Rails ? Serious question from someone who landed into a mature Rails app and is now decidedly running away from that ecosystem after a year working in it.
Rails provides a mature system to quickly "get the job done" in a way that can be easily maintained and generally "just works". It's especially good if you want to have a small team manage an application. More details below.
> Rails looks like an early take on MVC that was setup around fast project creation and encourages unmaintainable code - concerns are straight out retarded (a design pattern that encourages breaking encapsulation and makes dependencies impossible to trace) and fat models encourage coupling things that don't need to be coupled.
That hasn't been my experience. Used well, Rails code tends to be relatively easy to follow. It's true that concerns can be easily overused. But that's true of many power tools; I use concerns sparingly, and then they work just fine.
> Then there's Ruby which in my opinion goes against Zen of Python which I'm a fan of - aliases all over the place for same thing, pointless name shortening and stuff to sacrifice readability, rails breaking conventions of standard library...
This is an odd argument. I know both Ruby and Python. No, Ruby doesn't follow Python conventions. That's because Ruby is not Python. The shortening can in some places increase readability; YMMV.
Rails does extend the standard library in a few ways, but if you're using Rails, that's already baked in & you never see it otherwise. So again, that seems irrelevant. If you're using Rails... then you're using Rails.
> These days I'd take Python as a more popular language, that I would say is better designed and has far better Windows support.
Python & Ruby are similar in many ways, it's hard to argue one is objectively far "better".
I've heard from others that Rails runs great on Windows 10, just use Windows Subsystem for Linux (WSL). I don't know how well Rails runs on native Windows. I know at one time that was important for many people. But for many people today, "run on native Windows" is irrelevant. The web application that I maintain that uses Rails runs on a Linux system, and while it would probably work on any Unix-like system, there's no interest in getting it to run on native Windows. Why do that? There's absolutely no justification for ever running that application on Windows, it would just cost more & crash more. If I want to run it on Windows for debugging, VirtualBox is great. Windows is disappearing completely from many server-side environments. Even Microsoft's own Azure runs Linux more than Windows: https://www.zdnet.com/article/microsoft-developer-reveals-li...
> NPM/TypeScript and structural static typing is getting really good as well but is not as mature - but code sharing between client and server is a real thing.
In many situations the sharing is so minimal as to be not worth it. I've only found a single function that would make sense on both client & server, and it was trivially implemented twice. That's not, by itself, a reason to force both sides to be the same language.
In many applications, "not as mature" means "don't use it". I don't have time to waste debugging someone else's framework, especially when I'm implementing a relatively straightforward CRUD application. I'll gladly use an immature system if it provides a vital capability unavailable elsewhere, but for simple CRUD applications that's absurd. I loathe fad-based engineering.
> I don't really see where Rails is great compared to alternatives, it feels like legacy at this point. I'm not saying you can't write good maintainable apps in Rails either - I'm just saying other frameworks and languages will make it easier IMO.
I think the main advantages of Rails are that it is:
- mature - a lot of work has been spent to make sure the "special cases" work, and a large amount of functionality is there for immediate use. Like anything it has bugs & missing functions, but in general, if it's a common need it's easily available, easily works, and rarely has bugs.
- integrated - while Rails is implemented as a set of libraries, they're conceived as a whole, so there's no effort to "make the parts work together" - they already work together.
- simplifies & speeds development by making a lot of convention decisions for you, via "convention over configuration". For example, database tables use snake_case with plural names; model class names use CamelCase and are singular. Yes, Rails automatically does the singular/plural conversion! The conventions eliminates arguing & deciding conventions, doing the work to configure it, and makes the code more regular, but it's more painful if you hate their conventions & want to override them all.
Of course, there are many worthy alternatives. Rails isn't the be-all (what is?). But Rails is still a fine choice for many applications today; GitHub and GitLab are both Rails applications. E.g.: https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to...
The biggest problem with Rails is that Ruby, like Python, is an extremely slow language. In most applications this is irrelevant; the correct way to speed things up is to cache things, and then it really isn't a problem. Rails comes with a very easy-to-use caching system. So folks, like GitLab, have found that they can identify just the hotspot & reimplement just that small part. The same happens with Python, by the way; if you want fast Python, you either call "Python" code that's actually written in C, or rewrite your hotspot Python code into C. There's no such thing as a free lunch.
In particular, Rails has a large set of conventions. If you're willing to accept those conventions, a lot is done automatically for you. If you will continuously fight the conventions, then it's going to be a pain to work with.
Rails provides a mature system to quickly "get the job done" in a way that can be easily maintained and generally "just works". It's especially good if you want to have a small team manage an application. More details below.
> Rails looks like an early take on MVC that was setup around fast project creation and encourages unmaintainable code - concerns are straight out retarded (a design pattern that encourages breaking encapsulation and makes dependencies impossible to trace) and fat models encourage coupling things that don't need to be coupled.
That hasn't been my experience. Used well, Rails code tends to be relatively easy to follow. It's true that concerns can be easily overused. But that's true of many power tools; I use concerns sparingly, and then they work just fine.
> Then there's Ruby which in my opinion goes against Zen of Python which I'm a fan of - aliases all over the place for same thing, pointless name shortening and stuff to sacrifice readability, rails breaking conventions of standard library...
This is an odd argument. I know both Ruby and Python. No, Ruby doesn't follow Python conventions. That's because Ruby is not Python. The shortening can in some places increase readability; YMMV.
Rails does extend the standard library in a few ways, but if you're using Rails, that's already baked in & you never see it otherwise. So again, that seems irrelevant. If you're using Rails... then you're using Rails.
> These days I'd take Python as a more popular language, that I would say is better designed and has far better Windows support.
Python & Ruby are similar in many ways, it's hard to argue one is objectively far "better".
I've heard from others that Rails runs great on Windows 10, just use Windows Subsystem for Linux (WSL). I don't know how well Rails runs on native Windows. I know at one time that was important for many people. But for many people today, "run on native Windows" is irrelevant. The web application that I maintain that uses Rails runs on a Linux system, and while it would probably work on any Unix-like system, there's no interest in getting it to run on native Windows. Why do that? There's absolutely no justification for ever running that application on Windows, it would just cost more & crash more. If I want to run it on Windows for debugging, VirtualBox is great. Windows is disappearing completely from many server-side environments. Even Microsoft's own Azure runs Linux more than Windows: https://www.zdnet.com/article/microsoft-developer-reveals-li...
> NPM/TypeScript and structural static typing is getting really good as well but is not as mature - but code sharing between client and server is a real thing.
In many situations the sharing is so minimal as to be not worth it. I've only found a single function that would make sense on both client & server, and it was trivially implemented twice. That's not, by itself, a reason to force both sides to be the same language.
In many applications, "not as mature" means "don't use it". I don't have time to waste debugging someone else's framework, especially when I'm implementing a relatively straightforward CRUD application. I'll gladly use an immature system if it provides a vital capability unavailable elsewhere, but for simple CRUD applications that's absurd. I loathe fad-based engineering.
> I don't really see where Rails is great compared to alternatives, it feels like legacy at this point. I'm not saying you can't write good maintainable apps in Rails either - I'm just saying other frameworks and languages will make it easier IMO.
I think the main advantages of Rails are that it is:
- mature - a lot of work has been spent to make sure the "special cases" work, and a large amount of functionality is there for immediate use. Like anything it has bugs & missing functions, but in general, if it's a common need it's easily available, easily works, and rarely has bugs.
- integrated - while Rails is implemented as a set of libraries, they're conceived as a whole, so there's no effort to "make the parts work together" - they already work together.
- simplifies & speeds development by making a lot of convention decisions for you, via "convention over configuration". For example, database tables use snake_case with plural names; model class names use CamelCase and are singular. Yes, Rails automatically does the singular/plural conversion! The conventions eliminates arguing & deciding conventions, doing the work to configure it, and makes the code more regular, but it's more painful if you hate their conventions & want to override them all.
Of course, there are many worthy alternatives. Rails isn't the be-all (what is?). But Rails is still a fine choice for many applications today; GitHub and GitLab are both Rails applications. E.g.: https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to...
The biggest problem with Rails is that Ruby, like Python, is an extremely slow language. In most applications this is irrelevant; the correct way to speed things up is to cache things, and then it really isn't a problem. Rails comes with a very easy-to-use caching system. So folks, like GitLab, have found that they can identify just the hotspot & reimplement just that small part. The same happens with Python, by the way; if you want fast Python, you either call "Python" code that's actually written in C, or rewrite your hotspot Python code into C. There's no such thing as a free lunch.
In particular, Rails has a large set of conventions. If you're willing to accept those conventions, a lot is done automatically for you. If you will continuously fight the conventions, then it's going to be a pain to work with.
>But for many people today, "run on native Windows" is irrelevant.
Developing RoR on Windows is still a pain in the bottom. WSL doesn't work well, WSL2 solves this, but isn't widely available. Not everyone could afford a Mac, and Linux machine just throw off majority of people new to programming. Considering 90%+ people uses Windows as their PC, I dont think it is irrelevant or small issues at all.
Developing RoR on Windows is still a pain in the bottom. WSL doesn't work well, WSL2 solves this, but isn't widely available. Not everyone could afford a Mac, and Linux machine just throw off majority of people new to programming. Considering 90%+ people uses Windows as their PC, I dont think it is irrelevant or small issues at all.
They might use a Windows PC as their client, but they can easily develop & deploy on something else. VirtualBox is free, as are Ubuntu & Fedora. Deploying Rails apps to systems like Heroku is worlds easier than a typical Windows deploy.
I don't think Linux throws off many people. For many people that is the future. I've written batch scripts & worked around Windows nonsense, and Linux is generally a better experience. Middle schoolers don't seem have problems using Raspberry Pis running Linux.
I don't think Linux throws off many people. For many people that is the future. I've written batch scripts & worked around Windows nonsense, and Linux is generally a better experience. Middle schoolers don't seem have problems using Raspberry Pis running Linux.
Rails 2.0 came out and hundreds of people were able to get their own little slice of what the web could offer internally.
I made roughly $140,000 one year just because someone wanted a build for something that made them close to $10,000,000 in profit without having to do custom Windows / Mac development.
It was the fastest method to get something online and only go better through Rails 3 through 5. Yes, the mature code bases are a pain because people try upgrading in place and don't want to rewrite anything so you have Rails 3 code in a Rails 5 codebase because 'it works'
I made roughly $140,000 one year just because someone wanted a build for something that made them close to $10,000,000 in profit without having to do custom Windows / Mac development.
It was the fastest method to get something online and only go better through Rails 3 through 5. Yes, the mature code bases are a pain because people try upgrading in place and don't want to rewrite anything so you have Rails 3 code in a Rails 5 codebase because 'it works'
I’ve recently come back to a big Rails project after doing React/TS for a while and I feel exactly the same way. I feel like we’ve learned a lot about what kinds of tools help make programmers productive and help them write correct code since the original Rails heyday and none of those lessons have been absorbed into Rails.
Rails is a reasonable choice if by installing Rails and some gems you’re already 90% of the way to your goal. Every line of code you write after that is another step down the road to unhappiness.
Rails is a reasonable choice if by installing Rails and some gems you’re already 90% of the way to your goal. Every line of code you write after that is another step down the road to unhappiness.
Regardless of one's views on Rails (not a fan, personally), I think we can all agree that there is plenty of room to criticize it without using ableist slurs.
[deleted]
I've been using Rails for the past 6 or 7 years. The biggest positive is that almost any feature you need is available as a gem. We're using for an early stage startup right now, and it's crazy how quickly we've been able to crank out features. I've looked at the JS and Python ecosystems, and haven't seen the same mature ecosystem.
I don't disagree with your point around large Rails apps becoming un-maintainable. I think as long as you have a few developers who have dealt with big Rails app and know what to avoid (concerns, fat models, etc), it's not too hard to write a maintainable app in Rails.
I don't disagree with your point around large Rails apps becoming un-maintainable. I think as long as you have a few developers who have dealt with big Rails app and know what to avoid (concerns, fat models, etc), it's not too hard to write a maintainable app in Rails.
> Zen of Python
You lost me there. If you're coming from a Pythonic point of view you are doing Ruby wrong. It will never compare on that standard because it's not Python and it doesn't need to. Python's pillars are opinions, just like Ruby's.
Ruby's got it quirks but I consider Python to be the wrong one. Ruby's my zen (and luckily it pays good). Python syntax is wordy and awkward to me, and "Pythonic" might be great for newbies but I find its principles slow me down.
To each their own.
You lost me there. If you're coming from a Pythonic point of view you are doing Ruby wrong. It will never compare on that standard because it's not Python and it doesn't need to. Python's pillars are opinions, just like Ruby's.
Ruby's got it quirks but I consider Python to be the wrong one. Ruby's my zen (and luckily it pays good). Python syntax is wordy and awkward to me, and "Pythonic" might be great for newbies but I find its principles slow me down.
To each their own.
I would identify as a .NET developer primarily as that's most of my professional background, but I've used Python a lot and while I also prefer the syntax, the main thing for me is design decisions. Ruby has "10 ways to do the same thing, some identical and aliases some subtly different" and the worst part is nobody tells you "this is the preferred way" so you end up with everyone using their own preferred aliases, syntax approaches, etc. This just decreases maintainability and increases cognitive load.
Stuff like Pythons "one right way to do things" is something I see in other good frameworks, I just identify it with Python in this comparison because it's closes to Ruby. Most of Zen of Python is good SW engineering practices and Ruby gladly goes against those - which, predictably, leads to messy code base I've witnessed.
Stuff like Pythons "one right way to do things" is something I see in other good frameworks, I just identify it with Python in this comparison because it's closes to Ruby. Most of Zen of Python is good SW engineering practices and Ruby gladly goes against those - which, predictably, leads to messy code base I've witnessed.
Again, they are different approaches. Ruby derives heavily from Larry Wall's Perl, where multiple ways to do things was a feature. (Wall is a linguist and Perl reflects this) The way you manage this with a team is enforcing a "house style". This is how we did it when I worked at a Perl shop, it's how we do it at the Ruby shop I work at now, and it's how other teams manage their language's sprawling featuresets.
Python is unique in that it is one of the few languages to really hold "only one way to do things" as a pillar. I think it makes writing code more brainless and leaves less room for nuance and artistry, plus it forces you into weird syntax corners that you have no other way of getting out of. The python syntax for so many things is such a mess, I still cannot recall off the top of my head what the correct way is to do different kinds of iterations over a dictionary without looking it up. (And whenever I had to I remember hating it)
I've worked professionally in Python and it's not a bad language. (In fact I use it or its derivatives in a lot of personal projects.) It just so happens that I don't understand why people hold it up as this paragon of virtue, when it's literally just another tool, with another tired opinion.
> and the worst part is nobody tells you "this is the preferred way"
That's on your manager/team lead. In a good shop they'll tell you the preferred way.
Python is unique in that it is one of the few languages to really hold "only one way to do things" as a pillar. I think it makes writing code more brainless and leaves less room for nuance and artistry, plus it forces you into weird syntax corners that you have no other way of getting out of. The python syntax for so many things is such a mess, I still cannot recall off the top of my head what the correct way is to do different kinds of iterations over a dictionary without looking it up. (And whenever I had to I remember hating it)
I've worked professionally in Python and it's not a bad language. (In fact I use it or its derivatives in a lot of personal projects.) It just so happens that I don't understand why people hold it up as this paragon of virtue, when it's literally just another tool, with another tired opinion.
> and the worst part is nobody tells you "this is the preferred way"
That's on your manager/team lead. In a good shop they'll tell you the preferred way.
As someone who started with Rails in 2010, rode the JS/Node hype train for 5 years, and now working with Rails again:
I was one of those guys that blamed Rails hard for encouraging bad code. But now that I'm a much better software developer, I've discovered Rails is actually fantastic for writing highly maintainable code. This is thanks to several things:
[A] Standard patterns. Yes, "The Rails Way" may seem to encourage bad code at first glance. But that's because it doesn't hand-hold you in your software design decisions. It's a solid foundation that 1. Takes care of the most common stuff so you don't have to reinvent the same crap over and over again, and 2. Leaves room to extend with an architecture (in plain old Ruby code) that fits your app's specific needs.
[B] Testing. A lot of people opt to use Rspec, but Rails's built-in testing is actually really great. My absolute favorite is their "Fast Integration Tests", which lets you write a few lines of code to test a big chunk of your app in a single test case.
[C] Ruby. Yes, it's actually a good language. But man, is it a double-edge sword. Used well, you can write some of the most simple, elegant, and maintainable code in your career, while only mildly pushing the boundaries of The Rails Way.
[D] Everything your app needs. So much stuff is built-in, and stays out of your way until you need it. And the fewer gems you need to install, the more maintainable your code tends to be.
Indeed, Rails is not legacy at all. Well, maybe if you think monoliths are legacy :) But it keeps up with the good trends. Version 5 introduced webpack, so now you can build your fancy JS frontends... without configuring webpack! Or be like me and only use JS as a last resort :)
To be honest, the only framework I can see doing better than Rails is Phoenix. At the moment it doesn't have everything Rails has to offer, but it does have (IMO) a superior foundation that has great potential to supersede. Until then though, I'll keep pumping out projects using the best tool on the market.
I was one of those guys that blamed Rails hard for encouraging bad code. But now that I'm a much better software developer, I've discovered Rails is actually fantastic for writing highly maintainable code. This is thanks to several things:
[A] Standard patterns. Yes, "The Rails Way" may seem to encourage bad code at first glance. But that's because it doesn't hand-hold you in your software design decisions. It's a solid foundation that 1. Takes care of the most common stuff so you don't have to reinvent the same crap over and over again, and 2. Leaves room to extend with an architecture (in plain old Ruby code) that fits your app's specific needs.
[B] Testing. A lot of people opt to use Rspec, but Rails's built-in testing is actually really great. My absolute favorite is their "Fast Integration Tests", which lets you write a few lines of code to test a big chunk of your app in a single test case.
[C] Ruby. Yes, it's actually a good language. But man, is it a double-edge sword. Used well, you can write some of the most simple, elegant, and maintainable code in your career, while only mildly pushing the boundaries of The Rails Way.
[D] Everything your app needs. So much stuff is built-in, and stays out of your way until you need it. And the fewer gems you need to install, the more maintainable your code tends to be.
Indeed, Rails is not legacy at all. Well, maybe if you think monoliths are legacy :) But it keeps up with the good trends. Version 5 introduced webpack, so now you can build your fancy JS frontends... without configuring webpack! Or be like me and only use JS as a last resort :)
To be honest, the only framework I can see doing better than Rails is Phoenix. At the moment it doesn't have everything Rails has to offer, but it does have (IMO) a superior foundation that has great potential to supersede. Until then though, I'll keep pumping out projects using the best tool on the market.
It's funny you say that, I've seen people pick a tech stack based on developer cost.
Maybe five years back I had a client in NYC who had a predominantly Java stack, but due to internal politics, wouldn't pay more than 135k for a senior developer when the market was wanting at least 25-30k more than that. Everyone they hired was pretty terrible.
At the time, Node developers were cheaper, so we split the application in two, wrapped all the old Java code with APIs, which was then able to be maintained by a small team while the front-end and any new features were all re-written with Node. It was architectural a better system--but the driving factor was really developer cost (the business hated to make any investments in technology).
The problem was that Node got popular and it got harder and harder to hire good Node developers at what the company wanted to pay. They had a big re-org and fired everyone I knew, so I don't know what they are doing now, but I often wonder if they just repeated the process with whatever up-and-coming language was more hip.
Maybe five years back I had a client in NYC who had a predominantly Java stack, but due to internal politics, wouldn't pay more than 135k for a senior developer when the market was wanting at least 25-30k more than that. Everyone they hired was pretty terrible.
At the time, Node developers were cheaper, so we split the application in two, wrapped all the old Java code with APIs, which was then able to be maintained by a small team while the front-end and any new features were all re-written with Node. It was architectural a better system--but the driving factor was really developer cost (the business hated to make any investments in technology).
The problem was that Node got popular and it got harder and harder to hire good Node developers at what the company wanted to pay. They had a big re-org and fired everyone I knew, so I don't know what they are doing now, but I often wonder if they just repeated the process with whatever up-and-coming language was more hip.
Cost is always one of the resources people have to manage...
Would you buy your laptop for 70% of you annual salary, even though it literally lasts for a few years and recoups it's cost fairly fast?
Would you buy your laptop for 70% of you annual salary, even though it literally lasts for a few years and recoups it's cost fairly fast?
> I've seen people pick a tech stack based on developer cost
You'd be surprised how often cost of developers comes into the overall conversation especially when you have tech sprawl and have to spend to get qualified people.
I've seen projects green-light just on the premise that removing legacy code will lead to a payroll reduction.
You'd be surprised how often cost of developers comes into the overall conversation especially when you have tech sprawl and have to spend to get qualified people.
I've seen projects green-light just on the premise that removing legacy code will lead to a payroll reduction.
I've been out of this world for awhile so it's super interesting to see that JS was paying more than Rails. It was the opposite back in my day :)
What are some of the other exciting things happening in the Rails world these days?
Here are a few interesting pieces in Rails specifically:
- https://github.com/hopsoft/stimulus_reflex is inspired by Phoenix LiveView and is getting a bit of traction around me
- https://github.com/hopsoft/cable_ready is a companion project
- https://github.com/discourse/message_bus by Sam Saffron from Discourse is a nice way to implement live updates too, quite easily (video demo at https://twitter.com/thibaut_barrere/status/12565974431075860...)
- https://lamby.custominktech.com is a Rails + AWS Lambda integration which is also gaining a bit of traction
I also see interesting stuff in Ruby more generally these days:
- my own https://www.kiba-etl.org (data processing framework) is growing nicely
- https://sidekiq.org is very solid and used in most Rails app I've seen
- https://github.com/contribsys/faktory allows interop-jobs (e.g. create from Ruby, consume from something else), which is also interesting
- https://github.com/oracle/truffleruby is making very good progress
Just a few cherry-picked links, but I definitely think there are some nice evolutions going on in the Ruby world (speaking as someone also using Elixir in production!).
- https://github.com/hopsoft/stimulus_reflex is inspired by Phoenix LiveView and is getting a bit of traction around me
- https://github.com/hopsoft/cable_ready is a companion project
- https://github.com/discourse/message_bus by Sam Saffron from Discourse is a nice way to implement live updates too, quite easily (video demo at https://twitter.com/thibaut_barrere/status/12565974431075860...)
- https://lamby.custominktech.com is a Rails + AWS Lambda integration which is also gaining a bit of traction
I also see interesting stuff in Ruby more generally these days:
- my own https://www.kiba-etl.org (data processing framework) is growing nicely
- https://sidekiq.org is very solid and used in most Rails app I've seen
- https://github.com/contribsys/faktory allows interop-jobs (e.g. create from Ruby, consume from something else), which is also interesting
- https://github.com/oracle/truffleruby is making very good progress
Just a few cherry-picked links, but I definitely think there are some nice evolutions going on in the Ruby world (speaking as someone also using Elixir in production!).
Also inspired by liveview and worth Checking out.
https://github.com/unabridged/motion
https://github.com/unabridged/motion
In addition to that Hey.com[1] was recently released which is a new app built by Basecamp. Where Ruby on Rails originated.
This new app comes with many new technologies. Some of which are already extracted into Rails (ActionText and ActionMailer), and some that will be (completely new approach to frontend with an all new Turbolinks).
At the same time Stimulus is getting mature.
All the excited around this, is motivating people to start creating helpful Rails resources[2], which in turn leads to more excitement.
[1] https://hey.com
[2] https://twitter.com/marckohlbrugge/status/127174984488606105...
This new app comes with many new technologies. Some of which are already extracted into Rails (ActionText and ActionMailer), and some that will be (completely new approach to frontend with an all new Turbolinks).
At the same time Stimulus is getting mature.
All the excited around this, is motivating people to start creating helpful Rails resources[2], which in turn leads to more excitement.
[1] https://hey.com
[2] https://twitter.com/marckohlbrugge/status/127174984488606105...
They seem to be recreating turbolinks ever couple of years heh.
Yeah this was my thought too: if turbolinks needed to be recreated for Hey, have they been selling a bill of goods with the arguments that you don't need these newfangled front end frameworks because turbolinks is already plenty good enough?
There’s a new release about to drop which is Rails 6.1 that covers some interesting developments.
GitHub took all of their code associated with running “web scale” applications that need to talk to multiple databases at the same time and put that into a dead simple native Rails framework.
Basecamp have spent the last couple of years (I think?) working in secret on what they claim is an entirely new way of doing all things front end. There isn’t a LOT of details on that at the moment and it’s kind of built up as a big surprise. But everyone who has had a behind the scenes look seems unusually excited.
There’s a kind of interesting web based interface for describing and developing all kinds of boilerplate parts of your application that could really speed up development time https://github.com/rails/rails/pull/35489
Some cool security stuff like this https://twitter.com/dhh/status/1268236728134889475?s=21 and promoting things like WebAuthn to be first class citizens.
But basically at this stage you have 3 major contributors of Shopify, Basecamp / Hey and Github extracting huge parts of their internal systems and rewriting them as mini Rails frameworks with famously simple APIs for people to use.
That’s obviously in addition to everyone / everything else. I think the future looks bright.
GitHub took all of their code associated with running “web scale” applications that need to talk to multiple databases at the same time and put that into a dead simple native Rails framework.
Basecamp have spent the last couple of years (I think?) working in secret on what they claim is an entirely new way of doing all things front end. There isn’t a LOT of details on that at the moment and it’s kind of built up as a big surprise. But everyone who has had a behind the scenes look seems unusually excited.
There’s a kind of interesting web based interface for describing and developing all kinds of boilerplate parts of your application that could really speed up development time https://github.com/rails/rails/pull/35489
Some cool security stuff like this https://twitter.com/dhh/status/1268236728134889475?s=21 and promoting things like WebAuthn to be first class citizens.
But basically at this stage you have 3 major contributors of Shopify, Basecamp / Hey and Github extracting huge parts of their internal systems and rewriting them as mini Rails frameworks with famously simple APIs for people to use.
That’s obviously in addition to everyone / everything else. I think the future looks bright.
> an entirely new way of doing all things front end.
Are you referring to ViewComponent: https://github.com/github/view_component? As someone who's happily been continuing to make Rails apps for the past few years (and avoiding Angular like the plague after being burned badly with it), this seems really intriguing, but I'm waiting on the 6.1 release before I try it.
Are you referring to ViewComponent: https://github.com/github/view_component? As someone who's happily been continuing to make Rails apps for the past few years (and avoiding Angular like the plague after being burned badly with it), this seems really intriguing, but I'm waiting on the 6.1 release before I try it.
Nah this is entirely seperate from that although I think View Component (also from GitHub in case anyone is wondering) is super exciting.
They are basically doing a major rewrite of Turbolinks and Stimulus I think. If you go to app.hey.com and jump into dev tools you can kind of start to piece things together somewhat since they have made all of the source maps available. But I’m yet to see too many people really figure out what is happening and I’m excited for the official announcement.
I think the idea that Rails has somewhat lagged behind the rest of the front end world is in many ways an overblown talking point but is at the same time in no way an unfair criticism.
I’ve heard the core team make several references for the last year that this new approach is basically to front end what Rails was to backend when it launched in terms of developer experience while keeping 80-90% of the performance benefits of SPAs but with 0% of the drama. Or at least this is how I understand them to be promoting it.
They are basically doing a major rewrite of Turbolinks and Stimulus I think. If you go to app.hey.com and jump into dev tools you can kind of start to piece things together somewhat since they have made all of the source maps available. But I’m yet to see too many people really figure out what is happening and I’m excited for the official announcement.
I think the idea that Rails has somewhat lagged behind the rest of the front end world is in many ways an overblown talking point but is at the same time in no way an unfair criticism.
I’ve heard the core team make several references for the last year that this new approach is basically to front end what Rails was to backend when it launched in terms of developer experience while keeping 80-90% of the performance benefits of SPAs but with 0% of the drama. Or at least this is how I understand them to be promoting it.
I bet it is similar to Phoenix Live View: https://dockyard.com/blog/2018/12/12/phoenix-liveview-intera...
Which would be a big deal, indeed.
Which would be a big deal, indeed.
We built a gem on top of ViewComponents that works similarly to Phoenix Live View:
https://github.com/unabridged/motion
https://github.com/unabridged/motion
Thanks for asking - was curious the same. Always loved Rails, but then got into the "JS" toolset - which has been a cluster of extra required work.
I am one of those "refugees", although not related to JS specifically. I think the last nudge way Hey releasing which made me consider RoR after all these years.
The motivation on my part is simple - I am getting really tired of microservices everywhere.
I understand why they might make sense in large organizations but the endless splitting of stuff that works (or could work if the energy was invested in fixing existing code) to the point where you have 3-5 services per developer is tiring.
Anyways, ranting aside, I have been learning Rails on the weekends and it's fun, even if there aren't that many job opportunities. If I later end up working in an ecosystem where the majestic monolith is something to aspire to, even better.
The motivation on my part is simple - I am getting really tired of microservices everywhere.
I understand why they might make sense in large organizations but the endless splitting of stuff that works (or could work if the energy was invested in fixing existing code) to the point where you have 3-5 services per developer is tiring.
Anyways, ranting aside, I have been learning Rails on the weekends and it's fun, even if there aren't that many job opportunities. If I later end up working in an ecosystem where the majestic monolith is something to aspire to, even better.
I'm excited about ActiveJob because, finally, people will be able to build background jobs on things that aren't Redis. No hate on Redis - I'm a huge fan of it - but sometimes you have stronger consistency reqs that Redis does not have, and stronger availability reqs that SQS or PG don't have (aka multi-region jobs).
If the concurrency in ActiveJob become as good or better than Sidekiq, we can really get a lot of interesting stuff out there.
If the concurrency in ActiveJob become as good or better than Sidekiq, we can really get a lot of interesting stuff out there.
I used Que [0] in a number of projects in the last few years that had stronger consistency demands. It can plug into ActiveJob. Those projects are still humming along without trouble today.
[0]: https://github.com/que-rb/que
[0]: https://github.com/que-rb/que
As a fellow Que user, I'm happy to see this line in the README:
> For example, GoodJob is currently ~600 lines of code, whereas Que is ~1,200 lines,
I've tried to dig into the Que internals before (looking into deadlocks) and they were gnarly; skimming through the GoodJob internals, it looks a lot more like what I'd hope for.
> For example, GoodJob is currently ~600 lines of code, whereas Que is ~1,200 lines,
I've tried to dig into the Que internals before (looking into deadlocks) and they were gnarly; skimming through the GoodJob internals, it looks a lot more like what I'd hope for.
Unfortunately its no longer maintained and his 2.0 branch is completely broken for me. I've been very stressed trying to find a que replacement, as almost all of my needs for job need ACID guarantees, and Sidekiq/Redis "best effort" is nowhere near what I need. I also refuse to pay $xxxx for Sidekiq pro just to get those guarantees.
Good job looks pretty awesome, I'll have to give it a shot at some point.
Good job looks pretty awesome, I'll have to give it a shot at some point.
Sk pro is really not that much?
Que also has a Golang port [1], which can be a tool to get towards The Citadel [2] for performance or dependency-heavy jobs without going full SOA
[1] https://github.com/bgentry/que-go
[2] https://m.signalvnoise.com/the-majestic-monolith-can-become-...
[1] https://github.com/bgentry/que-go
[2] https://m.signalvnoise.com/the-majestic-monolith-can-become-...
It would be nice to have one less dependency in the tech stack but honestly have no complaints with Sidekiq so far.
I'm quite new to Ruby/Rails though — would be interesting to hear from others how they think it stacks up (unfortunately the author doesn't compare in the blog post)
I'm quite new to Ruby/Rails though — would be interesting to hear from others how they think it stacks up (unfortunately the author doesn't compare in the blog post)
Another small thing to consider: Adding redis to your stack adds a new failure mode. A service I built a while ago was running on Heroku where Redis add-ons aren't cheap, and our Redis would always running against memory limits, which would cause our entire service to fail.
Meanwhile, 250GB Postgres server has been churning along no problem.
Also, conceptually, architecture where state = DB, and business logic = web server is generally easier to reason about during service migrations and such.
If you don't use ActionCable which I think is the other core feature that relies on Redis, being able to remove it would've saved me a lot of early morning firefighting.
Meanwhile, 250GB Postgres server has been churning along no problem.
Also, conceptually, architecture where state = DB, and business logic = web server is generally easier to reason about during service migrations and such.
If you don't use ActionCable which I think is the other core feature that relies on Redis, being able to remove it would've saved me a lot of early morning firefighting.
This is exactly why I stuck with DelayedJob after all of these years. The community seemed to flock to Resque or Sidekiq, but I always felt my needs were met just fine with Delayed Job. I never was interested in adding more unnecessary complexity to our dev and staging environments for a trade-off I didn't need (there wouldn't be a significant performance increase for my needs).
I've seen this pattern happen with a lot of Ruby projects: there's a popular Gem that people use that grows over time, then someone writes a blog post about why that package isn't suitable for their needs due to a design trade-off (often introducing a new Gem that is advertised as superior to the previous one) and then suddenly the old Gem stagnates, causing the maintainer to lose interest and updates stop pacing Rails versions. Sometimes the old Gem gets deprecated altogether and maintainers stick a big DO NOT USE sign at the top of the github README, all but guaranteeing there is no further community traction or organization for the project.
Meanwhile those of us using the old project chug along with a forked Gem that we cobble a bunch of patches into to meet our needs because there's no longer a centralized place to contribute to anymore. In some ways it's the double-edged sword of OSS. Maintainers aren't obligated to stay as maintainers of course, but it does get frustrating how quickly the community is willing to drop support of things that a lot people are using in production systems.
It definitely makes me reconsider the "don't reinvent the wheel" advice that is so adamantly thought of as a common sense convention in our craft.
I've seen this pattern happen with a lot of Ruby projects: there's a popular Gem that people use that grows over time, then someone writes a blog post about why that package isn't suitable for their needs due to a design trade-off (often introducing a new Gem that is advertised as superior to the previous one) and then suddenly the old Gem stagnates, causing the maintainer to lose interest and updates stop pacing Rails versions. Sometimes the old Gem gets deprecated altogether and maintainers stick a big DO NOT USE sign at the top of the github README, all but guaranteeing there is no further community traction or organization for the project.
Meanwhile those of us using the old project chug along with a forked Gem that we cobble a bunch of patches into to meet our needs because there's no longer a centralized place to contribute to anymore. In some ways it's the double-edged sword of OSS. Maintainers aren't obligated to stay as maintainers of course, but it does get frustrating how quickly the community is willing to drop support of things that a lot people are using in production systems.
It definitely makes me reconsider the "don't reinvent the wheel" advice that is so adamantly thought of as a common sense convention in our craft.
That’s exactly why I implemented my own queue system. If it’s something that will only take a few hours to build, and I’ll need to maintain it long term, I’ll always favor rolling it myself.
There are very few “feature gems” that I’ve used in the past 15 years doing Rails development that I’ve been happy I used a year later. Development gems on the other hand like byebug are usually fine.
There are very few “feature gems” that I’ve used in the past 15 years doing Rails development that I’ve been happy I used a year later. Development gems on the other hand like byebug are usually fine.
Maybe repo sponsorships/bounties would help. There could be a basic maintenance bounty that could be chipped into, it could build up over time and then occasionally any of the repo contributors could jump in and tackle a bunch of PRs and issues all at once and pick up the pool of cash.
Yeah I mean if your project doesn’t get huge amount of usage, DJ is fine and quite simple.
Unfortunately at any moderate scale or above a RDMS backed jobs table doesn’t cut it
>Unfortunately at any moderate scale
The problem here is the definition of moderate scale and huge amount of usage.
We handle tens of millions of events per year using our primary RDMS as a queue with absolutely no problems at all. The vast majority of projects for most companies aren't going to get bigger than that.
If you have millions of users sending out tens or hundreds of messages per day, sure, that's a huge amount of usage.
The problem here is the definition of moderate scale and huge amount of usage.
We handle tens of millions of events per year using our primary RDMS as a queue with absolutely no problems at all. The vast majority of projects for most companies aren't going to get bigger than that.
If you have millions of users sending out tens or hundreds of messages per day, sure, that's a huge amount of usage.
[deleted]
That is why sometimes these Gems should be folded into Rails itself, assume it fits the need.
Agreed, Sidekiq is great and I don't imagine anyone switching away from it for existing projects.
But I can see GoodJob being better for spinning up new projects, as there are fewer moving parts to worry about, and it's easy to upgrade to Sidekiq, etc. when you need to (thanks to ActiveJob's standard abstraction).
But I can see GoodJob being better for spinning up new projects, as there are fewer moving parts to worry about, and it's easy to upgrade to Sidekiq, etc. when you need to (thanks to ActiveJob's standard abstraction).
Yea I agree, a comparison to sidekiq would be helpful in evaluating and was what I was expecting to be covered on the page.
Sidekiq is one of the best pieces of software I have ever used.
The standard advice used to be that neither Redis or Postgres are very good for using as a queue.
The creator and former maintainer of Redis was up until a few years ago discouraging its use as a queue, I think mainly because of its lack of durability and high availability at the time. He built a prototype Disque[0] to address the issues but it never became production ready. The other downside is that Redis is in-memory which means the queues have less capacity/are more expensive for the same capacity than an on-disk solution, but as memory gets cheaper over the years this becomes less and less of an issue. The upside is the throughput of Redis is very high.
I have personally worked on rails apps using redis-based queues like resque (and to a lesser extent sidekiq), and actually haven't run into any redis crashes or downtime in years of runtime, redis is very solid in general. You can also snapshot the redis instance periodically, to limit the number of jobs you would lose if it did crash.
In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects.
The issue is that eventually your queues will back up, whether it's due to a bug, surge of traffic, or just complex interaction of behavior in your app that cascades a ton of jobs at once when you run a backfill or something. When your app starts to get overloaded it's pretty trivial to increase the number of web instances running, so your bottleneck in these situations is going to be the db performance. As your queues get backed up, your queue workers are running at full speed processing jobs nonstop, which puts strain on your DB. Additionally, the act of enqueueing and dequeuing a job itself also puts strain on the db, so you can easily get into an unstable situation where each job that gets added to the queue makes every other job take longer.
If you allocate a separate DB instance that is only running your queue, that is much safer. Still, a DB like postgres is not great at doing constant writes and deletes, it creates additional auto-vacuum pressure for instance. But this will manifest as just getting worse throughput on the same hardware than you could get from a dedicated queue like rabbit mq, so if you're not at large scale it's a fine option.
Edit: And one other thing to add, for a lot of web apps the scope of what is needed from a queue these days is a lot less now than it was in the past. It used to be, and in large enterprise systems it often still is, the case that when people talked about a message queue they wanted something to facilitate passing messages between many completely separate apps. Now most apps just use a rest api for that (or perhaps protobufs or graphql or something but still over http). So I think historically an additional reason against using a simple datastore as a queue was that it didn't have enough features so you'd end up re-inventing the wheel with things like brokers, fan out and broadcast patterns, at-most-once vs at-least-once semantics, etc. But here I'm just considering the very limited usecase of a sidekiq-like queue, for processing jobs in the background for a single web app.
tl;dr: Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong.
[0] https://github.com/antirez/disque
The creator and former maintainer of Redis was up until a few years ago discouraging its use as a queue, I think mainly because of its lack of durability and high availability at the time. He built a prototype Disque[0] to address the issues but it never became production ready. The other downside is that Redis is in-memory which means the queues have less capacity/are more expensive for the same capacity than an on-disk solution, but as memory gets cheaper over the years this becomes less and less of an issue. The upside is the throughput of Redis is very high.
I have personally worked on rails apps using redis-based queues like resque (and to a lesser extent sidekiq), and actually haven't run into any redis crashes or downtime in years of runtime, redis is very solid in general. You can also snapshot the redis instance periodically, to limit the number of jobs you would lose if it did crash.
In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects.
The issue is that eventually your queues will back up, whether it's due to a bug, surge of traffic, or just complex interaction of behavior in your app that cascades a ton of jobs at once when you run a backfill or something. When your app starts to get overloaded it's pretty trivial to increase the number of web instances running, so your bottleneck in these situations is going to be the db performance. As your queues get backed up, your queue workers are running at full speed processing jobs nonstop, which puts strain on your DB. Additionally, the act of enqueueing and dequeuing a job itself also puts strain on the db, so you can easily get into an unstable situation where each job that gets added to the queue makes every other job take longer.
If you allocate a separate DB instance that is only running your queue, that is much safer. Still, a DB like postgres is not great at doing constant writes and deletes, it creates additional auto-vacuum pressure for instance. But this will manifest as just getting worse throughput on the same hardware than you could get from a dedicated queue like rabbit mq, so if you're not at large scale it's a fine option.
Edit: And one other thing to add, for a lot of web apps the scope of what is needed from a queue these days is a lot less now than it was in the past. It used to be, and in large enterprise systems it often still is, the case that when people talked about a message queue they wanted something to facilitate passing messages between many completely separate apps. Now most apps just use a rest api for that (or perhaps protobufs or graphql or something but still over http). So I think historically an additional reason against using a simple datastore as a queue was that it didn't have enough features so you'd end up re-inventing the wheel with things like brokers, fan out and broadcast patterns, at-most-once vs at-least-once semantics, etc. But here I'm just considering the very limited usecase of a sidekiq-like queue, for processing jobs in the background for a single web app.
tl;dr: Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong.
[0] https://github.com/antirez/disque
> Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong.
That is a broad generalization that assumes most applications are operating at mega scale. The benefits of simplified dependencies (a single database instance), transactional guarantees (a single database instance) and persistence (not using Redis) far outweigh the eventual possibility that the queue will place too large a load on your database.
As the author of Oban[0] (an PG backed persistent queue in Elixir) I'm definitely biased. However, the level of adoption in the Elixir community seems to signal that a lot of companies favor simplicity and safety over a possible scale issue down the road. The primary application I work on processes ~500k-1m jobs a day and the queue overhead is virtually invisible.
[0] https://github.com/sorentwo/oban
That is a broad generalization that assumes most applications are operating at mega scale. The benefits of simplified dependencies (a single database instance), transactional guarantees (a single database instance) and persistence (not using Redis) far outweigh the eventual possibility that the queue will place too large a load on your database.
As the author of Oban[0] (an PG backed persistent queue in Elixir) I'm definitely biased. However, the level of adoption in the Elixir community seems to signal that a lot of companies favor simplicity and safety over a possible scale issue down the road. The primary application I work on processes ~500k-1m jobs a day and the queue overhead is virtually invisible.
[0] https://github.com/sorentwo/oban
You actually bring up another point that in my mind is yet another argument against using your primary DB as a queue, so I'm sure I am also biased from being burned in the past :)
But in a past company I worked at, the company started out thinking sure just throw the queue in the primary db for simplicity. Eventually our slow query logs and db performance monitoring tools were showing that ~40% of the db load was due to the queue inserts and queries. It may be that it was doing something incredibly inefficient and unnecessary in the particular library we were using but we did look into it pretty thoroughly, this was a few years ago though so I don't recall all of the details. And that was at normal operation, then we ran into an issue that basically brought our site down when queues backed up.
At that point it was definitely time to split out the queue, and when we did it we realized that we had implicitly been depending on transactional consistency between the queue and the app data in a few places, which was then extra work to track down and fix these types of issues. This is IMO a code smell as well in general - your data and your infrastructure should ideally not be so tightly coupled.
Managed databases are so easy to set up these days, I would still definitely recommend a separate instance for the queue vs the primary db from day 1 in any new app I build. If you do want to combine them to save money on infra, use a separate logical db and separate connection pool and everything so that it's easy to split out in the future.
But in a past company I worked at, the company started out thinking sure just throw the queue in the primary db for simplicity. Eventually our slow query logs and db performance monitoring tools were showing that ~40% of the db load was due to the queue inserts and queries. It may be that it was doing something incredibly inefficient and unnecessary in the particular library we were using but we did look into it pretty thoroughly, this was a few years ago though so I don't recall all of the details. And that was at normal operation, then we ran into an issue that basically brought our site down when queues backed up.
At that point it was definitely time to split out the queue, and when we did it we realized that we had implicitly been depending on transactional consistency between the queue and the app data in a few places, which was then extra work to track down and fix these types of issues. This is IMO a code smell as well in general - your data and your infrastructure should ideally not be so tightly coupled.
Managed databases are so easy to set up these days, I would still definitely recommend a separate instance for the queue vs the primary db from day 1 in any new app I build. If you do want to combine them to save money on infra, use a separate logical db and separate connection pool and everything so that it's easy to split out in the future.
Agree that separate is more scalable. Still it also distributes the state, and that can be surprising if not planned for.
There's also another benefit around using of as a queue, you can just publish your messages from within the same transaction you are using on the request, and that's nice, all or nothing..
I will dig on your elixir queue, just build my own some days ago (mostly for fun) but also for solving some limitations in rabbit (mostly time based scheduling at short periods of time.. and control the throughput (to solve some rate limits).
Mine is here, https://github.com/vinissimus/jobs
The main feature is that it's built with pl/pgsql and allows to integrate so well with the rest of server backend (publishing jobs from triggers... ) Also listening on results with pg_notify
I will dig on your elixir queue, just build my own some days ago (mostly for fun) but also for solving some limitations in rabbit (mostly time based scheduling at short periods of time.. and control the throughput (to solve some rate limits).
Mine is here, https://github.com/vinissimus/jobs
The main feature is that it's built with pl/pgsql and allows to integrate so well with the rest of server backend (publishing jobs from triggers... ) Also listening on results with pg_notify
That completely depends on the amount of items you plan on adding to the queue. I’ve used the primary DB as a queue often over the last 10 years for things like sending notifications.
Sure there’s a point where this doesn’t make sense, but for the majority of cases there are ways to mitigate the positive reinforcement loop you’re taking about.
The easiest is to limit the number of workers to some number that won’t impact DB performance if they are running full tilt. You can even use an enum on existing records to determine the background job status if you have few enough rows (we do this for a table with a few hundred thousand job applicants).
I’ve found that in most cases we want a record that the background job was performed, so we were often updating the database anyway when a job was complete.
Sure if you are firing off so many events that PG can’t keep up with writing them, then PG isn’t a good option.
Sure there’s a point where this doesn’t make sense, but for the majority of cases there are ways to mitigate the positive reinforcement loop you’re taking about.
The easiest is to limit the number of workers to some number that won’t impact DB performance if they are running full tilt. You can even use an enum on existing records to determine the background job status if you have few enough rows (we do this for a table with a few hundred thousand job applicants).
I’ve found that in most cases we want a record that the background job was performed, so we were often updating the database anyway when a job was complete.
Sure if you are firing off so many events that PG can’t keep up with writing them, then PG isn’t a good option.
> In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects.
Postgres is fine as a queue for medium to large sized projects as it depends completely on your workload and what sort of performance characteristics you need. Thinking about it in terms of project size is the wrong way to evaluate it.
Postgres is fine as a queue for medium to large sized projects as it depends completely on your workload and what sort of performance characteristics you need. Thinking about it in terms of project size is the wrong way to evaluate it.
I have experienced this lifecycle many times:
1. Ah, we’re using the database as a queue.
2. Database is running hot, I wonder where all the load is coming from?
3. Oh, it’s all the queue tables.
It’s not even a scale/load issue. Even at smaller scales you’re introducing some major lock contention.
1. Ah, we’re using the database as a queue.
2. Database is running hot, I wonder where all the load is coming from?
3. Oh, it’s all the queue tables.
It’s not even a scale/load issue. Even at smaller scales you’re introducing some major lock contention.
If your database based queue is introducing lock contention then it is doing things wrong. Since Postgres 9.6 there is `FOR UPDATE SKIP LOCKED` which completely eliminates the issue.
Ah, cool! It’s fully possible I’ve been doing it wrong, or seen it done wrong, or at least that my experience is colored by older and less capable databases.
To make it scale well, two things are required:
1. TRUNCATE if cleanup is needed (no DELETEs) + partitioning.
2. SELECT .. FOR UPDATE SKIP LOCKED or advisory locks.
It's worth learning from Skype's PgQ developed in 2000-s.
Update: found the work with advisory locks in the code, great.
1. TRUNCATE if cleanup is needed (no DELETEs) + partitioning.
2. SELECT .. FOR UPDATE SKIP LOCKED or advisory locks.
It's worth learning from Skype's PgQ developed in 2000-s.
Update: found the work with advisory locks in the code, great.
Also see: https://news.ycombinator.com/item?id=9576864
Postgres Job Queues and Failure by MVCC
Despite my username, I haven't written any serious Ruby is like 5+ years; that is to say, I'm fairly removed from a lot of the changes in the Rails world.
Now, pardon me for my ignorance, but... isn't this just like Resque back in the day? Or like doesn't ActiveJob just support any ol' backend outta the box? What's the real use case for this? I'm genuinely curious not trying to be rude or diminish the work.
Now, pardon me for my ignorance, but... isn't this just like Resque back in the day? Or like doesn't ActiveJob just support any ol' backend outta the box? What's the real use case for this? I'm genuinely curious not trying to be rude or diminish the work.
GoodJob author here.
The core reason I wrote it was to see how lean/simple I could write a database-backed ActiveJob adapter today. And hope
In practice any adapter should _just work_, but also all the other adapters pre-date ActiveJob (Rails 4.2) and Concurrent::Ruby (adopted in Rails 5.0 I think). The assumption is that building GoodJob on top of what already exists in Rails today, it can be performant (enough) and simple (easy to understand, maintain, keep compatible with new versions of Rails, etc). That's not a big claim, but maybe it's compelling.
The core reason I wrote it was to see how lean/simple I could write a database-backed ActiveJob adapter today. And hope
In practice any adapter should _just work_, but also all the other adapters pre-date ActiveJob (Rails 4.2) and Concurrent::Ruby (adopted in Rails 5.0 I think). The assumption is that building GoodJob on top of what already exists in Rails today, it can be performant (enough) and simple (easy to understand, maintain, keep compatible with new versions of Rails, etc). That's not a big claim, but maybe it's compelling.
Awesome! Thank you so much for the reasoning behind it. I haven't had a chance to look at Concurrent::Ruby but that _sounds_ like a pretty good reason all by itself!
As a big plus for a lot of folks who don't need absurd throughput, this makes for one less dependency by using the same db you already have. I'd be sold given the right use-case, thanks for sharing this, mate and _good job!_ ;)
As a big plus for a lot of folks who don't need absurd throughput, this makes for one less dependency by using the same db you already have. I'd be sold given the right use-case, thanks for sharing this, mate and _good job!_ ;)
Yay!
You can implement background processing with less operational complexity. This allows you to leverage activejob to get something up and running quickly and easily.
Granted, if you start having higher throughput, you want to move on from using postgres as your message bus.
Granted, if you start having higher throughput, you want to move on from using postgres as your message bus.
I didn't look into this one yet besides the home page, but I always thought Postgres would make a killer backend for ActiveJob, leveraging advisory locks for synchronizing jobs and such. Resque is Redis backed, but if you don't want to add that as a dependency and already have Postgres setup, then this would be a great fit.
Because of the MVCC implementation (needing VACUUM) it's not ideal without tuning.
I believe Resque was fork based (at least initially, it may have changed, I haven't looked at it in a while!), while GoodJob appears to be multithreaded (like Sidekiq), so this is quite different in terms of performance & throughput.
For a very nice library in the Elixir/Phoenix ecosystem have a look at https://github.com/sorentwo/oban
I think one of my biggest annoyances, from a development perspective, with Sidekiq is that it typically moves you in the direction of running foreman or something like that, which makes using the pry a pain.
One thing I really like about Phoenix development is that I don't have to change my flow to use a REPL while still being able to run background jobs.
One thing I really like about Phoenix development is that I don't have to change my flow to use a REPL while still being able to run background jobs.
This looks like a great library on the surface. I would very much like to have a simpler alternative to Sidekiq for projects which doesn't require that level of complexity. Also, not having to add Redis just to run background processing makes life so much easier. I would like to thank the author of the library for all their efforts.
I was just assigned to a project based on zend 2.x php framework (2015 version) and angularjs(version1) (not angular.io). Just looking at the code, I just want to scream, because I was told to make changes and add features. I could re-implement everything in Rails in about a week's time but manager's opinions just drown me.
This is the exact sort of thing I love about the Rails community. Always innovating to keep things simple.
In this post, I noticed comparison with other job libraries, but not Sidekiq, the most popular one. Why?
I think they specifically wanted to compare themselves to queue systems that use the database as the storage for queued jobs.
GoodJob author here. The reason is that I don't have any production experience with Sidekiq.
My target right now is someone, like myself, who would typically spin up a new project with delayed_job or que, and provide a better alternative with GoodJob.
Maybe in the future I'll do some more research on understanding why someone might switch from a Sidekiq/Redis to not-Redis.
My target right now is someone, like myself, who would typically spin up a new project with delayed_job or que, and provide a better alternative with GoodJob.
Maybe in the future I'll do some more research on understanding why someone might switch from a Sidekiq/Redis to not-Redis.
Keeping your jobs in Redis (like Sidekiq does) seems to be a good feature to me. I’m curious how GoodJob compares when it comes to UI and queue management.
This looks great. I'd love to remove Redis from my tech stack on applications with low throughput background jobs. Sadly, it doesn't appear to have a UI.
Resque and Sidekiq are both good enough solutions, but what makes them my defacto choice is the robust web UIs that comes with them.
Resque and Sidekiq are both good enough solutions, but what makes them my defacto choice is the robust web UIs that comes with them.
This looks very promising! I'm currently using active job, so I will need to take a look at this. Thanks for letting me know.
This is awesome, thanks for sharing! Sidekiq works well for me but it’s good to have an option that doesn’t require Redis.
Hm. Got falsely excited. No listen/notify etc? This really is just delayed_job
GoodJob author here. I'm curious why you're specifically looking for listen/notify? I'm trying to prioritize that in the backlog.
Hi, sorry to hijack this thread, but since you mentioned you're the author, I just wanted to ask you something. Do you have any plans to use a different locking mechanism other than the advisory lock? I'm asking because eventually every postgres-backed app that reaches a certain scale, requires a connection pooler (pgbouncer being the de-facto standard nowadays). We usually run pgbouncer in transaction mode, which doesn't support advisory locks. Any thoughts?
That was mentioned in another thread. I have two thoughts about it:
- Going off of my own experience, I've never used PgBouncer and I've worked on some moderately sized applications (1s of millions of users). So I could say "not in scope for the target use-case".
- It would be possible to add an argument option to use a transaction-based Advisory Lock instead. I can imagine that would have downsides (long-running transactions, yikes).
I'm imagining that the lifecycle of GoodJob is that people starting new projects might choose it, and then the feature set would grow with their needs.
- Going off of my own experience, I've never used PgBouncer and I've worked on some moderately sized applications (1s of millions of users). So I could say "not in scope for the target use-case".
- It would be possible to add an argument option to use a transaction-based Advisory Lock instead. I can imagine that would have downsides (long-running transactions, yikes).
I'm imagining that the lifecycle of GoodJob is that people starting new projects might choose it, and then the feature set would grow with their needs.
Would this work with pgbouncer? I'm wondering because of the advisory locks.
GoodJob author here. Excellent observation.
It should work if PgBouncer is using session-level pooling, but not transaction-level pooling. I made a ticket to document this: https://github.com/bensheldon/good_job/issues/52
I don't have much experience with PgBouncer. What's the scale/need when a team would use PgBouncer? That would help me prioritize supporting it better.
It should work if PgBouncer is using session-level pooling, but not transaction-level pooling. I made a ticket to document this: https://github.com/bensheldon/good_job/issues/52
I don't have much experience with PgBouncer. What's the scale/need when a team would use PgBouncer? That would help me prioritize supporting it better.
The main reason I can think of would be when you have hundreds of workers accessing a DB but the DB only has a small connection limit, say 200 or something. You can have the workers access the connection through pgBouncer to bring that up into the thousands.
Thanks! That's really helpful context.
i'm not sure why i should use this instead of delayed job for instance
If you're using delayedjob, I strongly suggest you check out the Performance table in Sidekiq's readme. 465 seconds for DelayedJob against 14 seconds for Sidekiq for the same workload.
It's 7100 jobs/sec x 215 jobs/sec.
I wonder how GoodJob will compare.
https://github.com/mperham/sidekiq#performance
It's 7100 jobs/sec x 215 jobs/sec.
I wonder how GoodJob will compare.
https://github.com/mperham/sidekiq#performance
That benchmark is just showing job overhead. That Sidekiq can do nothing 7100 times a second only matters when overhead is significant, i.e. when your jobs do almost nothing.
delayed_job hasn't seen a release in 1 year, if that matters to you https://rubygems.org/gems/delayed_job/
There is a lot of exciting stuff landing in Rails and it’s surrounding ecosystems at the moment and I think it’s still probably the best choice for many new SAAS / web app based companies and startups.