Postgres Is Your Friend. ORM Is Not(hypha.pub)
hypha.pub
Postgres Is Your Friend. ORM Is Not
https://hypha.pub/postgres-is-your-friend-orm-is-not
56 comments
Ah, we haven't had an "ORMs bad" post for at least three days.
Everyone too busy writing and posting their "How I use Think for me Saas" / BrEaK It InTo SmAlLeR TaSkS blog posts.
Postgres is amazing and ORM is your friend. Migrations alone is a good reason why using ORMs is a good idea.
Migrations aren’t necessarily tied to ORMs. There are tons of tools out there to run migrations and nothing else.
Agreed, in many Spring projects I worked migrations were handled by flyway or liquibase while the ORM was always Hibernate.
There are valid reasons for not using an ORM but the points made in this article are plain false.
>“Roughly” because Django ORM doesn’t support the JSONB `?` operator.
The `has_key` [lookup](https://docs.djangoproject.com/en/6.0/topics/db/queries/#has...) does exactly that.
> And if you need real SQL intervals, Django pushes you towards raw expressions or `Func()` wrappers.
It's possible to use a very similar construct to SQL Alchemy here by using the `Now` [function](https://docs.djangoproject.com/en/6.0/ref/models/database-fu...) (it uses `STATEMENT_TIMESTAMP` which is likely more correct than `NOW()` here alternatively there is `TransactionNow`) by doing `Now() - timedelta(days=30)`.
The result is the following `filter` call
>“Roughly” because Django ORM doesn’t support the JSONB `?` operator.
The `has_key` [lookup](https://docs.djangoproject.com/en/6.0/topics/db/queries/#has...) does exactly that.
> And if you need real SQL intervals, Django pushes you towards raw expressions or `Func()` wrappers.
It's possible to use a very similar construct to SQL Alchemy here by using the `Now` [function](https://docs.djangoproject.com/en/6.0/ref/models/database-fu...) (it uses `STATEMENT_TIMESTAMP` which is likely more correct than `NOW()` here alternatively there is `TransactionNow`) by doing `Now() - timedelta(days=30)`.
The result is the following `filter` call
filter(
metadata__tags__has_key="python",
created_at__gte=(
Now() - timedelta(days=30)
),
)
("app_video"."metadata" -> 'tags') ? 'python'
AND "app_video"."created_at" >= (
STATEMENT_TIMESTAMP() - '30 days'::interval
)
Also, the django ORM is fairly easy to extend, it's not like it was written as one big blob.
Agreed, write raw SQL, this has never had any security impact whatsoever[1]
- Your friendly local pentester
[1] - https://en.wikipedia.org/wiki/SQL_injection
- Your friendly local pentester
[1] - https://en.wikipedia.org/wiki/SQL_injection
Parameterized queries have been a thing for decades, which mitigate SQL injection attacks.[1] This is true of the examples in the post too, they used this:
query = """
SELECT * from tasks
WHERE id = $1
AND state = $2
FOR UPDATE SKIP LOCKED
"""
rec = await self.db.fetchone(query=query, args=[task_id, TaskState.PENDING], connection=connection)
Parameterized queries fail to protect from SQL injection for decades, because database engine developers fail to listen. What could work instead, if any parameter could be safely injected:
SELECT $1, $2($3) FROM $4
WHERE $5 $6 $7
GROUP BY $1
ORDER BY $8 $9
Porsager’s Postgres package does a great job of letting you feel like you’re writing raw sql, but avoids the attack vectors.
Anyway, I agree that ORMs are pretty terrible. I like writing SQL or using a lightweight builder like Kysely. Was a huge Dapper fan back in my C# days.
There are plenty of reasonable alternatives to ORMs that don’t open you to SQL injection attacks.
Anyway, I agree that ORMs are pretty terrible. I like writing SQL or using a lightweight builder like Kysely. Was a huge Dapper fan back in my C# days.
There are plenty of reasonable alternatives to ORMs that don’t open you to SQL injection attacks.
Though the article mentions the distinction between ORMs and query builders, it doesn't really make a case against query builders being bad. After all, it wraps up by building a kinda crappy one-off query builder.
While there is a compelling case for leveraging the full power of Postgres (especially features like SKIP LOCKED and pg_notify), this approach feels like a classic trade-off between fine-grained control and long-term maintainability.
Relying solely on raw SQL and manual mapping certainly eliminates "ORM magic," but it replaces it with a significant maintenance burden.
For specialized, high-performance systems like video transcoding, this level of hand-tuning is a superpower; however, for the average CRUD-heavy SaaS app, the "boilerplate tax" of writing eighty lines of repository code for a simple related insert might eventually cost more in development velocity than the performance gains are worth.
Relying solely on raw SQL and manual mapping certainly eliminates "ORM magic," but it replaces it with a significant maintenance burden.
For specialized, high-performance systems like video transcoding, this level of hand-tuning is a superpower; however, for the average CRUD-heavy SaaS app, the "boilerplate tax" of writing eighty lines of repository code for a simple related insert might eventually cost more in development velocity than the performance gains are worth.
> for the average CRUD-heavy SaaS app, the "boilerplate tax" of writing eighty lines of repository code for a simple related insert might eventually cost more in development velocity than the performance gains are worth.
Perhaps, but IME this kind of thing is much more often the cause of poor performance in CRUD apps than the frontend frameworks that are usually blamed. I have been able to make very snappy SaaS apps by minimizing the number of queries that my API endpoints need to perform.
I've also found that the ORM mainly reduces boilerplate for Insert/Update operations, but often adds a very significant amount of boilerplate for read queries. We ended up using a very lightweight orms for simple inserts / upserts and writing raw SQL for anything else.
Perhaps, but IME this kind of thing is much more often the cause of poor performance in CRUD apps than the frontend frameworks that are usually blamed. I have been able to make very snappy SaaS apps by minimizing the number of queries that my API endpoints need to perform.
I've also found that the ORM mainly reduces boilerplate for Insert/Update operations, but often adds a very significant amount of boilerplate for read queries. We ended up using a very lightweight orms for simple inserts / upserts and writing raw SQL for anything else.
> For specialized, high-performance systems like video transcoding, this level of hand-tuning is a superpower
Where does SQL fall in the video transcoding pipeline?
Where does SQL fall in the video transcoding pipeline?
[deleted]
[deleted]
I sure loved delving into this blog post.
Yup, sloppy slop-slop
Is there a platform convention for this yet? Can we flag obvious slop?
This again...
No, just because raw SQL queries work great for your toy blog/todo app with 3 tables and simple relationships, doesn't mean they work great for real world business applications with 100 tables and complex networks of relationships. Try maintaining the latter before you make blanket claims like "ORM bad".
No, just because raw SQL queries work great for your toy blog/todo app with 3 tables and simple relationships, doesn't mean they work great for real world business applications with 100 tables and complex networks of relationships. Try maintaining the latter before you make blanket claims like "ORM bad".
> just because raw SQL queries work great for your toy blog/todo app with 3 tables
In my experience, ORMs work well for toy projects, but become cumbersome to maintain in enterprise ones, especially where performance matters. There is a large overlap between engineers who refuse to learn SQL because it's not "convenient", and those who prefer ORMs because they are "easier", resulting in cohorts that don't know how to use either.
But also, I don't see how ORMs make managing large databases any easier, other than those with embedded migration capabilities, which can be very well extracted to their own tools.
In my experience, ORMs work well for toy projects, but become cumbersome to maintain in enterprise ones, especially where performance matters. There is a large overlap between engineers who refuse to learn SQL because it's not "convenient", and those who prefer ORMs because they are "easier", resulting in cohorts that don't know how to use either.
But also, I don't see how ORMs make managing large databases any easier, other than those with embedded migration capabilities, which can be very well extracted to their own tools.
I guess I'll bite. How/When do you know you actually need an ORM?
If you're asking yourself "do I need an ORM?", then you should probably default to using one, unless you understand your complete use case well enough to know you'd be better off without one.
It's also important to note that not all ORMs are created equal. Some are more restrictive than others, and that should also be taken into account.
It's also important to note that not all ORMs are created equal. Some are more restrictive than others, and that should also be taken into account.
Maybe if you add a new variable to a table and need it in a query five views down? But honestly I much prefer using assisted queries like the supabase package, but leaving the tables alone. ORM can be very unwieldy in an unstable environment.
I’ve once tried a "type-safe" SQL extension and it was pretty neat.
Imho something like this is much more useful than a lot of ORM-overhead.
I’ve once tried a "type-safe" SQL extension and it was pretty neat.
Imho something like this is much more useful than a lot of ORM-overhead.
Is that python code that runs on postgres? how does this work..
When someone says that X is bad and not to use it, what I really hear is, “I’m ignorant to some use cases but that won’t stop me from having a loud opinion.”
I oscillate on being tired or amused by just how common tech people make this basic error. But I don’t believe it’s ever in bad faith. I think people in general suffer from perceiving their context as the context even though they’ve experienced maybe 1% of what there is out there.
I oscillate on being tired or amused by just how common tech people make this basic error. But I don’t believe it’s ever in bad faith. I think people in general suffer from perceiving their context as the context even though they’ve experienced maybe 1% of what there is out there.
Mostly agreed with the author about ORMs. The provided querying abstraction works against developers when queries reach a certain level of complexity, and at the end of the day, understanding these complexities is not optional.
But I would caution against adding too much business logic to the database, and tying message passing to your database doesn’t sound like the best of ideas.
But I would caution against adding too much business logic to the database, and tying message passing to your database doesn’t sound like the best of ideas.
But there's also value gained in it, isn't there? I very much like doctrine's query builders and being able to analyze and manipulate queries programmatically, e.g. dynamically add a filter to a query and a join if needed. That's pretty simple with a query builder once you've gotten comfortable with the concept and the ORM itself, but it's pretty hard to do with plain sql unless you write plenty of specific code to handle all the known things you might care about.
It sounds to me that you saying that SQL is hard because you’d rather learn the intricacies of an ORM.
Also, this whole point predicates upon the assumption that ORMs are infallible when translating queries into SQL, which most definitely are not.
Also, this whole point predicates upon the assumption that ORMs are infallible when translating queries into SQL, which most definitely are not.
No, I'm saying if you want to alter SQL queries programmatically, you'll either do some quick hacks with regexps that you'll regret, or you need to build something to do that, and that will look suspiciously like a query builder.
I’m having a hard time trying to understand exactly what you want to do, that cannot be done with SQL. Any concrete examples?
I personally rely on views to reuse base queries and then add filters on top of them.
I personally rely on views to reuse base queries and then add filters on top of them.
Sorry, got distracted.
Anything you can do with query builders, you can do with SQL of course, it's just more difficult / verbose (to me).
Let's say I have a lot of entities that are tied to domains, and I have users that have roles on those domains. I do a lot of reports, and for the vast majority, I don't have to worry about whether a user has the necessary access on the domain, I just call a central service, pass my query builder, the user, and the required access level, and it figures out whether the query builder already contains the necessary parts or not, which joins are required (e.g. if I'm selecting from subdomain, I need to join with domain and then join domain to user_domain) and add all of that transparently. I can treat service accounts differently, and I can just ignore all of that noise if a super admin is running the report.
I could absolutely do all of that in SQL, the SQL doesn't really change (if anything, query builders tend to produce worse SQL in my experience since you're not writing DB-specific code), just how I interact with it. The abstraction allows me to compose the query instead of writing it, giving me more flexibility.
Anything you can do with query builders, you can do with SQL of course, it's just more difficult / verbose (to me).
Let's say I have a lot of entities that are tied to domains, and I have users that have roles on those domains. I do a lot of reports, and for the vast majority, I don't have to worry about whether a user has the necessary access on the domain, I just call a central service, pass my query builder, the user, and the required access level, and it figures out whether the query builder already contains the necessary parts or not, which joins are required (e.g. if I'm selecting from subdomain, I need to join with domain and then join domain to user_domain) and add all of that transparently. I can treat service accounts differently, and I can just ignore all of that noise if a super admin is running the report.
I could absolutely do all of that in SQL, the SQL doesn't really change (if anything, query builders tend to produce worse SQL in my experience since you're not writing DB-specific code), just how I interact with it. The abstraction allows me to compose the query instead of writing it, giving me more flexibility.
This straw-mans ORMs by listing out what crappy ones do. I mean, accidental writes? You've either got a terrible ORM, no tests, or both.
Considering anybody with a noggin is going to be separating the SQL into it’s own module or whatever rather than just throwing straight inline SQL at your database wherever you it, you’re hardly less likely to have things like accidental writes, anyway. This is clearly someone who fell in love with Postgres, felt ORM abstractions that diluted the Postgres goodness were bad, and then did some mental experiments to consider all of the theoretical ways ORMs suck.
I gotta agree with you, total straw man. I haven't seen any of the issues this guy has with ORMs.
Yeah ORMs help when they're appropriate but ya gotta learn how they work and where the footguns are plus you still really want to know how a database server works. Given the articles title, I doubt the prerequisites were met.
MarcLore(7)
I’m glad you’re head-over-heels in love with Postgres— it’s really cool, and I’ve occasionally had projects that really benefited from it… but most of those incredible features just aren’t useful for run-of-the-mill projects. Learning how to profile your ORM queries is a lot easier than maintaining a bunch of code from a different language embedded into your code base. If you’re writing articles about Postgres, you probably have no idea how much of a PITA that context switch is in practice. It’s funny how getting expertise in something can make it more difficult to understand why it’s useful to other people, and how they use it.
There are projects, like SQLC, that cover most of the perceived advantages of ORMs, without the downsides.
One of these downsides is, in my opinion, the fact that they hide the very details of the implementation one necessarily needs to understand, in order to debug it.
One of these downsides is, in my opinion, the fact that they hide the very details of the implementation one necessarily needs to understand, in order to debug it.
Let me just say that I wrote my first (professional) SQL queries about 25 years ago and at various post points have worked extensively with Postgres, a bit less so with Oracle, and occasionally with MySQL and MSSQL. (And also some of the JSON object store databases before switching to Postgres for that stuff.) The only ones I’ve used ORMs with are Postgres and MySQL.
SQLC does not address most of the perceived advantages to ORMs. Sure it addresses some of the concerns of hand-writing and sending SQL to databases from various languages, but that’s not what most people I’ve spoken to in the past couple of decades most valued about ORMs. What most projects really need databases for is some place to essentially store context-sensitive variable values. Like what email address to send something to if the user ID is 12345. I’ve never, ever had to debug ORM’s SQL when doing things like that. Rarely have I needed to with more complex chains of filters or whatnot, and that usually involved taking a slightly different approach with the given ORM tools rather than modifying them or writing my own SQL. When I’ve had more complex needs that required using some of the more exotic Postgres features, writing my own queries has been trivial. It’s of paramount importance for developers to understand the frameworks and libraries, such as ORMs, they’re using because those implementation details touch everything in your code. Once you understand that, the code your ORM composes to make your queries is an IDE-click away.
Not having to context switch between writing SQL and whatever native language you’re working in, especially for simple tasks, has yielded so so so much more to my time and mental space than being exactly 100% sure that my code is using that left join in exactly the way I want it to.
SQLC does not address most of the perceived advantages to ORMs. Sure it addresses some of the concerns of hand-writing and sending SQL to databases from various languages, but that’s not what most people I’ve spoken to in the past couple of decades most valued about ORMs. What most projects really need databases for is some place to essentially store context-sensitive variable values. Like what email address to send something to if the user ID is 12345. I’ve never, ever had to debug ORM’s SQL when doing things like that. Rarely have I needed to with more complex chains of filters or whatnot, and that usually involved taking a slightly different approach with the given ORM tools rather than modifying them or writing my own SQL. When I’ve had more complex needs that required using some of the more exotic Postgres features, writing my own queries has been trivial. It’s of paramount importance for developers to understand the frameworks and libraries, such as ORMs, they’re using because those implementation details touch everything in your code. Once you understand that, the code your ORM composes to make your queries is an IDE-click away.
Not having to context switch between writing SQL and whatever native language you’re working in, especially for simple tasks, has yielded so so so much more to my time and mental space than being exactly 100% sure that my code is using that left join in exactly the way I want it to.
First of all, congrats on your career.
Second, an ORM is just a translation layer, i.e. it does not compile to any binary format the database understands, and instead it gets translated to SQL, which is the standard, minus extensions. SQL is ubiquitous. It’s the closest to a lingua franca that we have in the context of software engineering. And I’m going to be blunt here and say that purposefully avoid learning and understanding SQL if it is part of the job, should disqualify anyone from it.
I’ve been around for some decades too, and to me, ORMs haven’t worked out. They are vastly different one from another and they often create issues that are clear as day when the query is written as SQL. If I go from a Typescript codebase to Python to Java, then, according to you, I should learn the intricacies of Sequelize, SQLAlchemy, and JPA/Hibernate, instead of just SQL. And granted, different SQL dialects have different quirks, but more often than not, the pitfalls are more apparent than when switching between ORMs.
And I can guarantee that someone equipped with a good foundation in SQL will be more successful debugging a Sequelize based application, than someone who has relied on SQLAlchemy.
What most people I know and worked with need, is types. Types help glue SQL and any other language together. If I can run any SQL query and the result comes back as an object, I’m good.
Now, if your point is that ORMs are OK for toying around, I may agree, but still, why would I go through that trouble when I know SQL.
Second, an ORM is just a translation layer, i.e. it does not compile to any binary format the database understands, and instead it gets translated to SQL, which is the standard, minus extensions. SQL is ubiquitous. It’s the closest to a lingua franca that we have in the context of software engineering. And I’m going to be blunt here and say that purposefully avoid learning and understanding SQL if it is part of the job, should disqualify anyone from it.
I’ve been around for some decades too, and to me, ORMs haven’t worked out. They are vastly different one from another and they often create issues that are clear as day when the query is written as SQL. If I go from a Typescript codebase to Python to Java, then, according to you, I should learn the intricacies of Sequelize, SQLAlchemy, and JPA/Hibernate, instead of just SQL. And granted, different SQL dialects have different quirks, but more often than not, the pitfalls are more apparent than when switching between ORMs.
And I can guarantee that someone equipped with a good foundation in SQL will be more successful debugging a Sequelize based application, than someone who has relied on SQLAlchemy.
What most people I know and worked with need, is types. Types help glue SQL and any other language together. If I can run any SQL query and the result comes back as an object, I’m good.
Now, if your point is that ORMs are OK for toying around, I may agree, but still, why would I go through that trouble when I know SQL.
SQLC for me has been able to replace most cases of use an ORM for. It made most of the boilerplate of using plain SQL go away, I get type safe responses, and it forces me to be more mindful of the queries I write.
In an app where we do use an ORM (Prisma), we sometimes have weird database spikes and it’s almost always an unintended heavy ORM query.
The only two things I miss in solutions like sqlc are dynamic queries (filters, partial inserts) and the lack of a way to add something to every query by default (e.g., always filtering by tenant_id.)
In an app where we do use an ORM (Prisma), we sometimes have weird database spikes and it’s almost always an unintended heavy ORM query.
The only two things I miss in solutions like sqlc are dynamic queries (filters, partial inserts) and the lack of a way to add something to every query by default (e.g., always filtering by tenant_id.)