Make Your Back End Layer as Thin as Possible(kartick.substack.com)
kartick.substack.com
Make Your Back End Layer as Thin as Possible
https://kartick.substack.com/p/make-your-backend-layer-as-thin-as
63 comments
Author here.
> This article doesn't use the word "permission" or "validation"
The article talks about it multiple times: using a database user account with limited privledges, exposing some tables but not others, exposing read but not write access, giving users access only to data they own.
> it's nearly impossible to parse and check an arbitrary SQL query for malicious intent
Which is why the article doesn't propose trying to parse and check queries for malicious intent.
> it now passes the responsibility of query performance to the FE engineer. Are appropriate indexes in place? Does the query make inappropriate JOINs?
No, in the proposal, we have backend engineer(s) to advise and assist frontend engineers. Having appropriate indices and JOINs doesn't mean you have a layer that doesn't add business value. And when it does add value, write it!
> schema changes now mean that you need to update your front end code. This means guaranteed hard downtime, because you can't control what JS folks are running in the browser.
I don't know if I understood you, but you can just force a refresh in the browser.
> you also need to make sure that queries aren't designed to intentionally DoS your DB.
That's a valid point I didn't think of.
> There's a lot wrong with the ideas presented here.
It's hard to take your criticism seriously when many of your reactions are a result of your own misunderstanding of the proposal.
> This article doesn't use the word "permission" or "validation"
The article talks about it multiple times: using a database user account with limited privledges, exposing some tables but not others, exposing read but not write access, giving users access only to data they own.
> it's nearly impossible to parse and check an arbitrary SQL query for malicious intent
Which is why the article doesn't propose trying to parse and check queries for malicious intent.
> it now passes the responsibility of query performance to the FE engineer. Are appropriate indexes in place? Does the query make inappropriate JOINs?
No, in the proposal, we have backend engineer(s) to advise and assist frontend engineers. Having appropriate indices and JOINs doesn't mean you have a layer that doesn't add business value. And when it does add value, write it!
> schema changes now mean that you need to update your front end code. This means guaranteed hard downtime, because you can't control what JS folks are running in the browser.
I don't know if I understood you, but you can just force a refresh in the browser.
> you also need to make sure that queries aren't designed to intentionally DoS your DB.
That's a valid point I didn't think of.
> There's a lot wrong with the ideas presented here.
It's hard to take your criticism seriously when many of your reactions are a result of your own misunderstanding of the proposal.
While I largely share your broader concerns,
> ensuring that queries that should read only read
seems to be the purview of the DBMS, setting appropriate account permissions. The bit you quoted said "a database user account that has only read access".
> ensuring that queries that should read only read
seems to be the purview of the DBMS, setting appropriate account permissions. The bit you quoted said "a database user account that has only read access".
On MySQL, you can run SLEEP() or BENCHMARK() as a read-only user to your heart's content. You cannot restrict access to these functions, and letting the wide internets play with them will bring your server to its knees by clogging up all connections and hogging up its CPUs.
I wasn't contradicting the bits about DOS attacks, which remain very much a concern, even when no one is malicious.
Solution: Don't use MySQL
PostgreSQL's SQL is Turing-complete. Imagine the possibilities!
All rdbms have similar issues with built-in functions. Plus you can easily create pathological queries that do the same.
This assumes you only deal with non-sensitive data (e.g. user profiles).
Never mind that, how do you prevent denial of service attacks? I can run any allowed query on that database. Including 50 times cross join of the table I can read, sorted by random.
Never mind that, how do you prevent denial of service attacks? I can run any allowed query on that database. Including 50 times cross join of the table I can read, sorted by random.
> This assumes you only deal with non-sensitive data (e.g. user profiles).
Not really. You can do row or column level permissions. Details very much depend on your DBMS.
> Never mind that, how do you prevent denial of service attacks?
I have no real answer, and I never suggested that I would. Denial of service attacks (deliberate or accidental) are one piece of the "broader concerns" that I very much share.
In theory, a sufficiently advanced DBMS might be able to assign users quotas, and if your users are stable that may be sufficient for some use cases, but I am skeptical.
Not really. You can do row or column level permissions. Details very much depend on your DBMS.
> Never mind that, how do you prevent denial of service attacks?
I have no real answer, and I never suggested that I would. Denial of service attacks (deliberate or accidental) are one piece of the "broader concerns" that I very much share.
In theory, a sufficiently advanced DBMS might be able to assign users quotas, and if your users are stable that may be sufficient for some use cases, but I am skeptical.
Firebase was recommended in the post, which has built in security rules and auth.
Then the author should really search-and-replace every instance of "database" with "Firebase"; their advice doesn't work very well otherwise.
The article literally says "Instead, you can define an API that takes an SQL query from the client, runs it, and returns the results," so...
I get the reasoning behind this, and I agree with some of that. But I don't agree with this. For reasons:
1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious. Validations must be duplicated in both the UI (to show people that they can't do the thing), and the back end (to stop them from the doing the thing).
2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
3. The entities that make the most sense for storing your data into a database are not the same entities that make the most sense for using in a UI. Forcing your UI to use the same entities as your database will mean more work for both, and often ends up influencing the database design badly.
4. There are ways of writing APIs that allow for rapid change. Uncoupling the storage entities from the display entities is a key step, so you can change things in the database without affecting the front end (and vice versa).
5. There are ways of writing database schemas that allow for rapid change. My favourite is adding a hstore "metadata" field that I can use to create "temporary" fields in the table for experimental features.
I've written a few back ends, and the advice in the article strikes me as dangerously sensible-sounding while being mostly wrong. I can see a newbie product manager buying into this completely and making a huge mess.
1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious. Validations must be duplicated in both the UI (to show people that they can't do the thing), and the back end (to stop them from the doing the thing).
2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
3. The entities that make the most sense for storing your data into a database are not the same entities that make the most sense for using in a UI. Forcing your UI to use the same entities as your database will mean more work for both, and often ends up influencing the database design badly.
4. There are ways of writing APIs that allow for rapid change. Uncoupling the storage entities from the display entities is a key step, so you can change things in the database without affecting the front end (and vice versa).
5. There are ways of writing database schemas that allow for rapid change. My favourite is adding a hstore "metadata" field that I can use to create "temporary" fields in the table for experimental features.
I've written a few back ends, and the advice in the article strikes me as dangerously sensible-sounding while being mostly wrong. I can see a newbie product manager buying into this completely and making a huge mess.
Author here:
> 1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious.
This is already taken care of in the proposal, by permitting clients to access only data they own.
> 2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
No, because a) not every startup starts on multiple platforms b) you can always put common behavior into a piece that's reused. This can be a library linked in to multiple frontends, or a backend API. The post says your backend should be as thin AS POSSIBLE. It doesn't say you shouldn't have a backend at all costs, even if that means duplicating code.
> Forcing your UI to use the same entities as your database.
Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
> The advice in the article strikes me as dangerously sensible-sounding while being mostly wrong.
I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
> 1. You can't trust the front end. Backends must be written assuming that every call from the front end is malicious.
This is already taken care of in the proposal, by permitting clients to access only data they own.
> 2. There will be multiple front ends. If all your business logic is in the front end, you'll have to duplicate it into all of them. Where it will rapidly get out of synch and you'll have different behaviours on different clients.
No, because a) not every startup starts on multiple platforms b) you can always put common behavior into a piece that's reused. This can be a library linked in to multiple frontends, or a backend API. The post says your backend should be as thin AS POSSIBLE. It doesn't say you shouldn't have a backend at all costs, even if that means duplicating code.
> Forcing your UI to use the same entities as your database.
Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
> The advice in the article strikes me as dangerously sensible-sounding while being mostly wrong.
I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
> This is already taken care of in the proposal, by permitting clients to access only data they own.
But you don't propose any method for that apart from hand-waving about "database user accounts with read access", which is largely irrelevant unless you're proposing to give every one of your customers a database account and do row-level database security.
Let's talk about a concrete example: A call arrives at the back end to change the user's name to "dickwad".
In a conventional backend, you'd check the session, work out which user that session was logged in as, check that it's the same account as the name that's going to change, and maybe then run a profanity filter, or not.
How would you build your minimal backend to do that? Remember, you don't actually know that this call has come from the front end, it could be sent from curl.
> Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
True, but that's not mentioned at all in your article, and defeats half the point of it - the front end engineer still has to worry about what the database needs and how to do that mapping when it changes.
> I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
What misconceptions? I understood you (almost) perfectly. It's a well-written article. Still wrong, though.
But you don't propose any method for that apart from hand-waving about "database user accounts with read access", which is largely irrelevant unless you're proposing to give every one of your customers a database account and do row-level database security.
Let's talk about a concrete example: A call arrives at the back end to change the user's name to "dickwad".
In a conventional backend, you'd check the session, work out which user that session was logged in as, check that it's the same account as the name that's going to change, and maybe then run a profanity filter, or not.
How would you build your minimal backend to do that? Remember, you don't actually know that this call has come from the front end, it could be sent from curl.
> Nothing prevents you from mapping the entities to different ones for your frontend, in cases where you need that.
True, but that's not mentioned at all in your article, and defeats half the point of it - the front end engineer still has to worry about what the database needs and how to do that mapping when it changes.
> I don't think you even understood the advice in the first place, as I can see from your misconceptions above.
What misconceptions? I understood you (almost) perfectly. It's a well-written article. Still wrong, though.
> This is already taken care of in the proposal, by permitting clients to access only data they own.
I'm curious how this works. In a database that stores something like an address, how do you prevent someone from slightly changing the query to give them someone else's address stored in the same table. I've heard of giving per-table access, but never per-row.
I'm curious how this works. In a database that stores something like an address, how do you prevent someone from slightly changing the query to give them someone else's address stored in the same table. I've heard of giving per-table access, but never per-row.
alter table addresses enable row level security;
However this article is insane because of the security nightmare of trusting all incoming queries
However this article is insane because of the security nightmare of trusting all incoming queries
I strongly disagree with the advice in this article. There are many reasons. I’ll name a few.
Your DB is likely to contain internal state that has no business being sent to the client. Over time applications change and fields can change meaning, become deprecated, or become dependent on other fields through business logic.
If you do this, any business logic not captured in your backend has to be duplicated in each front end. Imagine needing to write and maintain the same business logic in a JavaScript framework, Java for Android, and Swift for your iOS app.
How do you make breaking DB changes if you take on this strategy of API design?
In most cases in my professional career, making a choice to have a thin RESTful API that corresponds closely to the DB structure has been a mistake that only time made clear. RPC-like APIs fared much better.
This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten. I would not recommend it for anyone wishing to build a lasting architecture.
Don’t take my word for it. Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
Your DB is likely to contain internal state that has no business being sent to the client. Over time applications change and fields can change meaning, become deprecated, or become dependent on other fields through business logic.
If you do this, any business logic not captured in your backend has to be duplicated in each front end. Imagine needing to write and maintain the same business logic in a JavaScript framework, Java for Android, and Swift for your iOS app.
How do you make breaking DB changes if you take on this strategy of API design?
In most cases in my professional career, making a choice to have a thin RESTful API that corresponds closely to the DB structure has been a mistake that only time made clear. RPC-like APIs fared much better.
This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten. I would not recommend it for anyone wishing to build a lasting architecture.
Don’t take my word for it. Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
If you use React Native, React and a shared Redux layer you can share all this business logic between iOS, Android and Web. If you make breaking DB changes you still have the REST layer to buffer them. If REST won't cover your breaking changes with data transforms because you are deleting whole tables and restructuring then it's time to refactor the front end as well anyway. I disagree with the front end sending SQL directly but a thin REST layer with a little validation seems ideal to me.
Op, please listen to the advice given here. If your blog post is a reflection of your startup's production stack, I would be very concerned.
Author here.
> Your DB is likely to contain internal state that has no business being sent to the client.
Which is why not all tables need to be sent to the client. The post gives an example of exposing the friends table to the client but not others.
> If you do this, any business logic not captured in your backend has to be duplicated in each front end.
No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
> How do you make breaking DB changes if you take on this strategy of API design?
You change the frontend!
> This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten.
Which is why the blog post begins by saying, "Say you've started a startup today"
> I would not recommend it for anyone wishing to build a lasting architecture.
In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
> Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
> Your DB is likely to contain internal state that has no business being sent to the client.
Which is why not all tables need to be sent to the client. The post gives an example of exposing the friends table to the client but not others.
> If you do this, any business logic not captured in your backend has to be duplicated in each front end.
No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
> How do you make breaking DB changes if you take on this strategy of API design?
You change the frontend!
> This advice may be good for a proof of concept, a prototype, or an early version which stands a good chance of getting rewritten.
Which is why the blog post begins by saying, "Say you've started a startup today"
> I would not recommend it for anyone wishing to build a lasting architecture.
In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
> Take a look at your favorite famous internet company and look at their API and try to deduce if they follow this advice or not.
As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
> Which is why not all tables need to be sent to the client. The post gives an example of exposing the friends table to the client but not others.
What if the "is_active" flag takes on more nuanced meaning over time, as you add "client_activation_status", and over time expand that meaning. All frontend clients of the API need to be aware of such low-level changes. An API acts as an insulation layer.
> No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
At one point the post says to get rid of it altogether and use SQL, making the architecture 2-layer. If you do have a backend layer, and it does have business logic, that business logic will stand between your DB and the client. This logic will vary in thickness, but in a mature application will be substantial.
> You change the frontend!
How do you force a synchronized iOS app update to all the people who have your app installed?
> Which is why the blog post begins by saying, "Say you've started a startup today"
Fair point. Still, it is not something I would do except in throwaway prototypes or proof-of-concepts.
> In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
Agreed on this advice. And you need to ask yourself how you will gradually adapt the software you've built so that it can change with the needs, risk profile of your clients, and growth of your company. An approach to API design you describe is fundamental, and it would be difficult to adapt it without scratching it altogether and starting afresh with v2.
> As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
Agree about not cargo-culting. Agree about adjusting approach to company maturity. My experience has showed me that the approach you suggest has too many disadvantages for building any production software.
I wish you best of luck and hope that at the very least I and others on this thread were able to offer another perspective.
What if the "is_active" flag takes on more nuanced meaning over time, as you add "client_activation_status", and over time expand that meaning. All frontend clients of the API need to be aware of such low-level changes. An API acts as an insulation layer.
> No, the post says to make your backend layer as thin AS POSSIBLE, not to not have a backend layer at all costs even if it means duplicating code.
At one point the post says to get rid of it altogether and use SQL, making the architecture 2-layer. If you do have a backend layer, and it does have business logic, that business logic will stand between your DB and the client. This logic will vary in thickness, but in a mature application will be substantial.
> You change the frontend!
How do you force a synchronized iOS app update to all the people who have your app installed?
> Which is why the blog post begins by saying, "Say you've started a startup today"
Fair point. Still, it is not something I would do except in throwaway prototypes or proof-of-concepts.
> In a startup, I wouldn't recommend over-designing on day 1, because that's a good way of not delivering enough business value fast enough and so going bust.
Agreed on this advice. And you need to ask yourself how you will gradually adapt the software you've built so that it can change with the needs, risk profile of your clients, and growth of your company. An approach to API design you describe is fundamental, and it would be difficult to adapt it without scratching it altogether and starting afresh with v2.
> As the blog post says, you shouldn't cargo-cult Google or Amazon. You don't operate at their scale, you don't have as much traffic, you don't have as many teams, and so on. You should what makes sense for you given your company's maturity and where in the product lifecycle you are.
Agree about not cargo-culting. Agree about adjusting approach to company maturity. My experience has showed me that the approach you suggest has too many disadvantages for building any production software.
I wish you best of luck and hope that at the very least I and others on this thread were able to offer another perspective.
Let's say my startup makes a product featuring some sort of kanban-like board. The user moves a card from one column to the other. According to this article, the frontend might just need to perform an UPDATE on the right table from the database and that would be it.
Yet, this is what my backend does:
* make sure the user is allowed to move that particular card.
* update the database.
* add an entry to the board's activity log.
* notify all connected clients listening to events from this board that a change occurred, so they can refresh the UI instantly.
* if some users asked to be notified about changes, generate notifications.
* if some of these users are offline but still want to be alerted, generate and queue some emails to send.
* add an entry in a raw text log for easy debugging.
* register the event in some kind of analytics storage for future stats.
* if the board is integrated with e.g. Slack, call Slack's API.
* if some users registered webhooks through my API, trigger those.
I'm so glad my frontend engineers actually do not have to worry about how to do any of that.
Edit: format
Yet, this is what my backend does:
* make sure the user is allowed to move that particular card.
* update the database.
* add an entry to the board's activity log.
* notify all connected clients listening to events from this board that a change occurred, so they can refresh the UI instantly.
* if some users asked to be notified about changes, generate notifications.
* if some of these users are offline but still want to be alerted, generate and queue some emails to send.
* add an entry in a raw text log for easy debugging.
* register the event in some kind of analytics storage for future stats.
* if the board is integrated with e.g. Slack, call Slack's API.
* if some users registered webhooks through my API, trigger those.
I'm so glad my frontend engineers actually do not have to worry about how to do any of that.
Edit: format
In this case, I wouldn't disagree with you if you were to do it on the backend.
The article says, "build a backend API when it has something to do, not as a pass through to your database."
It doesn't say "never write any backend code".
The article says, "build a backend API when it has something to do, not as a pass through to your database."
It doesn't say "never write any backend code".
The article's subtitle literally says "The best backend is no backend at all". The article gives the impression you have no idea what a backend is supposed to do, except expose some data.
All very easy to implement with no backend using Firebase.
Don't do it! Not if your application is more than a toy.
I get the allure of this kind of approach, but the costs in usability and defense against pathological queries (given nontrivial data volumes) are way too heavy to bear for the superficial convenience. And the justifications related to the sorts of problems avoided don't pass the smell test.
I get the allure of this kind of approach, but the costs in usability and defense against pathological queries (given nontrivial data volumes) are way too heavy to bear for the superficial convenience. And the justifications related to the sorts of problems avoided don't pass the smell test.
This is generally ok advice but it leaves out an important detail about making backwards compatible changes. If your backend doesn’t do anything, every change to your database will break your client unless they are updated simultaneously every time (impossible for mobile apps) or your client is aware of every backend change (very complex). Not to mention there is some data you simply don’t want to expose to a client in most situations, like user emails or hashed passwords.
Author here.
> every change to your database will break your client unless they are updated simultaneously every time (impossible for mobile apps)
You can just put up a UI forcing the user to update. Which is perfectly fine since you won't be making backward-incompatible changes to your database every month. Adding a new column doesn't break existing clients.
> or your client is aware of every backend change (very complex)
You can just put in a build date in your frontend and compare it against the date a breaking change was made. If it's <, refresh.
> every change to your database will break your client unless they are updated simultaneously every time (impossible for mobile apps)
You can just put up a UI forcing the user to update. Which is perfectly fine since you won't be making backward-incompatible changes to your database every month. Adding a new column doesn't break existing clients.
> or your client is aware of every backend change (very complex)
You can just put in a build date in your frontend and compare it against the date a breaking change was made. If it's <, refresh.
> You can just put up a UI forcing the user to update
sounds very user-friendly
sounds very user-friendly
Does the article mention business logic?
Even at MVP stage, many backends do quite a bit more than CRUD.
This is why I wouldn't pick e.g. https://github.com/PostgREST/postgrest even when it's a perfectly fine (and probably well thought-out) project.
Even at MVP stage, many backends do quite a bit more than CRUD.
This is why I wouldn't pick e.g. https://github.com/PostgREST/postgrest even when it's a perfectly fine (and probably well thought-out) project.
Business logic should not happen at API boundaries (regardless of what kinds of technologies or transport those interface boundaries are facilitating). Interfaces should be a validation and translation layer that receive incoming data, hand off to business logic, receive outgoing data and perform whatever translation and serialization required to satisfy the return/response interface. Business logic should be completely isolated from that responsibility, and ideally transport-agnostic.
In a traditional “MVC” style architecture this would be described as “thin controller”. But that kind of architecture often tightly couples other things that shouldn’t be, because they hide other transport/interface boundaries.
The way I approach this is pretty simple:
1. Interfaces between user input and “programs” of any kind are used for validation, sanitization, deserialization and interface translation. Only. No state changes can occur here. In the language of the article, this is the “thin backend”.
2. Any runtime business logic, using any technology for that runtime, clearly defines its input and output types. This layer is allowed to make local state change as appropriate, but any external side effects are forbidden.
3. If the layer described in #2 needs to communicate with an outside network, system, storage or service layer, recurse to #1. Contracts between these boundaries are only that and imply no implementation details.
4. If you implement #3 in the service providing #2, those responsibilities are explicitly separated.
5. Do business logic. Return data consistent with the business logic’s contract.
6. Rewrap the onion and eventually serialize the result for the outside client according to that layer’s expectation, at whatever interface boundary you’re at.
In a traditional “MVC” style architecture this would be described as “thin controller”. But that kind of architecture often tightly couples other things that shouldn’t be, because they hide other transport/interface boundaries.
The way I approach this is pretty simple:
1. Interfaces between user input and “programs” of any kind are used for validation, sanitization, deserialization and interface translation. Only. No state changes can occur here. In the language of the article, this is the “thin backend”.
2. Any runtime business logic, using any technology for that runtime, clearly defines its input and output types. This layer is allowed to make local state change as appropriate, but any external side effects are forbidden.
3. If the layer described in #2 needs to communicate with an outside network, system, storage or service layer, recurse to #1. Contracts between these boundaries are only that and imply no implementation details.
4. If you implement #3 in the service providing #2, those responsibilities are explicitly separated.
5. Do business logic. Return data consistent with the business logic’s contract.
6. Rewrap the onion and eventually serialize the result for the outside client according to that layer’s expectation, at whatever interface boundary you’re at.
While I don't disagree with your remarks, I'm curious whom you're educating here and why; parent post makes no statements about controller/view/service/repository responsibilities, only that something very often needs to exist between frontend and backend to mediate, filter, enrich, and so on.
My response was that the implied place for business logic in the article (the layer behind the “thin backend”) is pretty clear, and that the response above mine seemed to suggest that the “somewhere” a business logic belongs is the “backend”, which at least using the article’s definitions and delineations that’s the worst place to put it.
> Business logic should not happen at API boundaries
You're right, but the original post seems to believe there are only three layers: frontend, API, database. They haven't left any place for business logic at all, except maybe in the way the front-end would write the SQL queries. (Which I think is a terrible approach.)
You're right, but the original post seems to believe there are only three layers: frontend, API, database. They haven't left any place for business logic at all, except maybe in the way the front-end would write the SQL queries. (Which I think is a terrible approach.)
You’re right too. And ultimately it moves the business logic to the frontend, and likely causes that layer to accrue the separations of concern most of us associate with “backend”. It has to live somewhere.
Author here.
The article says "build a backend API when it has something to do, not as a pass through to your database."
The article says "build a backend API when it has something to do, not as a pass through to your database."
As a former pentester, this makes me happy inside.
As a DevSecOps Engineer, I field requests like this from frontend developers all the time. The answer is always no.
In this case it would be "I am sorry that you are not happy with inter-team planning and communication; please raise this with the appropriate project owner at the next scrum meeting. Sadly poor communication can not be allowed to drive well established systems planning and architecture best practices..."
As a DevSecOps Engineer, I field requests like this from frontend developers all the time. The answer is always no.
In this case it would be "I am sorry that you are not happy with inter-team planning and communication; please raise this with the appropriate project owner at the next scrum meeting. Sadly poor communication can not be allowed to drive well established systems planning and architecture best practices..."
Sounds kind of dismissive and passive aggressive. The architecture suggested in the article is terrible but I think the way to respond to it is to explain why it's terrible from a security standpoint and then also a maintenance standpoint. If they are smart they will get it and then move on to finding a way to fix the planning problems. Stonewalling and defending your corner is something I have personally seen evolve as a kind of toxic thing in big companies leading to breakdown of communication.
Just don't.
This is not good advise unless for a toy app. The only time I've seen a direct DB access API useful and even remotely a good idea is to give another internal application or service access - for reporting or metrics data. Definately not for your public facing client apps.
- The database may contain a lot more info than the client apps need, you are sending back more data than is needed, which costs money.
- Apparently you have no business logic to transform that raw database data?
- If you have more than one client, you are likely to end up duplicating some kind of business logic in each client (say a native iOS app, native Android app, web).
- Security? I'm not even gonna start...
This whole article smells of someone who has not had a lot of real-world experience. Large application database a full of data that have no place being handed over to a app developer without any context or documentation of the database. It is much better to give back exactly what is needed for a particular view and no more.
If you neeed more or differnt data later, you version the API and do a rolling deploy after your clients have been updated.
> You can bypass this whole inefficiency by making the backend a pass-through. That gives you one fewer thing to worry about, which any engineer will welcome.
Um, yeah, and now you have front end security issues, since all the expressive power that you have just given to your front end engineers is also available to any damned fool who can fire up a browser console.
I strongly agree that you should make the distance between the front end and the data as thin as possible, but this article comes at that from the front-end developer perspective.
There is another option: make the front end as thin as possible. Eliminate the huge front end framework and go with something closer to the HTML.
This keeps the expressiveness on the server side, which is a trusted computing environment, rather than exposing it on the client side, in an untrusted computing environment, where security concerns become much more complicated.
Um, yeah, and now you have front end security issues, since all the expressive power that you have just given to your front end engineers is also available to any damned fool who can fire up a browser console.
I strongly agree that you should make the distance between the front end and the data as thin as possible, but this article comes at that from the front-end developer perspective.
There is another option: make the front end as thin as possible. Eliminate the huge front end framework and go with something closer to the HTML.
This keeps the expressiveness on the server side, which is a trusted computing environment, rather than exposing it on the client side, in an untrusted computing environment, where security concerns become much more complicated.
> In the three-tier frontend-backend-database architecture, try to have as thin a middle layer as possible. Or even eliminate the backend layer altogether by directly exposing the database to the frontend.
Wat...
This article reads like a jumbled mess of ideas. But, the above is a total wat...
And it also assumes that your backend doesn't have to consume and tie together several resources or that you need a caching layer and so many other naive assumptions - and I'm not even addressing the security implications.
Take this article with a heavy cup of salt.
To me it just reads like something someone would write because they want full control and does't like working with and/or being dependant on other people/teams.
Wat...
This article reads like a jumbled mess of ideas. But, the above is a total wat...
And it also assumes that your backend doesn't have to consume and tie together several resources or that you need a caching layer and so many other naive assumptions - and I'm not even addressing the security implications.
Take this article with a heavy cup of salt.
To me it just reads like something someone would write because they want full control and does't like working with and/or being dependant on other people/teams.
What they really want to useis Firebase and the like, which is fine for some. For others disregard this advice.
The company I work for did this, basically because the people working there had to crank out features as quickly as possible and had no idea what a "back end" even is.
Fast forward almost a decade, and this approach has generated a huge headache. The database is an incomprehensible mess, there's a whole class of features we cannot implement because of this (lack of) architecture, business logic is all over the place (with a million ways to do the exact same thing), mostly in the clients, performance is bad because every client is doing all sorts of queries all the time, and training a new developer either takes months or doesn't really work out at all - because who wants to read a decade's worth of spaghetti code. I've been working on a way to fix this, but it takes a lot of time and effort. Also management still doesn't understand the concept of "overwhelming technical debt".
The only reason this has even been hanging by a thread for so long is that we only make software where the back-end and the clients are run on-premise in a separate network. Otherwise it would be a security nightmare as well.
This article is basically a set of instructions to get to where we are. It's like saying that traffic lights are unnecessary because if you just floor it through an intersection, you'll get to your destination much faster. And you know what, it might just work the first few times. Until you find out why these things exist in the first place.
Fast forward almost a decade, and this approach has generated a huge headache. The database is an incomprehensible mess, there's a whole class of features we cannot implement because of this (lack of) architecture, business logic is all over the place (with a million ways to do the exact same thing), mostly in the clients, performance is bad because every client is doing all sorts of queries all the time, and training a new developer either takes months or doesn't really work out at all - because who wants to read a decade's worth of spaghetti code. I've been working on a way to fix this, but it takes a lot of time and effort. Also management still doesn't understand the concept of "overwhelming technical debt".
The only reason this has even been hanging by a thread for so long is that we only make software where the back-end and the clients are run on-premise in a separate network. Otherwise it would be a security nightmare as well.
This article is basically a set of instructions to get to where we are. It's like saying that traffic lights are unnecessary because if you just floor it through an intersection, you'll get to your destination much faster. And you know what, it might just work the first few times. Until you find out why these things exist in the first place.
Author here.
> Fast forward almost a decade
If your system lasted a decade, that's a success, and any startup that just started would love to be in your shoes.
> management still doesn't understand the concept of "overwhelming technical debt".
That's a problem with your company, not with the suggestion in my post of adding in layers of abstraction when needed rather than ahead of time.
> Fast forward almost a decade
If your system lasted a decade, that's a success, and any startup that just started would love to be in your shoes.
> management still doesn't understand the concept of "overwhelming technical debt".
That's a problem with your company, not with the suggestion in my post of adding in layers of abstraction when needed rather than ahead of time.
I'm not saying they weren't successful. As someone who tends to over-engineer things, I do admire their ability to get a product out quickly - I think for a startup that was probably the right decision!
But that's what I've been missing from your article - more on the distinction between the approaches, and the drawbacks of having a thin back-end layer as the product / company is growing. In the beginning I thought you were talking about how to make your startup more efficient and flexible - basically consciously building some technical debt, having weighed your options.
But if that's not the idea - personally I don't know how to scale with this approach, neither technologically nor in terms of team structure. The thin back end layer could work though if you have a lot of small products that are independent of each other - developed by small teams that can afford to act like early startups. Maybe that's what you meant, but I didn't really get that from the article.
But that's what I've been missing from your article - more on the distinction between the approaches, and the drawbacks of having a thin back-end layer as the product / company is growing. In the beginning I thought you were talking about how to make your startup more efficient and flexible - basically consciously building some technical debt, having weighed your options.
But if that's not the idea - personally I don't know how to scale with this approach, neither technologically nor in terms of team structure. The thin back end layer could work though if you have a lot of small products that are independent of each other - developed by small teams that can afford to act like early startups. Maybe that's what you meant, but I didn't really get that from the article.
I get that you've disagreed with almost every point telling you this is a bad idea, but I think the point is particularly pertinent.
They tried this system, and then realised they painted themselves into a corner. Businesses wants features they can sell to customers, and sometimes fixing technical debt is a luxury. Especially when technical debt hidden to customers.
Adding layers of abstraction when needed might sound like great advice to a startup that needs money ASAP. But down the line when requirements change, maybe those layers of abstraction cost too much time to implement, especially if you're firefighting a dozen other poor choices.
They tried this system, and then realised they painted themselves into a corner. Businesses wants features they can sell to customers, and sometimes fixing technical debt is a luxury. Especially when technical debt hidden to customers.
Adding layers of abstraction when needed might sound like great advice to a startup that needs money ASAP. But down the line when requirements change, maybe those layers of abstraction cost too much time to implement, especially if you're firefighting a dozen other poor choices.
You have multiple different clients querying the same database?
Clients, services... the original developer didn't see anything wrong with it, since transactions exist and foreign key relationships make sure the data model stays consistent, right? And you can combine some frequently used methods accessing the database in some helper classes, and if some of the clients use those, that's basically your business logic!
It's a pretty impressive house of cards.
Refactoring (or rather re-designing) this sort of thing while still churning out new features is tricky to say the least.
It's a pretty impressive house of cards.
Refactoring (or rather re-designing) this sort of thing while still churning out new features is tricky to say the least.
The thinner your backend, the thicker your frontend. I much prefer writing in backend languages (e.g. python) than in frontend languages (e.g. javascript).
And as someone else pointed out, you can't enforce security on the frontend.
The line is getting blurrier between front and back these days with new frameworks, e.g. React. Personally, I'm not ready to move away from mature languages generating HTML on the backend, even though it means giving up some interactivity on the frontend.
And as someone else pointed out, you can't enforce security on the frontend.
The line is getting blurrier between front and back these days with new frameworks, e.g. React. Personally, I'm not ready to move away from mature languages generating HTML on the backend, even though it means giving up some interactivity on the frontend.
The main issue seems to be improper communication and process between the frontend/backend devs. The informal solution is to agree on something like a swagger doc. The proposed solution is to expose all the fields. I'd be careful on this as unwanted fields could be unecessarily exposed. A workaround addition to this is to make a blacklist filter of fields to not expose before sending and send everything else as a passthrough.
Best performing query is the query that's not executed.
You have discussed multiple ideas that fit for different use cases.
SQL is for transactional type of workload that's where it's best so data returned need to be less. In first part you are talking about not making any changes to the program when a new column is added. Now a days we are talking about column masking. So querying all columns is a no. Otherwise there needn't be a columnar DBs.
Seeing a generalized rest APIs(selecting many Columns than required) are also no but architects are recommending it nowadays. Here unnecessarily you are retreiving lots of data than required and filtering in frontend after retrieval (you have already done the heavy duty work).
Firebase is a special case, UX designer basically dictates the database schema rather than architects or dba. When you hit the node in firebase you retreive all the data in that node. Here there has to be duplication of data as it's designed based on UX no foreign keys or checks. To be efficient if UX changes you will have to change schema. If you are not changing UX but just adding more info or taking out things you can easily add more fields, no disruption.
What you do with firebase cant be doing with RDBMS, adding a column needs to be planned and also theres boyce codd normal form to avoid duplication and repeated groups. Again there is a reason why they had to build firestore when they have firebase.
Also one api call or three api calls again that depends on use case and performance testing. Three separate calls to get data from three separate tables or join table query again it depends in use case.
Don't do this, "Do a SELECT *, iterate over all the columns, and copy them to the JSON that will be sent back to the frontend."
As the data grows you will end up thrashing memory and unnecessary network data transfers.
Basically do work where there is lots of memory.
You have discussed multiple ideas that fit for different use cases.
SQL is for transactional type of workload that's where it's best so data returned need to be less. In first part you are talking about not making any changes to the program when a new column is added. Now a days we are talking about column masking. So querying all columns is a no. Otherwise there needn't be a columnar DBs.
Seeing a generalized rest APIs(selecting many Columns than required) are also no but architects are recommending it nowadays. Here unnecessarily you are retreiving lots of data than required and filtering in frontend after retrieval (you have already done the heavy duty work).
Firebase is a special case, UX designer basically dictates the database schema rather than architects or dba. When you hit the node in firebase you retreive all the data in that node. Here there has to be duplication of data as it's designed based on UX no foreign keys or checks. To be efficient if UX changes you will have to change schema. If you are not changing UX but just adding more info or taking out things you can easily add more fields, no disruption.
What you do with firebase cant be doing with RDBMS, adding a column needs to be planned and also theres boyce codd normal form to avoid duplication and repeated groups. Again there is a reason why they had to build firestore when they have firebase.
Also one api call or three api calls again that depends on use case and performance testing. Three separate calls to get data from three separate tables or join table query again it depends in use case.
Don't do this, "Do a SELECT *, iterate over all the columns, and copy them to the JSON that will be sent back to the frontend."
As the data grows you will end up thrashing memory and unnecessary network data transfers.
Basically do work where there is lots of memory.
The conclusion is a bit spotty, but there are some good points in there.
If you have dedicated backend developers and front-end developers — or worse, separate teams — then each and every change becomes painful. Writing bog standard UI and backend code shouldn't require in-depth specialist knowledge, and you can usually collapse those down into a single role.
I also agree that the frontend requirements should drive the shape of the API. If you have a backend team they tend to picture an idealized consumer of their API. They stress over whether the API follows RESTful principles and how to model things out correctly, and version breaking changes so their consumers can have their own update cycle. But when you step back and realise that the API has one, or maybe two, consumers. And it turns out writing some weird, esoteric, heavily coupled endpoint for a specific use case is not so horrible.
If you have dedicated backend developers and front-end developers — or worse, separate teams — then each and every change becomes painful. Writing bog standard UI and backend code shouldn't require in-depth specialist knowledge, and you can usually collapse those down into a single role.
I also agree that the frontend requirements should drive the shape of the API. If you have a backend team they tend to picture an idealized consumer of their API. They stress over whether the API follows RESTful principles and how to model things out correctly, and version breaking changes so their consumers can have their own update cycle. But when you step back and realise that the API has one, or maybe two, consumers. And it turns out writing some weird, esoteric, heavily coupled endpoint for a specific use case is not so horrible.
I'm seeing a lot of criticism along the lines of "the trade-offs are not worth it in <context>" or "the article didn't address every detail". Well, yeah -- no advice is universal, and this blog post isn't a 500 page reference book. I think it would be more productive to help the author articulate the trade-offs and forces pushing for and against this architecture.
FWIW, Reviewable (https://reviewable.io) follows roughly this design and, after 6 years, it's still a fine system to develop in. It runs on top of Firebase RTDB which takes care of the security rules; if anything, I feel better having all the security rules articulated in one central location (the schema) rather than spread throughout the backend. Communication between client and backend also flows through the database (with very few exceptions), which lets the backend get away with an easy to scale queue-based design.
There are downsides, of course. As many posters alluded to, changing the schema gets tricky. It basically must remain backwards compatible for at least a few releases so as not to inconvenience the users, and some kinds of changes are just not feasible. And yeah, how the client wants to fetch data does affect the schema as well, though in practice I haven't found the impact to be too bad. Also, Reviewable has only one (browser) client, so I don't need to worry about duplicating logic -- I probably wouldn't pick this architecture if multiple clients were a requirement. Putting much of the business logic in a client-side model layer integrated with the database structure actually works out really nicely, though I still struggle with how to efficiently share some of the logic with server-side code.
Overall, I'm satisfied with how things worked out and would definitely consider using a similar architecture again in the right circumstances. The article is right on the money that if you're just getting started on your MVP it's worth thinking about a thin- or no-backend approach.
FWIW, Reviewable (https://reviewable.io) follows roughly this design and, after 6 years, it's still a fine system to develop in. It runs on top of Firebase RTDB which takes care of the security rules; if anything, I feel better having all the security rules articulated in one central location (the schema) rather than spread throughout the backend. Communication between client and backend also flows through the database (with very few exceptions), which lets the backend get away with an easy to scale queue-based design.
There are downsides, of course. As many posters alluded to, changing the schema gets tricky. It basically must remain backwards compatible for at least a few releases so as not to inconvenience the users, and some kinds of changes are just not feasible. And yeah, how the client wants to fetch data does affect the schema as well, though in practice I haven't found the impact to be too bad. Also, Reviewable has only one (browser) client, so I don't need to worry about duplicating logic -- I probably wouldn't pick this architecture if multiple clients were a requirement. Putting much of the business logic in a client-side model layer integrated with the database structure actually works out really nicely, though I still struggle with how to efficiently share some of the logic with server-side code.
Overall, I'm satisfied with how things worked out and would definitely consider using a similar architecture again in the right circumstances. The article is right on the money that if you're just getting started on your MVP it's worth thinking about a thin- or no-backend approach.
Author here.
It's good to see a considered response, rather than the typical knee-jerk reactions common on Internet forums including HN.
It's good to see a considered response, rather than the typical knee-jerk reactions common on Internet forums including HN.
Sounds like a huge security risk waiting to happen.
It would not surprise me if the same kind of thinking led to Grindr returning password reset tokens in the API.
Absolutely terrible advice!
I think the problem this is trying to solve is a poorly defined interface from the backend. It's not to everyone's taste but openapi/swagger solves that problem. The hypothetical conversation around the missing property is resolved simply by specifying what the API returns - if it's not in the spec then it's not going to come back.
Alternatively, it speaks of a dysfunctional dynamic between front-end and back-end developers, but no amount of technical solution fixes that.
Alternatively, it speaks of a dysfunctional dynamic between front-end and back-end developers, but no amount of technical solution fixes that.
I'm not a big fan of GraphQL, but I think that the author would benefit from reading about and using it for awhile. It comes closest to the ideal implementation they're dreaming of, though hooking a GraphQL backend directly to the database would be a tremendously terrible idea.
No thank you.
> That goes for a lot of things — you don’t need to have automated tests as a general practice.
Man, this article is just handing out terrible advice left right and centre
Man, this article is just handing out terrible advice left right and centre
The only mention of security hand waves away the idea of access control to a "thin back end". That's the whole point of this post! The details of this are critical!
> you can define an API that takes an SQL query from the client, runs it, and returns the results. This query can be run under a database user account that has only read access, that too only to the friends table. This can accommodate unforeseen uses, like getting the number of friends rather than the actual list. Or if there’s a search box where the user can filter his friends, you can do a LIKE query. Or you can limit the number of friends returned to how many will be displayed in the UI.
Without mentioning how this has been solved elsewhere, this is an incredibly reckless piece of advice. For one, it's nearly impossible to parse and check an arbitrary SQL query for malicious intent. Besides ensuring that queries that should read only read, you also need to make sure that queries aren't designed to intentionally DoS your DB.
And moreover, it now passes the responsibility of query performance to the FE engineer. Are appropriate indexes in place? Does the query make inappropriate JOINs?
And on top of all of that, schema changes now mean that you need to update your front end code. This means guaranteed hard downtime, because you can't control what JS folks are running in the browser.
There's a lot wrong with the ideas presented here.