Layered Design in Go(jerf.org)
jerf.org
Layered Design in Go
https://jerf.org/iri/post/2025/go_layered_design/
40 コメント
In the abstract, I think I agree with you. But in reality, what I see is that go projects use far fewer packages than, say, programs in Java. Many go projects use one or two omnibus packages--principally, I expect, to avoid having to worry about circularity issues.
By forcing this design pattern on developers (something no other language does), I think the result has been overall worse code rather than better.
Perhaps a warning, rather than stop-the-compiler error would have been a better choice. Not sure.
Either way though, I wholly agree that the compiler gives too little information, which is curious because it knows the needed data and should easily be able to present it in a useful way.
By forcing this design pattern on developers (something no other language does), I think the result has been overall worse code rather than better.
Perhaps a warning, rather than stop-the-compiler error would have been a better choice. Not sure.
Either way though, I wholly agree that the compiler gives too little information, which is curious because it knows the needed data and should easily be able to present it in a useful way.
I see the opposite in most Java programs: poor organization with packages based on a type of thing (eg models, controllers) rather than related behaviors.
Go doesn’t have warnings, which is great - if something is worth warning about, it is also worth erroring about. It never ceases to amaze me when a brand new JavaScript project spits out dozens of errors after pulling in a common library, and everyone thinks that is ok.
Go doesn’t have warnings, which is great - if something is worth warning about, it is also worth erroring about. It never ceases to amaze me when a brand new JavaScript project spits out dozens of errors after pulling in a common library, and everyone thinks that is ok.
Of course Go has warnings.
It has go vet, which is not a linter, and according to the authors doesn't need comments to ignore checks, because the checks are always correct about you having written shitty code. Except where it might warn you about something completely outside your control.
It has go vet, which is not a linter, and according to the authors doesn't need comments to ignore checks, because the checks are always correct about you having written shitty code. Except where it might warn you about something completely outside your control.
Well then why not just integrate it into the compiler and make them errors?
> If you get a circular dependency something is wrong with your design
Packages not being able import from each other circularly is purely a compiler limitation. It says nothing about the realities of software development.
This notion stems from the idea that software design is inherently hierarchical, and that there is always a clear "higher level" and "lower level" between every possible software module.
What I've found in practice is that this is a fictional concept. Circularity between modules is very common and natural (especially as business requirements change over time). The workarounds people invent to avoid circularity literally always result in a codebase that is harder to understand and maintain, rather than easier.
> It forces you to separate your concerns properly.
Nah. It's not separation of concerns, it's separation of implementation. Two functions that in every other way shape and form deal directly with the same concepts, end up needing to be in separate modules purely because they differ in the functionality they import. And if later their imports change, they may need to be moved again. Which means implementation details are leaking into your design, which makes code less discoverable (since you now need to know implementation details in order to reasonably predict where a given function might be defined).
Packages not being able import from each other circularly is purely a compiler limitation. It says nothing about the realities of software development.
This notion stems from the idea that software design is inherently hierarchical, and that there is always a clear "higher level" and "lower level" between every possible software module.
What I've found in practice is that this is a fictional concept. Circularity between modules is very common and natural (especially as business requirements change over time). The workarounds people invent to avoid circularity literally always result in a codebase that is harder to understand and maintain, rather than easier.
> It forces you to separate your concerns properly.
Nah. It's not separation of concerns, it's separation of implementation. Two functions that in every other way shape and form deal directly with the same concepts, end up needing to be in separate modules purely because they differ in the functionality they import. And if later their imports change, they may need to be moved again. Which means implementation details are leaking into your design, which makes code less discoverable (since you now need to know implementation details in order to reasonably predict where a given function might be defined).
> Circularity between modules is very common and natural
In theory, but in practice it isn't because there are very few language that do not see software design as being hierarchical. That stems primarily from most languages being based on a hierarchical filesystem, which imposes a hierarchical view of the world at the very core. Circular references in languages that are hierarchical end up being very awkward.
There are a small handful of languages that reject all things hierarchical, including the filesystem, but they are few and far between and most probably have never heard of them and they certainly aren't what you are going to find in production. For better or worse, we've settled on a hierarchical model.
In theory, but in practice it isn't because there are very few language that do not see software design as being hierarchical. That stems primarily from most languages being based on a hierarchical filesystem, which imposes a hierarchical view of the world at the very core. Circular references in languages that are hierarchical end up being very awkward.
There are a small handful of languages that reject all things hierarchical, including the filesystem, but they are few and far between and most probably have never heard of them and they certainly aren't what you are going to find in production. For better or worse, we've settled on a hierarchical model.
> in practice it isn't because there are very few language that do not see software design as being hierarchical
It has been possible since the days of C (via forward declarations / header files) for two compilation units to call functionality in each other circularly. Java and many other languages have followed suit. I don't buy the argument that it is some sort of new or esoteric thing for a compiler to allow this.
It has been possible since the days of C (via forward declarations / header files) for two compilation units to call functionality in each other circularly. Java and many other languages have followed suit. I don't buy the argument that it is some sort of new or esoteric thing for a compiler to allow this.
[deleted]
Is there a reason you decided to reply after only reading the first sentence? This off-topic straw man you have imaged doesn't exist.
You're going to have to elaborate in words what your actual problem with my comment is. It's not clear to me why you believe I only read the first sentence, nor in what way I am addressing a strawman. It is perhaps you who have misunderstood my point, rather than the other way around. But, again, you've provided not enough details for me to ascertain this.
Your point is understood, and nobody would disagree with it, but your point is towards a straw man. Nobody ever in the history of computing has made this "argument" you have imagined. If you honestly believe that you didn't make it up arbitrarily, where did you get it from?
The argument I mentioned:
> I don't buy the argument that it is some sort of new or esoteric thing for a compiler to allow this
Where I got it from:
> there are very few language that do not see software design as being hierarchical
> Circular references in languages that are hierarchical end up being very awkward.
> There are a small handful of languages that reject all things hierarchical, including the filesystem, but they are few and far between and most probably have never heard of them and they certainly aren't what you are going to find in production
My point being that it is not "awkward" at all for a language to allow circular references between modules. It is only awkward in languages that make it awkward (like Go does). In C (a language over 50 years old) and Java (one of the most widely used languages in the world) it's the most straightforward and natural thing in the world. So it's certainly not correct to say that it is "awkward" in all languages except ones nobody has ever heard of (aka "esoteric", the term I used in my comment).
So yeah I'm still not fully following how I failed to address your comment, or attacked a strawman. Based on what you're saying now, it's clear that you feel I have misunderstood the point you were trying to make. But I can say that it certainly was not based on a single sentence, or a lack of reading on my part. Probably just a confusion of terminology.
So, what did you actually mean?
> I don't buy the argument that it is some sort of new or esoteric thing for a compiler to allow this
Where I got it from:
> there are very few language that do not see software design as being hierarchical
> Circular references in languages that are hierarchical end up being very awkward.
> There are a small handful of languages that reject all things hierarchical, including the filesystem, but they are few and far between and most probably have never heard of them and they certainly aren't what you are going to find in production
My point being that it is not "awkward" at all for a language to allow circular references between modules. It is only awkward in languages that make it awkward (like Go does). In C (a language over 50 years old) and Java (one of the most widely used languages in the world) it's the most straightforward and natural thing in the world. So it's certainly not correct to say that it is "awkward" in all languages except ones nobody has ever heard of (aka "esoteric", the term I used in my comment).
So yeah I'm still not fully following how I failed to address your comment, or attacked a strawman. Based on what you're saying now, it's clear that you feel I have misunderstood the point you were trying to make. But I can say that it certainly was not based on a single sentence, or a lack of reading on my part. Probably just a confusion of terminology.
So, what did you actually mean?
> Where I got it from:
Which must mean that you only read the first sentence and dreamt up the rest, or didn't read it at all, else you'd know there was nothing said about circular imports being esoteric. In fact, the comment you originally replied to wouldn't work if they were esoteric.
> It is only awkward in languages that make it awkward (like Go does).
How could it be awkward in Go? It doesn't support circular references. It cannot be awkward – it isn't possible at all! This was already told in earlier comments, so how did you manage to get here not knowing that other than by not reading the comments?
Which must mean that you only read the first sentence and dreamt up the rest, or didn't read it at all, else you'd know there was nothing said about circular imports being esoteric. In fact, the comment you originally replied to wouldn't work if they were esoteric.
> It is only awkward in languages that make it awkward (like Go does).
How could it be awkward in Go? It doesn't support circular references. It cannot be awkward – it isn't possible at all! This was already told in earlier comments, so how did you manage to get here not knowing that other than by not reading the comments?
> there was nothing said about circular imports being esoteric
If you aren't saying that circular imports are not common and natural, then don't reply to the statement "circularity between modules is very common and natural" with the statement "In theory, but in practice it isn't"
I have no quarrel with you. If there has been a misunderstanding, then simply correct it and elaborate. That could have been a 2-second interaction - I would have replied "oh, my mistake, I misunderstood what you were saying" and this conversation would be over.
On the other hand, if your goal is just to look smart or superior, or make me look dumb because you believe I'm incapable of reading, then feel free. I have better things to do with my time.
I won't be replying further.
> How could it be awkward in Go? It doesn't support circular references. It cannot be awkward – it isn't possible at all!
There are workarounds. That was the whole point of the beginning of this entire comment chain - me pointing out that these workarounds are worse quality code than if the compiler just supported doing things in a more straightforward way.
If you aren't saying that circular imports are not common and natural, then don't reply to the statement "circularity between modules is very common and natural" with the statement "In theory, but in practice it isn't"
I have no quarrel with you. If there has been a misunderstanding, then simply correct it and elaborate. That could have been a 2-second interaction - I would have replied "oh, my mistake, I misunderstood what you were saying" and this conversation would be over.
On the other hand, if your goal is just to look smart or superior, or make me look dumb because you believe I'm incapable of reading, then feel free. I have better things to do with my time.
I won't be replying further.
> How could it be awkward in Go? It doesn't support circular references. It cannot be awkward – it isn't possible at all!
There are workarounds. That was the whole point of the beginning of this entire comment chain - me pointing out that these workarounds are worse quality code than if the compiler just supported doing things in a more straightforward way.
> If you aren't saying that circular imports are not common and natural...
Like said, in theory they are natural. In practice they usually aren't, because the languages people use are usually designed around the idea of being hierarchical. That is true of C, and especially true of Java which doubles down on the concept. As before – the part you seemed to not read – circular imports in those languages is possible, but awkward due to their hierarchical view of the world.
> There are workarounds.
You're right that you can achieve a similar effect by hacking up the use of go:linkname, but you're completely bypassing the import system to do that. I don't think that is reasonably considered circular imports. Besides, that approach is not really awkward. That approach is foolish, if not plain stupid.
Like said, in theory they are natural. In practice they usually aren't, because the languages people use are usually designed around the idea of being hierarchical. That is true of C, and especially true of Java which doubles down on the concept. As before – the part you seemed to not read – circular imports in those languages is possible, but awkward due to their hierarchical view of the world.
> There are workarounds.
You're right that you can achieve a similar effect by hacking up the use of go:linkname, but you're completely bypassing the import system to do that. I don't think that is reasonably considered circular imports. Besides, that approach is not really awkward. That approach is foolish, if not plain stupid.
I prefer extremely fast compile times.
Other languages (Pascal, Ocaml, maybe Zig) have proven that it's possible to implement a very fast compiler that emits efficient machine code while not dumbing down the language to complete brain death.
Add Common Lisp to the list. And even Scala has fast compile times today.
It's not fast if it fails every time you commented a line because a variable becomes unused though.
[The following is intended as language-agnostic.]
It’s useful to distinguish between interface and implementation dependencies. I agree that there shouldn’t be circular interface dependencies between modules. The absence of circular interface dependencies allows separate compilation of modules. It also means that at least in principle, the implementations can be made non-circular (can be refactored to non-circular without breaking any of the existing interfaces). But it’s often okay for the implementation of A to depend on the interface of B, and at the same time the implementation of B to depend on the interface of A, as long as there is no mutual dependency between the interfaces of A and B.
It’s useful to distinguish between interface and implementation dependencies. I agree that there shouldn’t be circular interface dependencies between modules. The absence of circular interface dependencies allows separate compilation of modules. It also means that at least in principle, the implementations can be made non-circular (can be refactored to non-circular without breaking any of the existing interfaces). But it’s often okay for the implementation of A to depend on the interface of B, and at the same time the implementation of B to depend on the interface of A, as long as there is no mutual dependency between the interfaces of A and B.
> In my opinion, not allowing circular dependencies is a great design choice for building large programs.
I have a hobby project with maybe 20 packages involved. Circular dependencies were getting harder and harder to solve.
What worked in this situation was to separate the whole hairball into two layers. The packages in the app layer could import packages from the utilities layer, but not vice-versa.
This introduced enough structure to simplify the removal of circularities and prevent new outbreaks.
I have a hobby project with maybe 20 packages involved. Circular dependencies were getting harder and harder to solve.
What worked in this situation was to separate the whole hairball into two layers. The packages in the app layer could import packages from the utilities layer, but not vice-versa.
This introduced enough structure to simplify the removal of circularities and prevent new outbreaks.
But, don't you know you can't have a utils package either?
A package must implement one functionality, and it must be clear from the short import name what that functionality is.
A package is also the only way to enforce visibility (and often, with that, other useful properties like immutability).
So packages can't be too big, or too generic, but they can't also be too small, or too specific.
I also have 2 (or 3?) layers, and a utils package, and the packages are huge, and yet I need various cheats to allow cyclic dependencies, and have some repetition with other tricks to avoid some of it.
It's the real world, and guess what, the very well designed standard library does… all of the above too.
A package must implement one functionality, and it must be clear from the short import name what that functionality is.
A package is also the only way to enforce visibility (and often, with that, other useful properties like immutability).
So packages can't be too big, or too generic, but they can't also be too small, or too specific.
I also have 2 (or 3?) layers, and a utils package, and the packages are huge, and yet I need various cheats to allow cyclic dependencies, and have some repetition with other tricks to avoid some of it.
It's the real world, and guess what, the very well designed standard library does… all of the above too.
> But, don't you know you can't have a utils package either?
Oh, I know. What I have is over half a dozen such packages named "*utils".
Oh, I know. What I have is over half a dozen such packages named "*utils".
What do you mean by "can't"?
"Can not" means "not possible", but it's clearly possible.
"Can not" means "not possible", but it's clearly possible.
I obviously meant "can't" not RFC 2119 "MUST NOT".
I could list more sources, but I'll just leave you with these:
https://go.dev/blog/package-names
https://google.github.io/styleguide/go/best-practices#util-p...
https://dave.cheney.net/2019/01/08/avoid-package-names-like-...
The Google style guide is worth a read through, as the subsequent point recognizes the difficulties around package size. It can make sense to have a package with a single function, and it can almost make sense to stuff your entire program/library in a single package.
The two rules work that exacerbate this tension are precisely visibility and no cyclic dependencies.
The perfect is the enemy of the good. Forbidding cyclic dependencies is one such case, IMO.
I could list more sources, but I'll just leave you with these:
https://go.dev/blog/package-names
https://google.github.io/styleguide/go/best-practices#util-p...
https://dave.cheney.net/2019/01/08/avoid-package-names-like-...
The Google style guide is worth a read through, as the subsequent point recognizes the difficulties around package size. It can make sense to have a package with a single function, and it can almost make sense to stuff your entire program/library in a single package.
The two rules work that exacerbate this tension are precisely visibility and no cyclic dependencies.
The perfect is the enemy of the good. Forbidding cyclic dependencies is one such case, IMO.
great blog post! also this website has a lot of incredible posts, if you like learning about functional programming, you should check out
https://jerf.org/iri/blogbooks/functional-programming-lesson...
https://jerf.org/iri/blogbooks/functional-programming-lesson...
One bonus technique related to the “move to a third package” advice: generating many of your model structures (SQL, Protobuf, graphql, etc) allows you to set up obvious directionality between generated layers and to provide all generated code as “base packages” to your application code, which then composes everything together.
Prior to this technique we often had “models importing models circularly” as an issue but that’s entirely disappeared due to the introduction of the structural additional layer.
Prior to this technique we often had “models importing models circularly” as an issue but that’s entirely disappeared due to the introduction of the structural additional layer.
Looks like I am reading a book about Yourdon structured method.
Since I meant this just as a "how I do it" post I suppose I forgot the disclaimer that I'm not particularly claiming to have invented anything or to be the first. Indeed to a large degree I consider myself just to be following the grain of Go and hardly doing anything myself.
That said, after some quick googling around, I don't think I feel bad not knowing what Yourdon design is, as it seems to be somewhat proprietary and behind paywalls, so it's hard for me to tell if there's much similarity. Certainly it has a lot of stuff I tend to eschew; lots of references to diagrams and state charts and such. I tend to prefer a more "agile but wait before you panic I mean 'original' agile not 'scrum' or whatever other abomination it was turned into", my formal method is more based around exploring the design space with extensive unit tests and code rather than that sort of up-front design.
That said, after some quick googling around, I don't think I feel bad not knowing what Yourdon design is, as it seems to be somewhat proprietary and behind paywalls, so it's hard for me to tell if there's much similarity. Certainly it has a lot of stuff I tend to eschew; lots of references to diagrams and state charts and such. I tend to prefer a more "agile but wait before you panic I mean 'original' agile not 'scrum' or whatever other abomination it was turned into", my formal method is more based around exploring the design space with extensive unit tests and code rather than that sort of up-front design.
It was a common way to structure enterprise C code during the 1990's, and the snarky remark is how the anti-enterprise culture from Go ends up adopting the same big corporation principles, given enough wind behind its sails.
Yourdon is the big wave of enterprise methodologies immediately predating the OOP wave with Booch, UML, GoF and friends.
I can gladly bet there are some Go pattern books around the corner as well.
As for the book paywall, it is certainly available in many libraries, given its age.
Yourdon is the big wave of enterprise methodologies immediately predating the OOP wave with Booch, UML, GoF and friends.
I can gladly bet there are some Go pattern books around the corner as well.
As for the book paywall, it is certainly available in many libraries, given its age.
Nobody has accused my code bases of being "too enterprise" yet.
Edit: I should probably elaborate on that before my edit window closes. Other than pervasive use of dependency injection, done directly with no framework simply by passing values around, there are effectively no "Enterprise" structures in sight in my code base. That's what I mean by "this design is sufficient for me". The only thing that resembles a "factory" is in the precise place I need to construct values from a type specified by an input string. No patterns put in place "just in case". No top-level frameworks used for 3% of their functionality. I use a process monitor but the interface that requires is "Serve(context.Context)", which is just the minimum you need to be able to monitor a service.
There are what I'd call "patterns", but they're there to do their job to the full, not guesses about what maybe I'll need later.
I've actually got a half-written pattern book for Go I've been trying to figure out what to do with, but the introduction is basically "why pattern books shouldn't just be a recitation of the original GoF patterns", because that is, well, stupid. Even the original book bit off too much trying to straddle Smalltalk and C++ in one shot. They require different patterns. My pattern book is how to solve Go problems in Go, not to give people words to slap in their code in case someday their code might grow enough to need it.
Edit: I should probably elaborate on that before my edit window closes. Other than pervasive use of dependency injection, done directly with no framework simply by passing values around, there are effectively no "Enterprise" structures in sight in my code base. That's what I mean by "this design is sufficient for me". The only thing that resembles a "factory" is in the precise place I need to construct values from a type specified by an input string. No patterns put in place "just in case". No top-level frameworks used for 3% of their functionality. I use a process monitor but the interface that requires is "Serve(context.Context)", which is just the minimum you need to be able to monitor a service.
There are what I'd call "patterns", but they're there to do their job to the full, not guesses about what maybe I'll need later.
I've actually got a half-written pattern book for Go I've been trying to figure out what to do with, but the introduction is basically "why pattern books shouldn't just be a recitation of the original GoF patterns", because that is, well, stupid. Even the original book bit off too much trying to straddle Smalltalk and C++ in one shot. They require different patterns. My pattern book is how to solve Go problems in Go, not to give people words to slap in their code in case someday their code might grow enough to need it.
> Packages may not circularly reference each other.
Actually possible with go:linkname.
Actually possible with go:linkname.
Fair, but I think we can classify that under "unsafe" and ignore it under normal circumstances. I can also say things like "Go doesn't have pointer arithmetic" with a straight face, even though unsafe permits pointer arithmetic just fine. If you're programming with that routinely, you're out of the bounds of my advice for architecture anyhow. Whether for good or bad reasons would be left as an exercise for the architect in question.
Looking at the diagram for the SQLite VFS page I didn't think I was going overboard with designing my driver around 3 packages: https://sqlite.org/vfs.html
One layer wraps SQLite C API, below it lives a pure Go VFS, and above all that database/sql driver.
Even this coarse split (with a collection internal packages, the bigger one of them “utils”) is enough to need various band-aids to accommodate the impossibility of circular dependencies.
I honestly don't think it helps much.
At the module level, there are obvious benefits from the impossibility. Just like all the pain around v2 modules can be justified, even if I find it annoying.
When packages are also the only layer at which you can enforce visibility, it becomes worse.
One layer wraps SQLite C API, below it lives a pure Go VFS, and above all that database/sql driver.
Even this coarse split (with a collection internal packages, the bigger one of them “utils”) is enough to need various band-aids to accommodate the impossibility of circular dependencies.
I honestly don't think it helps much.
At the module level, there are obvious benefits from the impossibility. Just like all the pain around v2 modules can be justified, even if I find it annoying.
When packages are also the only layer at which you can enforce visibility, it becomes worse.
See the last case of how to handle circular dependencies.
Ports are special cases. Always are.
Ports are special cases. Always are.
Kind of reminds me of the concept of spheres in randomizers
A funny quirk about golang is you cannot have circular dependencies at the package level, but you can have circular dependencies in go.mod
The tl;dr is don't do that either.
The tl;dr is don't do that either.
go.mod files have no dependencies, so neither they can have circular dependencies.
go.mod files just list the dependencies of their respective package.
go.mod files just list the dependencies of their respective package.
Cool description of how jerf thinks about packages and how he deals with circular dependencies!
If you get a circular dependency something is wrong with your design and the article does a good job on how to fix them.
I sometimes use function pointers which other packages override to fix circular dependencies which I don't think was mentioned in the article.
My only wish is that the go compiler gave more helpful output when you make a circular dependency. Currently it gives a list of all the packages involved in the loop which can be quite long, though generally it is the last thing you changed which caused the problem.