Golang web service framework with DI, Telemetry and more(github.com)
github.com
Golang web service framework with DI, Telemetry and more
https://github.com/go-masonry/mortar
29 comments
I always try to implement these things and then eventually just end up using Gin and some plugins and building something my self. This often leads to a smaller code footprint and a lower cognitive load when returning to the code 3-6 months later to add some feature.
Every time i tried gin i ended up just using the standard lib handlers and gorilla mux and some middlewares. Am i missing out on something? I've been doing it for years now
I wonder, is there a good even-handed critique out there of the dependency-injection pattern and related philosophies?
For me and probably a lot of people here, the pattern fundamentally offends my sensibilities; mainly, but not exclusively, because it forces you to put everything into classes. I'd much rather pass plain functions when I need inversion of control, and use closures if I really need to "configure" some piece of code. I'd also prefer to import and pass down those functions explicitly, instead of having values magically resolve in some runtime context that lives outside of the "real language". I do acknowledge that languages like Java don't, or at least didn't for a long time, support these other language features and so may have had no choice but to use DI as a crutch. But I've also seen it used in languages that do have those features, like JavaScript.
On the other hand, I'm not sure I've used the pattern enough to critique it fairly. And obviously it's been used for decades at many very large and successful companies. So what I'd really like is to hear from someone who's experienced with both worlds about the advantages and disadvantages, assuming that both exist.
For me and probably a lot of people here, the pattern fundamentally offends my sensibilities; mainly, but not exclusively, because it forces you to put everything into classes. I'd much rather pass plain functions when I need inversion of control, and use closures if I really need to "configure" some piece of code. I'd also prefer to import and pass down those functions explicitly, instead of having values magically resolve in some runtime context that lives outside of the "real language". I do acknowledge that languages like Java don't, or at least didn't for a long time, support these other language features and so may have had no choice but to use DI as a crutch. But I've also seen it used in languages that do have those features, like JavaScript.
On the other hand, I'm not sure I've used the pattern enough to critique it fairly. And obviously it's been used for decades at many very large and successful companies. So what I'd really like is to hear from someone who's experienced with both worlds about the advantages and disadvantages, assuming that both exist.
It’s not clear whether you dislike DI in general or specifically DI containers.
I think it’s an important point to clarify because DI on its own absolutely does not live outside of the real language and while it may be much more common to use classes than functions it’s not necessary.
I agree with your feelings in relation to DI containers however.
I think it’s an important point to clarify because DI on its own absolutely does not live outside of the real language and while it may be much more common to use classes than functions it’s not necessary.
I agree with your feelings in relation to DI containers however.
I wasn't actually aware of the distinction. I think you're right, that what I mainly dislike are DI containers. If DI just means "passing some value to something for it to use later" - which would even include currying, really - then yeah, I suppose I don't have a problem with DI/IoC themselves.
DI has nothing to do with the differences between classes/objects and functions/closures, DI and IoC are about reusable code and being able to switch out implementations. You can do IoC in either pattern.
In Java functions/closures and objects/classes are the same thing. You have the best of both worlds. You can for instance define a function that takes an argument a and a function b and pass it either a functional lambda or a class that implements said function:
In Java functions/closures and objects/classes are the same thing. You have the best of both worlds. You can for instance define a function that takes an argument a and a function b and pass it either a functional lambda or a class that implements said function:
static void doSomething(int a, Runnable b) {
if (a > 0) {
b.run();
}
}
doSomething(1, () -> System.out.println("hello"));
Runnable r = new SomeClassThatImplementsRunnable();
doSomething(0, r);IoC container is not DI and IoC container is a pattern with a lot of downsides. In Ruby, I use [the doctrine of useful objects](http://docs.eventide-project.org/user-guide/useful-objects.h...), which without a gem is very clear, no meta magic and it boils down to just DI with setters and getters. Simple and effective.
And it has the power of locality over an IoC container.
That being said, please remember that a class and a closure are essentially the same thing with a different syntax.
And it has the power of locality over an IoC container.
That being said, please remember that a class and a closure are essentially the same thing with a different syntax.
I posted this under another reply, but yes, I didn't actually realize the distinction of "containers" and how that's just one specialization of the pattern. Containers appear to be the main thing that bothers me, particularly when the container implementation requires that all injectable entities are classes and not functions. It creates this "viral" effect where nearly everything you implement ends up having to be wrapped up in a class, even when there's no reason to do so other than the injection container.
ok! Yeah the ioc pattern is not great, there are better alternatives out there without the need of having an object dependent on the entirety of the software.
I'm curious what you think the fundamental difference is between passing configuration through functions and closures, and passing it with classes. Classes are just fancy closures, after all.
More ceremony & semantic load of course and you end up with having to define single use abstractions so that you can mock them in tests
I've long been in the pro DI camp and used to think that interfaces and brittle tests were unavoidable..
Only after i recently got into FP with f#, I've come to appreciate how i used to conflate good IoC design with DI usage.. I'm pretty sure that I'm not alone
I've long been in the pro DI camp and used to think that interfaces and brittle tests were unavoidable..
Only after i recently got into FP with f#, I've come to appreciate how i used to conflate good IoC design with DI usage.. I'm pretty sure that I'm not alone
If you find yourself creating a lot of single-use abstractions, I think you're probably doing something wrong. In particular a lot of containers (I know Autofac) support Func<> very well.
As for ceremony and semantic load, I can't argue here, though I think following DI restricts some of the semantic load that comes with abstracting mainly through inheritance and through managing stateful dependencies.
As for ceremony and semantic load, I can't argue here, though I think following DI restricts some of the semantic load that comes with abstracting mainly through inheritance and through managing stateful dependencies.
I look at classes as holders of mutable state. While a closure can technically hold mutable state, subjectively I feel there's a convention that closure captures are often treated as constant (mainly by association with the functional paradigm).
One doesn't need to think of classes that way, though. A class could hold immutable references to immutable objects/functions/whatever (these are often the easiest classes of all to reason about).
Call me a weirdo, but I enjoy using Spring DI with Scala. It's quite easy to inject into a constructor like "class Foo(f: Int => String)" even if a little bit of "manual currying" is sometimes needed.
Call me a weirdo, but I enjoy using Spring DI with Scala. It's quite easy to inject into a constructor like "class Foo(f: Int => String)" even if a little bit of "manual currying" is sometimes needed.
Spring DI implies automatic DI containers, right? So how would that work with an abstract function type (as opposed to a specific class that gets instantiated)?
I actually like fairly explicit XML-based config with Spring. Something like "<constructor-arg value='#{thing.fnFor("x")}'/>" for example. Connects the pieces without relying on inference/magic.
Discussion a couple of days around .Net 5 release announcement ended up going into di vs no di..
I'm on mobile but you can get to it from my comment history
IME most in DI heavy C# code (e.g. Asp.Net Core) you're using classes but those classes are really just a set of named functions bundled together with their dependencies, not unlike a closure. It's not very common to see classes that are stateful.
I've never tried it but I'm fairly certain that most C# IOC containers should allow you to resolve functions if you really wanted to go that route.
I've never tried it but I'm fairly certain that most C# IOC containers should allow you to resolve functions if you really wanted to go that route.
I have my own take on this: https://github.com/jrockway/opinionated-server
Some things we do differently, and some things we do the same. We give each RPC handler a logger, so you know where logs are coming from (and the trace ID associated with every message). We collect detailed prometheus metrics for every RPC, and send everything we know to Jaeger. We provide clients for outgoing RPCs that log and trace in the same way as incoming requests.
We pick concrete types for logging/tracing/metrics, which makes it easier to figure out what code is actually running. Having used the abstraction layers, I find that when they don't do what you want, it can take a long time to figure out why. If you have to switch from Prometheus to Statsd or something, the performance and configuration characteristics of your app change, so you'll probably want to manually revisit every metric. It's a straightforward refactoring that you will probably never do more than once, so I don't think it's worth making it "easy" out of the box.
Finally, we let you configure your app with either flags or environment variables. Generally, I like flags for running things while developing (go run main.go --foo=bar) and environment variables for production. Why pick one or the other when you can do both!
Extensive care has been taken to be able to log everything, including all payloads for bidirectional streams. (HTTP logging is a bit of a disaster; check out the code you have to write to be able to hook every type of http.ResponseWriter without the consumer noticing: https://github.com/prometheus/client_golang/blob/master/prom....)
Some features that are missing or sub-optimal:
* A long chain of interceptors for every request is very expensive. They should all be condensed into one wrapper that does "everything" explicitly. I don't think it's worth the debugging complexity to be able to compose a bunch of middleware at runtime; just write a function that wraps every request that does the stuff you want. If you want more stuff, type it in right there. Easy to read, easy for the computer to execute.
* There should be fine-grained control over logging at runtime. I think you really want to be able to pass in configuration for every sub-logger, rather than a global log level. For example, you may want to adjust log sampling rate at runtime for a particular component, or upgrade a particular component from INFO to DEBUG for temporary investigation. (Without these controls, people stop writing log statements because they're too spammy, and then you have absolutely nothing to help you with ongoing issues.)
* There is absolutely no support for mTLS, which is a crime in this day and age. This is probably the next thing that I'll add; you should be able to watch a volume for changes in TLS key/cert/ca material (so you can rotate keys frequently and without restarting), and require that incoming requests have a certificate trusted by the loaded CA. Outgoing requests should also be marked as "external" or "internal", and use the correct client cert for internal requests. (I somewhat promise to add this soon, because it's sorely needed.)
* I don't think applications should support any non-production features for development, like a different logging configuration. opinionated-server apps take a --pretty_logs argument that configures it to print plain text instead of JSON, but I don't use it anymore; I just pipe the logs through jlog (`go install github.com/jrockway/json-logs/cmd/jlog`) and the results are much prettier than what zap would do itself. And it's the same tool I use every day to look at production logs, so there are no surprises.
Ultimately, I wrote this because I like small pieces of software that do one thing, and found myself writing the same boilerplate over and over. Overall, this has been a success. I have written small utilities (github.com/jrockway/alertmanager-status, github.com/jrockway/nodedns, github.com/jrockway/alertmanager-discord, which I forked from someone else and converted to use opinionated-server), and larger applications (github.com/jrockway/jsso2, github.com/jrockway/ekglue), and overall not having to write the same monitoring/logging code over and over has proven quite productive. I have not achieved my goal of "1 hour to make a production application" quite yet, but opinionated-server is certainly helping.
Some things we do differently, and some things we do the same. We give each RPC handler a logger, so you know where logs are coming from (and the trace ID associated with every message). We collect detailed prometheus metrics for every RPC, and send everything we know to Jaeger. We provide clients for outgoing RPCs that log and trace in the same way as incoming requests.
We pick concrete types for logging/tracing/metrics, which makes it easier to figure out what code is actually running. Having used the abstraction layers, I find that when they don't do what you want, it can take a long time to figure out why. If you have to switch from Prometheus to Statsd or something, the performance and configuration characteristics of your app change, so you'll probably want to manually revisit every metric. It's a straightforward refactoring that you will probably never do more than once, so I don't think it's worth making it "easy" out of the box.
Finally, we let you configure your app with either flags or environment variables. Generally, I like flags for running things while developing (go run main.go --foo=bar) and environment variables for production. Why pick one or the other when you can do both!
Extensive care has been taken to be able to log everything, including all payloads for bidirectional streams. (HTTP logging is a bit of a disaster; check out the code you have to write to be able to hook every type of http.ResponseWriter without the consumer noticing: https://github.com/prometheus/client_golang/blob/master/prom....)
Some features that are missing or sub-optimal:
* A long chain of interceptors for every request is very expensive. They should all be condensed into one wrapper that does "everything" explicitly. I don't think it's worth the debugging complexity to be able to compose a bunch of middleware at runtime; just write a function that wraps every request that does the stuff you want. If you want more stuff, type it in right there. Easy to read, easy for the computer to execute.
* There should be fine-grained control over logging at runtime. I think you really want to be able to pass in configuration for every sub-logger, rather than a global log level. For example, you may want to adjust log sampling rate at runtime for a particular component, or upgrade a particular component from INFO to DEBUG for temporary investigation. (Without these controls, people stop writing log statements because they're too spammy, and then you have absolutely nothing to help you with ongoing issues.)
* There is absolutely no support for mTLS, which is a crime in this day and age. This is probably the next thing that I'll add; you should be able to watch a volume for changes in TLS key/cert/ca material (so you can rotate keys frequently and without restarting), and require that incoming requests have a certificate trusted by the loaded CA. Outgoing requests should also be marked as "external" or "internal", and use the correct client cert for internal requests. (I somewhat promise to add this soon, because it's sorely needed.)
* I don't think applications should support any non-production features for development, like a different logging configuration. opinionated-server apps take a --pretty_logs argument that configures it to print plain text instead of JSON, but I don't use it anymore; I just pipe the logs through jlog (`go install github.com/jrockway/json-logs/cmd/jlog`) and the results are much prettier than what zap would do itself. And it's the same tool I use every day to look at production logs, so there are no surprises.
Ultimately, I wrote this because I like small pieces of software that do one thing, and found myself writing the same boilerplate over and over. Overall, this has been a success. I have written small utilities (github.com/jrockway/alertmanager-status, github.com/jrockway/nodedns, github.com/jrockway/alertmanager-discord, which I forked from someone else and converted to use opinionated-server), and larger applications (github.com/jrockway/jsso2, github.com/jrockway/ekglue), and overall not having to write the same monitoring/logging code over and over has proven quite productive. I have not achieved my goal of "1 hour to make a production application" quite yet, but opinionated-server is certainly helping.
I just needed to wrap ResponseWriter yesterday (for logging purposes) and found this package to be quite helpful: https://github.com/felixge/httpsnoop
That is a great find, I'm glad someone wrote that :)
Thank for such a long and very detailed comment. I'll try to explain our design goals.
I think the main difference is that mortar is heavily based on Dependency Injection (uber.Fx) and this affects a lot.
If you want you can have multiple instance of the Logger interface, just 'name' the instance and FX will help you inject the one you like. https://godoc.org/go.uber.org/fx#Annotated
This makes it's extremely easy to inject any instance anywhere you like.
What is not mentioned on the main page is the Builder approach we use https://github.com/go-masonry/mortar/blob/master/wiki/builde...
This allows to partially configure any concrete implementation ahead of time, say in some company-platform library and the developer will only need to "Build" it in the final project or adjust parameters if needed.
This project is actually a second version of an internal project we created at Here Mobility, eventually this design helped us to increase developer velocity by introducing - a unified project structure - devops preconfigured libraries to help integrate the services within our internal cloud setup - we had several groups and each had more than one products with backends so we needed to allow every group to develop the way they wanted, but at the same time maintain company standards and devops practices. It's not easy to make sure all your company services are aligned to the same versions unless you work on top of mono repo and we didn't. Mortar design goal was some what a compromise to allow customization but if not a good and reliable standard.
What is not mentioned on the main page is the Builder approach we use https://github.com/go-masonry/mortar/blob/master/wiki/builde...
This allows to partially configure any concrete implementation ahead of time, say in some company-platform library and the developer will only need to "Build" it in the final project or adjust parameters if needed.
This project is actually a second version of an internal project we created at Here Mobility, eventually this design helped us to increase developer velocity by introducing - a unified project structure - devops preconfigured libraries to help integrate the services within our internal cloud setup - we had several groups and each had more than one products with backends so we needed to allow every group to develop the way they wanted, but at the same time maintain company standards and devops practices. It's not easy to make sure all your company services are aligned to the same versions unless you work on top of mono repo and we didn't. Mortar design goal was some what a compromise to allow customization but if not a good and reliable standard.
Yeah, having that sort of framework is essential. It is enough work to add all the instrumentation (and ancillary concerns, like authorization/ACLs) that people will not do it if there isn't a framework for it, and of course the interesting bug will be in the component that wasn't instrumented. Easier to get it right once, and then everyone at your org gets it for free.
I haven't really found a way of doing dependency injection that I like. I did a lot of Java in a past life, and the dependency injection framework made it too easy to give unrelated components data they shouldn't have had access to, which ends up making testing (and debugging) a nightmare. (But, I don't miss all those functions that injected a clock instead of calling time.Now. Much easier to test!) Ultimately, I decided to just "manually" do it every time, which is tedious but easy to read. In the future, I may explore codegen on top of gRPC (to unpack things from the context, like the authenticated user, logger, etc. and pass those to handlers that take those as explicit parameters, so tests need less work to setup.)
As for monorepos, I am upset whenever I don't have one. For example, I hate that my open source projects are all in separate repositories -- it means there are n places to collect bug reports, n places to write CI rules, n places to write release scripts, etc. As a result, things get missed, and some things fall out of sync with whatever my current "best practices" are. (Sometimes I fear that we have created a new class of software engineer to manage all these concerns, when the computer could be doing it for us with just a little extra tooling.) The actual code is easy to share, but all those ancillary concerns are a huge time sink.
I haven't really found a way of doing dependency injection that I like. I did a lot of Java in a past life, and the dependency injection framework made it too easy to give unrelated components data they shouldn't have had access to, which ends up making testing (and debugging) a nightmare. (But, I don't miss all those functions that injected a clock instead of calling time.Now. Much easier to test!) Ultimately, I decided to just "manually" do it every time, which is tedious but easy to read. In the future, I may explore codegen on top of gRPC (to unpack things from the context, like the authenticated user, logger, etc. and pass those to handlers that take those as explicit parameters, so tests need less work to setup.)
As for monorepos, I am upset whenever I don't have one. For example, I hate that my open source projects are all in separate repositories -- it means there are n places to collect bug reports, n places to write CI rules, n places to write release scripts, etc. As a result, things get missed, and some things fall out of sync with whatever my current "best practices" are. (Sometimes I fear that we have created a new class of software engineer to manage all these concerns, when the computer could be doing it for us with just a little extra tooling.) The actual code is easy to share, but all those ancillary concerns are a huge time sink.
burnthrow(1)