Instrumentation in Go(gbws.io)
gbws.io
Instrumentation in Go
https://gbws.io/articles/instrumentation-in-go/
49 comments
I think all mentioned issues are related to the implementation of the user probes, not the pattern.
You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.
You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.
The thing I keep in mind is that state is mutated far more than it is observed. You might handle 1000 or more requests per second and only export the number of requests once per second or less. In light of this ratio it’s important to make the recording path as simple and cheap as possible, and it’s ok if the observing path has to be complicated to compensate.
I usually refuse to use atomic increments for services because it scales very poorly. Even a mutex-protected increment scales much better than atomic increment.
I usually refuse to use atomic increments for services because it scales very poorly. Even a mutex-protected increment scales much better than atomic increment.
I see your point that recording must be much cheaper than exporting, and I completely agree with it.
Anyway how will you collect that counters inside your component (to be observed later on demand)?
Anyway how will you collect that counters inside your component (to be observed later on demand)?
"Logging is a feature", as the authors of Growing Object-Oriented Software Guided by Tests state. As such, we need to treat them as we would treat any other feature. One relatively straightforward way of adding logs is by decorating your objects. Logging decorators.
For example, if `Client` is defined as an interface, we can have two concrete implementations. One that holds business logic – `NetClient` – and one that holds logging/developer logic – `VerboseClient`. As we inject dependencies, we can compose a client like this: `NewVerboseClient(NewClient())`. This way you may even test you logging logic by composing it with different implementations of clients (one that always fails, for example) to test the different kinds of logs.
For example, if `Client` is defined as an interface, we can have two concrete implementations. One that holds business logic – `NetClient` – and one that holds logging/developer logic – `VerboseClient`. As we inject dependencies, we can compose a client like this: `NewVerboseClient(NewClient())`. This way you may even test you logging logic by composing it with different implementations of clients (one that always fails, for example) to test the different kinds of logs.
Thats true that such approach also exists.
But it doesn't work well in Go by several reasons.
For example, you usually define interface not in the package where its implemented, but where its needed. Thus you must provide such wrappers in every place where some (subset) of the `Client`'s methods are used.
Another reason is that such thing rejects an ability to use your struct as is, without interfaces (and thus sometimes without heap allocation for your struct).
And, finally, if you want to adopt such approach in proprietary/internal software, where interfaces usually change more often than in libraries, you will change code in N places instead of 1 (in best case); where N is number of instrumentation methods you want to provide "as feature".
But it doesn't work well in Go by several reasons.
For example, you usually define interface not in the package where its implemented, but where its needed. Thus you must provide such wrappers in every place where some (subset) of the `Client`'s methods are used.
Another reason is that such thing rejects an ability to use your struct as is, without interfaces (and thus sometimes without heap allocation for your struct).
And, finally, if you want to adopt such approach in proprietary/internal software, where interfaces usually change more often than in libraries, you will change code in N places instead of 1 (in best case); where N is number of instrumentation methods you want to provide "as feature".
Doing this is going to make navigating your code harder for little gain. It’s ok to put logging next to business logic. If you do this and actually follow through with it you can end up with a dozen wrapping classes with only one business logic class. I’ve seen this at my work. It makes debugging/finding/mutating the business class harder because you have to figure out which class you actually care about and how the classes were organized when they were constructed. Just make 1 class. It’ll save you plenty of code and be easier to understand/read.
Why do Go articles so often have no syntax highlighting?
> Why do Go articles so often have no syntax highlighting?
Rob Pike, one of the 3 original Go architects, has stated in the past he doesn’t like syntax highlighting [1].
> Syntax highlighting is juvenile. When I was a child, I was taught arithmetic using colored rods (http://en.wikipedia.org/wiki/Cuisenaire_rods). I grew up and today I use monochromatic numerals. — Rob Pike
Maybe Sergey Kamardin, the author of this article, is following Rob Pike’s philosophy. That being said, the website has references to a Go library called Chroma [2] which purpose is to colorize source code. The library borrows ideas from a popular JavaScript library called “highlight.js” [3] and that is why you can also see “highlight” CSS classes. The website downloads this CSS file [4] which contains a couple of instructions to make the code look the way it looks.
The official Go Blog has popularized this type of design [5] and many Go programmers has adopted it in their own websites.
[1] https://groups.google.com/d/msg/golang-nuts/hJHCAaiL0so/kG3B...
[2] https://github.com/alecthomas/chroma
[3] https://highlightjs.org/
[4] https://gbws.io/scss/syntax.min.css
[5] https://blog.golang.org/go-fonts
Rob Pike, one of the 3 original Go architects, has stated in the past he doesn’t like syntax highlighting [1].
> Syntax highlighting is juvenile. When I was a child, I was taught arithmetic using colored rods (http://en.wikipedia.org/wiki/Cuisenaire_rods). I grew up and today I use monochromatic numerals. — Rob Pike
Maybe Sergey Kamardin, the author of this article, is following Rob Pike’s philosophy. That being said, the website has references to a Go library called Chroma [2] which purpose is to colorize source code. The library borrows ideas from a popular JavaScript library called “highlight.js” [3] and that is why you can also see “highlight” CSS classes. The website downloads this CSS file [4] which contains a couple of instructions to make the code look the way it looks.
The official Go Blog has popularized this type of design [5] and many Go programmers has adopted it in their own websites.
[1] https://groups.google.com/d/msg/golang-nuts/hJHCAaiL0so/kG3B...
[2] https://github.com/alecthomas/chroma
[3] https://highlightjs.org/
[4] https://gbws.io/scss/syntax.min.css
[5] https://blog.golang.org/go-fonts
That is very bizarre
Go is very opinionated in a lot of bizarre ways.
I would call it juvenile...
Well, he has a point. If you need a variable to be colored to know it's a variable, then I'd argue that you don't really know what you're doing.
If you don't need a variable to be colored to know it's a variable, then why do you color them?
A counter argument would be that you have this huge visual cortex in your brain which can use cues like this to help you quickly find what you're looking for.
A counter argument to that would be that you should know your code well enough to not need to rely on anything like that.
When I started programming, I had notepad and that was it. I didn't need colors. I use colors now, though. What's that say? I have no idea.
If you don't need a variable to be colored to know it's a variable, then why do you color them?
A counter argument would be that you have this huge visual cortex in your brain which can use cues like this to help you quickly find what you're looking for.
A counter argument to that would be that you should know your code well enough to not need to rely on anything like that.
When I started programming, I had notepad and that was it. I didn't need colors. I use colors now, though. What's that say? I have no idea.
> A counter argument to that would be that you should know your code well enough to not need to rely on anything like that.
that's not a counter argument to that. nobody in the world no matter how brilliant knows giant complicated codebases inside out. and the best way to parse those is to use colors to aid your brain in differentiation. otherwise its all one garbled mess.
that's not a counter argument to that. nobody in the world no matter how brilliant knows giant complicated codebases inside out. and the best way to parse those is to use colors to aid your brain in differentiation. otherwise its all one garbled mess.
[deleted]
Not everything you do is about need. I think highlighting helps reduce some of the cognitive load involved in mentally parsing a code block. Especially in a language you might be unfamiliar with. I also happen to like the way it looks.
[deleted]
FYI Chroma is based on Pygments, not highlightjs.
> Syntax highlighting is juvenile. When I was a child, I was taught arithmetic using colored rods (http://en.wikipedia.org/wiki/Cuisenaire_rods). I grew up and today I use monochromatic numerals. — Rob Pike
When I was a child I had a plush gopher.
When I was a child I had a plush gopher.
Go philosophy is to optimize for the brain, to lower cognitive load.
This statement is in absurd opposition to this philosophy.
No need for benchmarcks to know that a developper is less productive and more error prone without semantic highlighting.
No need for benchmarcks to know that a developper is less productive and more error prone without semantic highlighting.
Yeah everyone "knows" this, but where's the proof? I've never seen any proof that syntax highlighting helps someone who knows their code well. Things that are supposed to be common knowledge like this often turn out to be false when someone eventually decides to challenge their assumptions.
I programmed for a full decade with Notepad alone. I don't think colored keywords are important, though I use syntax highlighting today.
I programmed for a full decade with Notepad alone. I don't think colored keywords are important, though I use syntax highlighting today.
There are probably tasks you can perform quicker when you're accustomed to syntax coloring, since the brain tends to capitalize on association with easily detected features. Finding the start of a function could be such as task, but the gain would be low (a few 100s of milliseconds, I'd guess).
At the same time, it might slow down other tasks, which can be harder to identify. I was just changing some html template, and I made a dumb "extra quote"" error because of highlighting. It probably took me more than a minute, which would offset a great number of small speedups.
I guess the conclusive proof won't be easy to find.
At the same time, it might slow down other tasks, which can be harder to identify. I was just changing some html template, and I made a dumb "extra quote"" error because of highlighting. It probably took me more than a minute, which would offset a great number of small speedups.
I guess the conclusive proof won't be easy to find.
Why would I know whatever code I'm reading well? Most of the time I'm reading code it's unfamiliar to me, like for instance library code, picking up a project after a long break, or in this case on a blog.
Article has very bland design, making it hard to read the embedded code. Comments are light grey on white, not easy to read on some screens.
The Go Tour has no syntax highlighting, but the design on the pages are fit for the purpose of learning the language: https://tour.golang.org/welcome/1
The typical Go-presentations (http://go-talks.appspot.com/github.com/SatishTalim/slides/sa...) tend to be very short and readable, but you'll find www-pages/videos about Go having both syntax highlighting and not, good design and not. Good Go code tend to be short. Many gophers prefer syntax highlighting, so there's no need to force-adopt "opinions" when not strictly necessary or ideal. Alot of Go is very community and stdlib-driven though.
How to do it yourself: https://blog.joshsoftware.com/2014/03/10/how-do-i-create-a-p...
The Go Tour has no syntax highlighting, but the design on the pages are fit for the purpose of learning the language: https://tour.golang.org/welcome/1
The typical Go-presentations (http://go-talks.appspot.com/github.com/SatishTalim/slides/sa...) tend to be very short and readable, but you'll find www-pages/videos about Go having both syntax highlighting and not, good design and not. Good Go code tend to be short. Many gophers prefer syntax highlighting, so there's no need to force-adopt "opinions" when not strictly necessary or ideal. Alot of Go is very community and stdlib-driven though.
How to do it yourself: https://blog.joshsoftware.com/2014/03/10/how-do-i-create-a-p...
The code comments are a lighter shade, so it Seems there is some form of very subtle syntactic styling going on.
[deleted]
It's typographically more pleasant.
I don't think that's the reason, otherwise I'd see it evenly in all languages and not only Go. I don't think Go users inherently value typography more than users of other languages
Maybe you're right.
But did you know there's a Go font?
https://blog.golang.org/go-fonts
But did you know there's a Go font?
https://blog.golang.org/go-fonts
Or you could just adopt https://opentelemetry.io/ and then everyone can benefit from all OT enabled libraries without having to write extra code, and you won't have to rewrite in order to change implementations since it's just an API.
The go implementation of OT makes extensive use of the Context [1] object to support tracing, logging, and metrics. You write once, and then the user only needs to decide which exporter to use, and possibly filters to apply.
[1] https://golang.org/pkg/context/
The go implementation of OT makes extensive use of the Context [1] object to support tracing, logging, and metrics. You write once, and then the user only needs to decide which exporter to use, and possibly filters to apply.
[1] https://golang.org/pkg/context/
What happened to OpenTracing?
OpenTelemetry is the project formed from the merger of OpenTracing and OpenCensus, under the CNCF: https://opentelemetry.io/about/
Is opentelemetry ready for production usage? As my understanding this project is a merge between OpenCensus and OpenTracing, but it’s still in beta and the documentation is really lacking. Does anyone have any hands-on experience with this library?
SDKs for languages are in beta and might not be production ready but the collector is production ready. You can use any of the OpenTracing or OpenCensus instrumentation libraries but deploy the OpenTelemetry collector and once Otel SKDs mature, migrate OC/OT to Otel or not if you don't want to.
I've used it, and it's very similar to OpenCensus and OpenTracing. If you've used either, it's not much of a stretch to use OpenTelemetry.
That said, it is still beta, and will be until Q4.
That said, it is still beta, and will be until Q4.
As a library developer, don't you need to define exactly the same trace points (hooks) in your code?
Am I understand it right, that you suggest instead of doing it in a generic way (any tracing method could be plugged in this way), to stick to one particular library?
Am I understand it right, that you suggest instead of doing it in a generic way (any tracing method could be plugged in this way), to stick to one particular library?
Yes, because then the library developer can define all of the things that COULD be interesting, and leave it to the user of the library to decide WHAT to record (filters), and WHERE to record it (exporters).
And since this is a standard, it means that when a user includes a bunch of disparate libraries/remote calls in his app, he can define his instrumentation wishes once for everything during app initialization. The libraries can even cooperate since they use the same underlying API and concepts for instrumentation.
And since this is a standard, it means that when a user includes a bunch of disparate libraries/remote calls in his app, he can define his instrumentation wishes once for everything during app initialization. The libraries can even cooperate since they use the same underlying API and concepts for instrumentation.
> Yes, because then the library developer can define all of the things that COULD be interesting, and leave it to the user of the library to decide WHAT to record (filters), and WHERE to record it (exporters).
I agree that this uppercase words can be called as somewhat purposes of the tracing API. I believe also that I was talking exactly about the same approach in the article. But I can't get how sticking to one particular library will help here? In other words, I think it will just greatly reduce the usage options.
What Im saying is that you can leave the generic hooks in the library (as it described in the article and gtrace) and then let the user choose which library to use inside what hooks.
I agree that this uppercase words can be called as somewhat purposes of the tracing API. I believe also that I was talking exactly about the same approach in the article. But I can't get how sticking to one particular library will help here? In other words, I think it will just greatly reduce the usage options.
What Im saying is that you can leave the generic hooks in the library (as it described in the article and gtrace) and then let the user choose which library to use inside what hooks.
With a standard library and API that everyone adheres to, you just pass in some initial configuration during app initialization to decide which libraries record what kinds of information. The user only touches one file to enable and choose instrumentation.
With hooks, you get nothing unless you hook, which means you have to write hooks for everything you're interested in (and also come up with a way to export the data), and so the user must touch code all over the place.
With hooks, you get nothing unless you hook, which means you have to write hooks for everything you're interested in (and also come up with a way to export the data), and so the user must touch code all over the place.
I think what OP is getting at is more like how to manually instrument your code. You can use OpenTelemetry but you might still want to manually instrument code using OpenTelemetry and that involves making calls to records metrics, create spans etc. The pattern OP is proposing fits perfectly well with using something like OpenTelemetry.
Tracing is a microservices phenomenon, right? In other words, would it benefit monolithic systems that already log and collect metrics?
Even monoliths are dealing with outbound network calls.
I assume it depends on your current solution, but for a new project, I find that setting up instrumentation and pushing traces to Zipkin or Jaeger is very straightforward, at least on the JVM. Out of the box I can see the overhead on top of DB queries and plain HTTP calls. And if some bit of internal logic is particularly time consuming, I can add a custom span in two lines of code and figure out what's going on. There are of course other ways to achieve the same thing, but the experience feels nicer.
I assume it depends on your current solution, but for a new project, I find that setting up instrumentation and pushing traces to Zipkin or Jaeger is very straightforward, at least on the JVM. Out of the box I can see the overhead on top of DB queries and plain HTTP calls. And if some bit of internal logic is particularly time consuming, I can add a custom span in two lines of code and figure out what's going on. There are of course other ways to achieve the same thing, but the experience feels nicer.
Tracing term is kind of overloaded. In this article its used not as _distributed tracing_, but as _instrumenting_. E.g., Linux's trace points, or any other tools such dtrace, strace and so on. So it doesn't relate to systems design decisions.
if you've got an IT ecosystem with enough monoliths / macroservices calling each other, being able to trace across service boundaries is still pretty useful
Turns out you can use any language to write Java code.
Something to note here is that everything is known and done statically in the source code. This could be addressed through metaprogramming if only the Go language had some ^^
We have similar needs at Sqreen but for security monitoring and protection reasons: we need to dynamically instrument functions at run time, while not asking any code modification to our users. To do so, we instead leverage the Go compiler to perform compile-time instrumentation that inserts hooks anywhere interesting. You can read more about this approach at https://blog.sqreen.com/dynamic-instrumentation-go/
We have similar needs at Sqreen but for security monitoring and protection reasons: we need to dynamically instrument functions at run time, while not asking any code modification to our users. To do so, we instead leverage the Go compiler to perform compile-time instrumentation that inserts hooks anywhere interesting. You can read more about this approach at https://blog.sqreen.com/dynamic-instrumentation-go/
I have read this article before and I really like the idea of doing so. I would love to read more about how you guys modified the compiler!
We haven't modified the compiler but instead plugged into it an external instrumentation tool using the compiler option `-toolexec`.
With this option, the compiler invokes every toolchain binary (compile, asm, link, etc.) through the provided program. So you can basically write a proxy program intercepting calls to `compile` to do source-code instrumentation, exactly like you would with go generate, but now automatically done during compilation on every package.
With this option, the compiler invokes every toolchain binary (compile, asm, link, etc.) through the provided program. So you can basically write a proxy program intercepting calls to `compile` to do source-code instrumentation, exactly like you would with go generate, but now automatically done during compilation on every package.
Ah, cool! Thank you so much, didn't know about this parameter :D Will try it definitely.
The problem is there are a lot of off-the-shelf observability frameworks for Go that require the pattern in the article: you emit a value to a hook function at the point of production. This sucks at any reasonable scale because you are now required to just take a global lock (or write a channel, which is the same thing) every time you record a value. This is a significant contributing reason why Go servers fall apart when given access to more than a handful of CPU cores.