Monitoring Microservices with Synthetic Transactions in Go(labs.unacast.com)
labs.unacast.com
Monitoring Microservices with Synthetic Transactions in Go
http://labs.unacast.com/2016/03/10/monitoring-microservices-synthetic-transactions-in-go/
21 comments
> Also, I'm not a Go developer but is this idiomatic?
The error handling is kinda strange.
I would do some like this:
The error handling is kinda strange.
I would do some like this:
var ErrUnexpectedResponse = errors.New("unexpected response")
....
func syntheticHttpRequest(url string, apiToken string) error {
...
} else if resp.StatusCode != 202 {
return ErrUnexpectedResponse
}
return nil
}
What's strange about this?
We're doing something similar but for our whole stack.
We have a large, complex pipeline for processing incoming data. Historically it has been very opaque which made debugging data errors hard.
We added event logging throughout the pipeline. We are building a tool that feeds known data in one end and checks the final output. We can then use the event logging in between to monitor state throughout for errors. It also allows us to see how long it takes to transition between events.
Ideally the tool will check every event for deviations from the expected output and alert us if any events fail. It will also alert us if the time between events rises above the average by a predefined threshold.
I'd love to hear about any open source tools designed to do something like this.
We have a large, complex pipeline for processing incoming data. Historically it has been very opaque which made debugging data errors hard.
We added event logging throughout the pipeline. We are building a tool that feeds known data in one end and checks the final output. We can then use the event logging in between to monitor state throughout for errors. It also allows us to see how long it takes to transition between events.
Ideally the tool will check every event for deviations from the expected output and alert us if any events fail. It will also alert us if the time between events rises above the average by a predefined threshold.
I'd love to hear about any open source tools designed to do something like this.
Well, of course. There are other things you need if you do this. Large systems need some way to direct internal transactions to specific servers. In systems with load balancing, one server may be in trouble, but the others are carrying the load and the overall system seems to be OK.
To monitor this, you need some kind of dashboard which displays the state of all your servers, and shows the dependency relationships. If A calls B and B calls C, and B fails, you'll see A and B as down. You need to be able to establish that B is the problem. (Microservice architectures which are not DAGs, i.e., they have loops, are a huge pain in this sense.)
To monitor this, you need some kind of dashboard which displays the state of all your servers, and shows the dependency relationships. If A calls B and B calls C, and B fails, you'll see A and B as down. You need to be able to establish that B is the problem. (Microservice architectures which are not DAGs, i.e., they have loops, are a huge pain in this sense.)
Note that this approach isn't just useful for microservices - it's a great way to monitor any complex system, even if it's a monolith.
I wasn't familiar with "synthetic transactions" until I read this post--great write up! Synthetic transactions are kind of like TDD on a live system (really loose analogy). It's a cool concept for keeping microservices consistent and bug-free by verifying responses return data that are expected.
Why does the author use double ticks? As far as I know you use only one.
ID string ``json:"id"``The code examples are quite non-idiomatic and have lots of issues, to be honest.
- constant strings (defined as vars) as return values instead of the standard error type
- not formatted
- synthetic request's response body not closed - a memory leak
- log.Fatal (which results in os.Exit(1) immediately killing the program) for all the non-critical errors
- timestamp as a string instead of time.Time in the struct
- not re-using http.DefaultClient
- 3rd party package with a custom DSL instead of a simple time.Ticker
Sorry for the pedantry, but if such code examples live on the internet, newcomers to Go will learn from it (if not blindly copy-paste) and it will spread even further indirectly giving bad reputation to the language.
P.S. i do like this idea of continuous integration testing in production, will probably start using it at our company.
- constant strings (defined as vars) as return values instead of the standard error type
- not formatted
- synthetic request's response body not closed - a memory leak
- log.Fatal (which results in os.Exit(1) immediately killing the program) for all the non-critical errors
- timestamp as a string instead of time.Time in the struct
- not re-using http.DefaultClient
- 3rd party package with a custom DSL instead of a simple time.Ticker
Sorry for the pedantry, but if such code examples live on the internet, newcomers to Go will learn from it (if not blindly copy-paste) and it will spread even further indirectly giving bad reputation to the language.
P.S. i do like this idea of continuous integration testing in production, will probably start using it at our company.
That is an error and would not compile.
This code example has not gone through gofmt or been compiled... it is pseudo-Go
This code example has not gone through gofmt or been compiled... it is pseudo-Go
Well, that is a typo. I'll fix it =) Thank you!
datadoghq link is broken as well.
Ah! Seems like I messed up the markdown. Fixed now. Thank you for your feedback.
I'm going to risk a plug since our service is highly relevant. If you're using AWS, Opsee will let you define service-level health checks and automatically detect instance membership. You can set detailed assertions on response bodies, including JSON keys, to verify that your services are responding as expected. This all works inside your environment, running checks from an instance we spin up. More info at opsee.com
The microservices trend is great but this sort of monitoring starts to fall apart when you horizontally scale for HA. In this case we separate out our monitoring by doing synthetic requests at the service level, and then very specific health checking at the "instance" level (really containers now). Instance level health checks will ensure connectivity to outside dependencies, databases, filesystems etc. The trick is to know when a failure is a localized failure or a widespread failure. No sense in taking out all of the instances if they all can't talk to the DB.
Also, I'm not a Go developer but is this idiomatic?