gRPC Is Easy to Misconfigure(evanjones.ca)
evanjones.ca
gRPC Is Easy to Misconfigure
https://www.evanjones.ca/grpc-is-tricky.html
44 comments
> You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS
You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.
You shouldn't, but it's easy to accidentally do. Having a limit on the server and a limit on the client that's (by default) half the limit on the server seems like an unforced error - given that the server does truncate at a certain limit, surely it would be better for that limit to be smaller than the default client limit. Postel's law and all that.
> You probably shouldn't be sending back >8kb error messages to the client
While I agree, it seems like a strange limit to have as part of your spec.
While I agree, it seems like a strange limit to have as part of your spec.
There is no spec, as far as I can tell. That's why the article says different implementations disagree on the max size of the error message.
As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs.
Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
As for large errors, there are plenty of cases where they can be useful. Not every RPC is crossing a privacy boundary: if an administration CLI performs an RPC that fails, it's more convenient to have lots of debugging information printed to the admin than to force them to go and dig through the server logs.
Honestly, gRPC makes me a bit sad. It's clearly a duplicate of Stubby. I don't know what Google use these days, but Stubby was a truly great RPC system. For inexplicable reasons Google constantly choose to badly reimplement their internal tech as part of open sourcing it, and gRPC is like some cheap knockoff in comparison.
Part of the problem with open sourcing Google's software is that the software is often integrated with a bunch of the rest of the google stack. This could just be dependencies on little libraries that may or may not already be open source, all the way up to depending on various large components to do some of the heavy lifting around security/scheduling/storage etc.
If some library (that the thing being opened up uses) has an open open source version, it is necessary to worry about versioning concerns, because unlike inside google3, the open source versions can no longer make breaking changes merely by updating every single consumer. So the internal version in some cases will have drifted from the open source version. Even when a library or tool needed is already opened sourced, it is not rare for a newly open source program to use a stub version instead of the full tool/library to avoid such headaches. (e.g. I've seen super stripped down alternative to gtest shipped as part of some opened libraries, because trying to utilize the open source version was more hassle than it was worth.)
For libraries or tools not open sourced, some form of stub or adapter to some similar publicly available software is needed. In some cases that does not exist, making directly open sourcing basically impossible (but redeveloping the tool might be possible). In other cases there are alternatives available, but they might be lacking features that made the integration nicer inside Google.
Overall I can totally understand why Googlers sometimes want to basically redesign an internal tool as the approach to open sourcing the underlying concept, rather than try to publish the equivalent-ish internal tool.
If some library (that the thing being opened up uses) has an open open source version, it is necessary to worry about versioning concerns, because unlike inside google3, the open source versions can no longer make breaking changes merely by updating every single consumer. So the internal version in some cases will have drifted from the open source version. Even when a library or tool needed is already opened sourced, it is not rare for a newly open source program to use a stub version instead of the full tool/library to avoid such headaches. (e.g. I've seen super stripped down alternative to gtest shipped as part of some opened libraries, because trying to utilize the open source version was more hassle than it was worth.)
For libraries or tools not open sourced, some form of stub or adapter to some similar publicly available software is needed. In some cases that does not exist, making directly open sourcing basically impossible (but redeveloping the tool might be possible). In other cases there are alternatives available, but they might be lacking features that made the integration nicer inside Google.
Overall I can totally understand why Googlers sometimes want to basically redesign an internal tool as the approach to open sourcing the underlying concept, rather than try to publish the equivalent-ish internal tool.
I know how Googlers justify it, I used to work there. But it's bizarre. Their codebase is highly modular but it's as if they forget all about modules, interfaces and so on the moment they talk about releasing stuff as open source.
I long ago concluded the actual motivating factor is that it's more fun to write new code than refactor existing code, and especially, it's more promotion worthy.
I long ago concluded the actual motivating factor is that it's more fun to write new code than refactor existing code, and especially, it's more promotion worthy.
The error message is sent as a HTTP trailing header. Their server or client has a header size limit of 8kb.
Can be quite useful to enable verbose errors during dev and initial tests.
Maybe. But the protocol should not set a size limit on the error response.
Yep, it's also quite easy to leak sensitive data in errors so you shouldn't just send every error your code produces to the client. Maybe less a concern for gRPC as it's often used internally, but some systems still might propagate the error logs to untrusted clients. In the worst case your traceback will contain environment variables, which in turn might contain database secrets or API tokens (of course it's also bad practice storing secrets in environment variables but people still do it a lot).
In our projects we solve this by implementing internal and external error types. External errors (e.g. form validation errors) will be sent to the client, internal errors will just send a generic error message (e.g. "please contact tech support. ID: #fa3ca....") and will be logged internally so we can trace them back to their source and ideally fix them.
In our projects we solve this by implementing internal and external error types. External errors (e.g. form validation errors) will be sent to the client, internal errors will just send a generic error message (e.g. "please contact tech support. ID: #fa3ca....") and will be logged internally so we can trace them back to their source and ideally fix them.
storing secrets in environment variables
This is exactly how a lot of frameworks tell you to provide secrets, and it's also how at least some cluster management tools pass secrets into apps as well. It always seemed strange to me that the environment was considered "secure".
This is exactly how a lot of frameworks tell you to provide secrets, and it's also how at least some cluster management tools pass secrets into apps as well. It always seemed strange to me that the environment was considered "secure".
> should log error details locally and the logs should be inspected or scraped separately
Adding extra parts to a already complex system does not solving problems. It only transfers the problem to other components. e.g. what happens when the "scraper" also use gRPC and want to yield a >8KB error?
Adding extra parts to a already complex system does not solving problems. It only transfers the problem to other components. e.g. what happens when the "scraper" also use gRPC and want to yield a >8KB error?
I would say gRPC is fine as long as you don’t have the following:
1) work on a team where people will make assumptions about how gRPC works because they will be wrong, for example see the article, but more generally it can be quite obtuse.
2) want to be able to easily debug over the wire information
3) expect there to be a wealth of stack overflow knowledge to be able to debug weird incompatibilities between go and JavaScript implementations
4) schema that you expect to rapidly change over time and/or versioning of said schema to avoid strange results. It’s extremely difficult to track these changes on a large codebase.
1) work on a team where people will make assumptions about how gRPC works because they will be wrong, for example see the article, but more generally it can be quite obtuse.
2) want to be able to easily debug over the wire information
3) expect there to be a wealth of stack overflow knowledge to be able to debug weird incompatibilities between go and JavaScript implementations
4) schema that you expect to rapidly change over time and/or versioning of said schema to avoid strange results. It’s extremely difficult to track these changes on a large codebase.
If you were at a shop that used grpc a lot, would you not have tools ready to go that can turns the protobuf back into text?
I've often thought how odd it is that the world is spending time and energy parsing things into text formats and back, when its just two computers communicating...making more work for themselves and the entire internet backbone....just because we can't be bothered to use a simple tool when debugging over the wire.
I've often thought how odd it is that the world is spending time and energy parsing things into text formats and back, when its just two computers communicating...making more work for themselves and the entire internet backbone....just because we can't be bothered to use a simple tool when debugging over the wire.
That is exactly what we do. You can write a python tool in like a dozen lines to cast to and from.
There is also a way to use one of the built in protobuf tools to do it but you also have to point at the proto files yourself.
I think there are other advantages to text protocols besides debugging on the wire, namely ease of composing tools. Streams of text is extremely versatile and economic for many workflows. Of course you lose that ease once you start embedding binary data.
There is also a way to use one of the built in protobuf tools to do it but you also have to point at the proto files yourself.
I think there are other advantages to text protocols besides debugging on the wire, namely ease of composing tools. Streams of text is extremely versatile and economic for many workflows. Of course you lose that ease once you start embedding binary data.
Wireshark has supported gRPC for years. Its really not any harder than looking at HTTP2 traffic.
Even with binary data we spend our time flipping the bytes to network order.
If on-the-wire readability is important you can always opt for a different marshaling method, like JSON (or even roll your own). Yes it's some extra work but not a showstopper if that's the only downside of gRPC/Protobufs.
If 8kB error output limit and incredibly long keepalive means it's easy to misconfigure gRPC then so is everything else.
> In my opinion, this means this setting is virtually unusable and should be avoided
I've used the gRPC keepalives intentionally for long-lived one directional streaming. You definitely need to pay careful attention to load balancing in the middle, but it does work well if you configure it properly.
If your stream messages are infrequent, you should also add the occasional Ping message in order for the server to identify and reap stale connections.
I had small ~1 core, 512mb containers handling some ~50k connections without breaking a sweat this way. The main scaling point was the goroutine stack size for each.
I've used the gRPC keepalives intentionally for long-lived one directional streaming. You definitely need to pay careful attention to load balancing in the middle, but it does work well if you configure it properly.
If your stream messages are infrequent, you should also add the occasional Ping message in order for the server to identify and reap stale connections.
I had small ~1 core, 512mb containers handling some ~50k connections without breaking a sweat this way. The main scaling point was the goroutine stack size for each.
> when client keepalive is too aggressive, client RPCs will fail with gRPC code UNAVAILABLE (14) and message "transport is closing"
It seems like the correct way for the client to handle this, would be to re-establish the connection and retry if this happens. And that should be done by the grpc client library.
It seems like the correct way for the client to handle this, would be to re-establish the connection and retry if this happens. And that should be done by the grpc client library.
It normally does this in my experience (same when disconnection happens due to network unavailability). Have faced issues in dart where it takes some time to reconnect but it always reconnects automatically.
Recently I started delving into gRPC vs RPC over RabbitMq using json as the mesage format. I saw that for small to medium sized messages, gRPC is actually slower. Of course, this was just a small scale experiment so I don't think nothing of it.
Does anyone have a sort of infrastructure/architecture guide at a bigger scale for gRPC?
My biggest questions range from: How do you actually load balance the servers? What happens if you have a sudden influx of requests but don't want to auto-scale? Do you still need a sort of queue-ing system in front of the gRPC server?
In my research I wasn't able to find some noteworthy articles about this and thus triggered my curiosity.
Does anyone have a sort of infrastructure/architecture guide at a bigger scale for gRPC?
My biggest questions range from: How do you actually load balance the servers? What happens if you have a sudden influx of requests but don't want to auto-scale? Do you still need a sort of queue-ing system in front of the gRPC server?
In my research I wasn't able to find some noteworthy articles about this and thus triggered my curiosity.
I don't know if there is a unified playbook, but there are random blog posts. One that comes to mind (disclosure: I worked on the system, mostly after the blog entry) is https://dropbox.tech/infrastructure/courier-dropbox-migratio... (and perhaps https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...)
I suppose the problem with an easy go-to playbook is if that were easy to pick one solution for every problem, gRPC would have likely picked it by default (I can attest the team, and the external contributors, are top notch, having worked alongside them). Unfortunately some problems, at scale, need to be answered depending on architecture with the holistic system in mind, and are not just gRPC issues. Truth is large scale systems are hard to build and operate. Perhaps that is why you get help from experienced individuals and consultants.
I do agree some documentation/tooling is lacking and could be improved to guide folks through the process.
I suppose the problem with an easy go-to playbook is if that were easy to pick one solution for every problem, gRPC would have likely picked it by default (I can attest the team, and the external contributors, are top notch, having worked alongside them). Unfortunately some problems, at scale, need to be answered depending on architecture with the holistic system in mind, and are not just gRPC issues. Truth is large scale systems are hard to build and operate. Perhaps that is why you get help from experienced individuals and consultants.
I do agree some documentation/tooling is lacking and could be improved to guide folks through the process.
Thank you very much for these articles. Really top-notch work and provided me with some great insights!
I found gRPC not particularly fast on a per-message basis unless you're using the streaming feature. If you have iterated calls, consider streaming instead.
Because it supports full duplex streaming, there's a risk of tunneling your own less than fully specified protocol on top of gRPC. In some circumstances that may be worth taking advantage of, because gRPC takes care of session management, reconnecting, authorization (i.e. it has ways for you to add authorization, like headers) etc.
If you need queuing I think you should use a queue instead.
Because it supports full duplex streaming, there's a risk of tunneling your own less than fully specified protocol on top of gRPC. In some circumstances that may be worth taking advantage of, because gRPC takes care of session management, reconnecting, authorization (i.e. it has ways for you to add authorization, like headers) etc.
If you need queuing I think you should use a queue instead.
Very interesting. Did you also try Protobuf or MsgPack / SMILE over RabbitMq?
Never tried with MsgPack, but I did try Protobuf. The reason I dislike Protobuf is that there is an extra code generation step I need to do. When using a compiled language, I guess you don't really mind the extra code generation step since you also get some safety from the compiler so you don't wind up miss-using the generated code. It is not the same when you are using an interpreted language that doesn't have really good strong typing. You will have to run a static code analyser and be very careful every time you do a change to the interface.
Since in my experiments I was calling a method in Php through RabbitMq from a Rust worker, so it just proved allot simpler to just use json. Also, I measured the time it took to:
1. Make a request to the Php API
2. Php sends the rpc message on the queue
3. Rust processes that message
4. Php catches the response from the worker
5. Php returns the response to the http client
It was <10ms running the cluster locally regardless if I used Protobuf or simply json.
Since in my experiments I was calling a method in Php through RabbitMq from a Rust worker, so it just proved allot simpler to just use json. Also, I measured the time it took to:
1. Make a request to the Php API
2. Php sends the rpc message on the queue
3. Rust processes that message
4. Php catches the response from the worker
5. Php returns the response to the http client
It was <10ms running the cluster locally regardless if I used Protobuf or simply json.
The protobuf vs msgpack benchmarks are not too bad. Msgpack performs very decently.
https://github.com/alecthomas/go_serialization_benchmarks
https://github.com/alecthomas/go_serialization_benchmarks
One observation / warning I have WRT msgpack is that I have seen situations where data getting serialized directly to msgpack can't be rendered as proper json. Specifically JSON requires some proper encoding of characters while msgpack is perfectly happy to convey some binary characters.
The specific situation I saw was 1) antique perl application barfs a sql dump into msgpack 2) fluentd takes that, turns it into a thing like json but with a control character in it (think old time cyan blinky happy face or \0x03) 3) that thing gets dumped into kafka 4) logstash pulls that thing out of kafka and tries to feed it to elasticsearch 5) elasticsearch reports that it doesn't like the blinky happy face, generates an extensive log, and stores the event minus the key/value pair with the blinky happy face.
Certainly many of these steps have an implicit "don't do that!" or "Update to a newer thinger!" but the root cause (perl serializer into msgpack renders a thing that's valid msgpack but not valid json) is surprising in an unpleasant way.
The specific situation I saw was 1) antique perl application barfs a sql dump into msgpack 2) fluentd takes that, turns it into a thing like json but with a control character in it (think old time cyan blinky happy face or \0x03) 3) that thing gets dumped into kafka 4) logstash pulls that thing out of kafka and tries to feed it to elasticsearch 5) elasticsearch reports that it doesn't like the blinky happy face, generates an extensive log, and stores the event minus the key/value pair with the blinky happy face.
Certainly many of these steps have an implicit "don't do that!" or "Update to a newer thinger!" but the root cause (perl serializer into msgpack renders a thing that's valid msgpack but not valid json) is surprising in an unpleasant way.
You didn't define small or medium size and number of messages.
From my experience server stream while great for memory and giving the client time to process messages as they come can be slower than unary. One way to solve that is to have a batched message with a repeated field of the underline message you want to send, it's an order faster.
I read sometime ago about about server pushback, maybe you need to override/implement your own StreamObserver to notify the client to slow down as the server isn't ready yet or configure the amount of data the gRPC server can queue there are quite a few configurations (a lot of them obscured) you can make to the server/grpc service.
I read sometime ago about about server pushback, maybe you need to override/implement your own StreamObserver to notify the client to slow down as the server isn't ready yet or configure the amount of data the gRPC server can queue there are quite a few configurations (a lot of them obscured) you can make to the server/grpc service.
Seems so much work for something already implemented with messaging brokers, though.
In my opinion, clients themselves shouldn't worry about if the server is ready or not, only handle if the server does not respond in x seconds and then simply crash or error out. It is the same you would to with any other external service call.
I think the biggest change to gRPC is to the way I thought. Dumb clients was always something that I chose because it was an order of magnitude simpler to reason about. gRPC comes in and changes this by making the clients smart and the servers smarter, which brings allot of complexity to the table.
Also, indeed the configurations are so obscure.
In my opinion, clients themselves shouldn't worry about if the server is ready or not, only handle if the server does not respond in x seconds and then simply crash or error out. It is the same you would to with any other external service call.
I think the biggest change to gRPC is to the way I thought. Dumb clients was always something that I chose because it was an order of magnitude simpler to reason about. gRPC comes in and changes this by making the clients smart and the servers smarter, which brings allot of complexity to the table.
Also, indeed the configurations are so obscure.
It's just a different framework for a different use case. The fact that it's http 2 and support bi-directional streams is nice for low latency applications.
I doesn't make RPC over message brokers deprecated and if that works for you under you conditions that's great.
You can say the same about message brokers. Is it LIFO, FIFO or something random? what about acknowledge, is it late or not? Can the queue hold responses (Rabbit advices against using it as a result store)?
In that regard message brokers are complicated, lets just do http calls and let the load balancer take care of routing, it's much simpler and the client is very dumb.
I doesn't make RPC over message brokers deprecated and if that works for you under you conditions that's great.
You can say the same about message brokers. Is it LIFO, FIFO or something random? what about acknowledge, is it late or not? Can the queue hold responses (Rabbit advices against using it as a result store)?
In that regard message brokers are complicated, lets just do http calls and let the load balancer take care of routing, it's much simpler and the client is very dumb.
My experience is that it “Just Works”
Same here, for the vast majority of non-netflix scale use cases it works extremely well out of the box.
I've seen similar type errors when our ingress proxy switched from nginx to envoy. My golang http(s) rest requests were randomly getting unexpected EOF/transport closed errors. Ended up solving it, by simply putting .Close = true into all the requests to force no pipelining (or at least as I understand it).
How do you use a non-Google oidc idp though?
That's the main thing keeping me away from it.
You have the non-choice of using Google Auth or include the token in every request outside of credentials
You can construct credentials from any ID token, it ends up as a standard Authorization: Bearer header on each RPC. I believe this is the case for Google Auth too.
The keepalive thing is a bit of a problem; we've been using the C# API in a client/server way, between processes on the same system written in different languages, and finding out if the other end of the connection has fallen over is surprisingly hard.
Wow, sounds like AWS and Azure network services are really unsuited for a lot of real world TCP applications. You couldn't even use these to implement a load balanced telnet server for playing NetHack.
gRPC docs do need some tlc. Cloud providers providing support for gRPC also need to do better documenting their support for gRPC.
AWS Application Load Balancer is advertised to work with gRPC but we've been seeing sporadic errors. Keepalive isn't the issue, we set the IdleTimeout to the max (4000 seconds) and use server side keepalive to gc old connections.
AWS Application Load Balancer is advertised to work with gRPC but we've been seeing sporadic errors. Keepalive isn't the issue, we set the IdleTimeout to the max (4000 seconds) and use server side keepalive to gc old connections.
> If a client sends keepalive pings too often, servers close the connection. This is to prevent large numbers of idle clients from consuming too many resources.
This is horrible. Let users configure what they choose. If your network needs higher frequency keepalives, why would the authors of the spec try to know better.
I agree with the next statement:
> In my opinion, this means this setting is virtually unusable
This is horrible. Let users configure what they choose. If your network needs higher frequency keepalives, why would the authors of the spec try to know better.
I agree with the next statement:
> In my opinion, this means this setting is virtually unusable
Define the "user" in the scenario where it is "my" gRPC server but "your" client affecting "my" SLA.
If you can't handle the client, why did you accept the connection in the first place? It has nothing to do with what you're using the connection for. I have an IP address on my ISP connection whether or not I use any data. It's factored into their scaling already.
You probably shouldn't be sending back >8kb error messages to the client. The server should log error details locally and the logs should be inspected or scraped separately. The client doesn't need to know the full details like the contents of a variable length DS
I develop gRPC clients and servers at work that handle about 40k QPS (typically <50ms response times) whilst using default gRPC configuration. The only changes I had to make was to the max message size.