Stream Compression in NATS(qaze.app)
qaze.app
Stream Compression in NATS
https://qaze.app/blog/nats-stream-compression/
23 comments
> Seems like I need to write a service utilizing NATS which talks to the OIDC server.
Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration.
> NATS auth story is a complicated one, and now with auth callout it's even more complicated.
There is a spectrum of auth options, starting with simple config-based, token or user/pass leading up to decentralized auth for use cases that need it. Auth callout is an opt-in thing, so it should only be adopted if it is truly necessary.
> you have to have dedicated functions which you would call in the request handler [...] instead of being able to define a chain of middleware functions
I don't quite understand this statement. Wrapping a NATS handler is the same approach as wrapping an HTTP handler (within the same client app). The function would take a handler and return a handler. There can be inspection of the message within that function and the choice of calling the next handler, responding early, doing some external call, or doing nothing.
If you want to distribute this middleware, then you would need to have separate, explicit subjects that each middleware would subscribe to and then publish to for the next element in the chain (for a choreography approach).
There is also the "message slip pattern" where the ingest component sets the path as metadata (e.g. headers) that each middleware component uses that to know (at request time) which subject to publish the result to next.
Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration.
> NATS auth story is a complicated one, and now with auth callout it's even more complicated.
There is a spectrum of auth options, starting with simple config-based, token or user/pass leading up to decentralized auth for use cases that need it. Auth callout is an opt-in thing, so it should only be adopted if it is truly necessary.
> you have to have dedicated functions which you would call in the request handler [...] instead of being able to define a chain of middleware functions
I don't quite understand this statement. Wrapping a NATS handler is the same approach as wrapping an HTTP handler (within the same client app). The function would take a handler and return a handler. There can be inspection of the message within that function and the choice of calling the next handler, responding early, doing some external call, or doing nothing.
If you want to distribute this middleware, then you would need to have separate, explicit subjects that each middleware would subscribe to and then publish to for the next element in the chain (for a choreography approach).
There is also the "message slip pattern" where the ingest component sets the path as metadata (e.g. headers) that each middleware component uses that to know (at request time) which subject to publish the result to next.
Regarding middleware, last time I tried there was a lot of information to process.
Now I tried again and
Regarding auth, yes, spectrum of options, but for "production" use it's very complicated and you don't want a shared key, you need the whole operator, account, user thing, and external auth aka auth callout. The documentation is lacking. nkey, xkey what? It's confusing is complicated. Managing auth is a pain, more than once I've been locked out of my test server because of how complicated auth is and had to wipe it. It's so complicated because you have have isolated "namespaces" or operator spaces? Naming an organization operator, or a solution or service group doesn't help. user is synonymous with client. All that you don't get from the documentation, you have to watch Youtube videos to understand it. The documentation is bad.
How do I consume a stream from the deno/ws client without the whole client app blocking? No idea. The code snippets don't help. All there is, is a pay for company that offers to host NATS clusters, which you can pay to ask. Slack is not a good medium.
I'm guessing that the documentation is bad so one has to pay for support. And that's a no-go for me. I get that you spent time and effort into it, I'm assuming you're one of the NATS people, but what good is it when I don't know how to effectively use the software? The natsbyexample page is also hard to digest.
You're in your own little bubble and expect people to be mind readers.
If it's too complicated people won't adopt it. I know I won't because it's too complicated.
I would've loved to build upon it, because the core functionality is nice and works, easy to grasp.
I've not seen a single Jetstream stream and consumer tutorial with the new API and nats.ws and Go. Old API sure, STAN sure. New API no. Some real world stuff, not just code demo snippets. Why not? Because it's complicated.
Auth callout, that was the big thing everyone was waiting for? I would work on actually solving problems instead of building components so I can use this extra piece of software who's documentation is so bad.
You know how many Kafka tutorials there are? A LOT.
I want to like it, it seems very promising, but it suffers from the "coder no likey documentation" illness.
Today I saw a Twitch stream of some Netflix dev, talking about DX, developer experience.
NATS' DX has a lot of room for improvement, to put it friendly.
func loggingMiddleware(next micro.Handler) micro.Handler {
return micro.HandlerFunc(func(req micro.Request) {
spew.Dump(req.Data())
next.Handle(req)
})
}
Regarding auth, yes, spectrum of options, but for "production" use it's very complicated and you don't want a shared key, you need the whole operator, account, user thing, and external auth aka auth callout. The documentation is lacking. nkey, xkey what? It's confusing is complicated. Managing auth is a pain, more than once I've been locked out of my test server because of how complicated auth is and had to wipe it. It's so complicated because you have have isolated "namespaces" or operator spaces? Naming an organization operator, or a solution or service group doesn't help. user is synonymous with client. All that you don't get from the documentation, you have to watch Youtube videos to understand it. The documentation is bad.
How do I consume a stream from the deno/ws client without the whole client app blocking? No idea. The code snippets don't help. All there is, is a pay for company that offers to host NATS clusters, which you can pay to ask. Slack is not a good medium.
I'm guessing that the documentation is bad so one has to pay for support. And that's a no-go for me. I get that you spent time and effort into it, I'm assuming you're one of the NATS people, but what good is it when I don't know how to effectively use the software? The natsbyexample page is also hard to digest.
You're in your own little bubble and expect people to be mind readers.
If it's too complicated people won't adopt it. I know I won't because it's too complicated.
I would've loved to build upon it, because the core functionality is nice and works, easy to grasp.
I've not seen a single Jetstream stream and consumer tutorial with the new API and nats.ws and Go. Old API sure, STAN sure. New API no. Some real world stuff, not just code demo snippets. Why not? Because it's complicated.
Auth callout, that was the big thing everyone was waiting for? I would work on actually solving problems instead of building components so I can use this extra piece of software who's documentation is so bad.
You know how many Kafka tutorials there are? A LOT.
I want to like it, it seems very promising, but it suffers from the "coder no likey documentation" illness.
Today I saw a Twitch stream of some Netflix dev, talking about DX, developer experience.
NATS' DX has a lot of room for improvement, to put it friendly.
> You're in your own little bubble and expect people to be mind readers.
Out of curiosity, have you asked questions in Slack or Github? If so, and you had a bad experience with the interaction, I get the sentiment and would offer help. But this comment is not constructive without context.
> I'm guessing that the documentation is bad so one has to pay for support.
The NATS project has been open source for around 12 years, and part of the CNCF since 2018. This is an incorrect statement and very poor assumption to make because the documentation doesn't make sense for you.
I 100% agree it can be improved and we are working on a new docs site, but it is not quite ready.
In case its helpful, there is an increasing collection of examples on https://natsbyexample.com with new JetStream client examples among others. If you have specific requests, feel free to open issues in the corresponding repo: https://github.com/ConnectEverything/nats-by-example
Out of curiosity, have you asked questions in Slack or Github? If so, and you had a bad experience with the interaction, I get the sentiment and would offer help. But this comment is not constructive without context.
> I'm guessing that the documentation is bad so one has to pay for support.
The NATS project has been open source for around 12 years, and part of the CNCF since 2018. This is an incorrect statement and very poor assumption to make because the documentation doesn't make sense for you.
I 100% agree it can be improved and we are working on a new docs site, but it is not quite ready.
In case its helpful, there is an increasing collection of examples on https://natsbyexample.com with new JetStream client examples among others. If you have specific requests, feel free to open issues in the corresponding repo: https://github.com/ConnectEverything/nats-by-example
It really doesn't help that Slack requires an account, isn't searchable and doesn't preserve history :)
If a learning resource is lacking or confusing, Slack or GitHub issues/discussions is a way to engage and provide feedback so they can be improved. If the docs were confusing, there are other channels to get help to unblock folks. The outcome of that interaction would lead to improvements in the docs.
Fair! And I have to say that the docs are pretty good considering the huge surface they cover! Same underlying API, but so many languages.
Same. I tried out some toy use-cases using nats.js over websocket a few months ago. The prospect of possibly being able to "directly" consume messaging or key/value store from the browser with only a thin gateway between was really interesting to me but I couldn't square up the NATS-internal JWT cookie thing with how you would handle auth in a traditional web-app (OAuth client on a gateway plus a session cookie).
I found some threads saying auth callout in 2.10 would solve this and decided to table the project until 2.10 but it's really really unclear how to work through the details of converting a "traditional" OAuth access token into the NATS-specified access token required by the auth callout contract.
I found some threads saying auth callout in 2.10 would solve this and decided to table the project until 2.10 but it's really really unclear how to work through the details of converting a "traditional" OAuth access token into the NATS-specified access token required by the auth callout contract.
Auth callout is post-NATS client authentication, so it would not solve the "auth web flow" for authentication. Instead, the resulting token from that would be set as a cookie that then would be passed into the nats.ws client connection. The auth callout service would use that token to map to the concrete NATS user. The mechanism of doing that is up to the implementation. One option is to manage NATS claims into the OIDC provider (for the user authenticating) and then the auth service would decode that source JWT and extract the NATS claims and generate the NATS user JWT in the response.
Thanks that observation is extremely helpful. If I have it down then the intended flow looks something like?
- Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform client auth - If further authz/fine-grained control is needed the auth callout mechanism can be used. This would have access to the provided cookie/token so any claims needed for access control could be stapled on during step one and used at this point?
For GPs original question -- I'm running a fairly old Keycloak version (v8) but it does appear to set a JWT in KEYCLOAK_IDENTITY and KEYCLOAK_IDENTITY_LEGACY.
Am I right in understanding that IFF the token is signed with Ed25519 and both sub and iss are an NKEY value this is sufficient for NATS to accept that cookie as a credential?
- Web client is directed to some token vending service. This service implement authn in a manner of its choosing (i.e. OAuth) then sets a NATS client JWT in the cookie per https://docs.nats.io/running-a-nats-service/configuration/se... - Nats.ws client connection provides cookie during connection to perform client auth - If further authz/fine-grained control is needed the auth callout mechanism can be used. This would have access to the provided cookie/token so any claims needed for access control could be stapled on during step one and used at this point?
For GPs original question -- I'm running a fairly old Keycloak version (v8) but it does appear to set a JWT in KEYCLOAK_IDENTITY and KEYCLOAK_IDENTITY_LEGACY.
Am I right in understanding that IFF the token is signed with Ed25519 and both sub and iss are an NKEY value this is sufficient for NATS to accept that cookie as a credential?
Yes that reads correct. The `sub` would a NATS user public nkey, the `iss` would be the NATS account public nkey (either the issuer nkey in config-mode or existing nkey in decentralized auth).
As long as it can verify the chain of trust for the user JWT that is returned, it should work.
The three schema types are shown here: https://docs.nats.io/running-a-nats-service/configuration/se...
auth request comes in -> generate user jwt, sign + encode -> respond with auth response.
As long as the necessary bits of the response and user JWT conform, it will work.
As long as it can verify the chain of trust for the user JWT that is returned, it should work.
The three schema types are shown here: https://docs.nats.io/running-a-nats-service/configuration/se...
auth request comes in -> generate user jwt, sign + encode -> respond with auth response.
As long as the necessary bits of the response and user JWT conform, it will work.
I was hoping the blog post would include a tentative roadmap for adding tiered storage.
This post is about a product for NATS.
I presume you are talking about the roadmap after the 2.10 release?
I presume you are talking about the roadmap after the 2.10 release?
Yea, after reading the article I went and read the 2.10 release blog post and then came back and commented as if the article posted was the 2.10 post. That was my mistake.
[deleted]
Didn't know what NATS is, page doesn't explain it.
Tried to click on the Docs link at the bottom of the page but that doesn't work.
Stopped looking.
Tried to click on the Docs link at the bottom of the page but that doesn't work.
Stopped looking.
NATS is a server for streaming data. It can be used as a pub/sub service like Redis, but it also implements RAFT consensus so that it can be used for globally-ordered, durable streams (more like Kafka). The server itself is a lightweight Go binary and it's much nicer to manage than alternatives.
NATS is a message broker.
It uses TCP or websocket, support "core" messaging aka simple fire and forget messaging, request/response, pub/sub.
And then message queues, subscriptions, topics, consumers, memory and replay functionality.
Also a MQTT compatible protocol.
I felt the same but had more luck. I found this comparison in their docs[1]: 'NATS Comparison to Kafka, Rabbit, gRPC, and others'.
1: https://docs.nats.io/nats-concepts/overview/compare-nats
1: https://docs.nats.io/nats-concepts/overview/compare-nats
Not sure that it makes sense to compare NATS to gRPC. Maybe the thinking is because gRPC supports bi-directional streaming? The integrations are wildly different, though.
I didn't stop looking and followed link to the version announcement, then went to the front page of nats.io from the top left of that page.
And I also still have absolutely no fucking idea what any of this is.
And I also still have absolutely no fucking idea what any of this is.
if you've got a bunch of programs running that want to send messages to one another, in various paradigms and various languages, nats.io may well help you.
If you look at [0] how does this explain anything? Seems like I need to write a service utilizing NATS which talks to the OIDC server.
NATS auth story is a complicated one, and now with auth callout it's even more complicated.
What I also quite didn't succeed in were JS client Jetstream consumers, async polling or w/e. All the examples, async or sync would block execution.
The regular request/response with microservices went well and straightforward. No issues at all. But writing middleware is also not very pretty, you have to have dedicated functions which you would call in the request handler (or microservice if you will) instead of being able to define a chain of middleware functions. Potejto potato but organization and structure and naming is half the code.
And more now you can't rely on http error codes, even if you can use them.
Then debugging is difficult, because messages are sent in binary format, and with no middleware to conditionally plug in, it's even more painful. Yeah you could write a dedicated service listening to >.* and printing deserialized JSON payloads but that's also extra work.
However dropping the HTTP overhead leads to a lot more throughput.
For now I'll stay with HTTP.
The reason I experimented with it was there is a company which required NATS knowledge.
[0] https://docs.nats.io/running-a-nats-service/configuration/se...