End-to-End Encryption through Kafka(github.com)
github.com
End-to-End Encryption through Kafka
https://github.com/ockam-network/ockam/tree/develop/documentation/use-cases/end-to-end-encryption-through-kafka
16 comments
Does the threat model include malicious system software on the endpoints? Or slightly differently, have you looked at any mechanisms to provide encryption when in use.
We have a little bit and are watching the development of techniques like computing on Homomorphically Encrypted data, Multi-party Computation and Zero Knowledge Proofs with great interest.
For example, the credentials functionality in Ockam is using BBS+ signatures that enable selective disclosure of attested attributes. This has both privacy and bandwidth benefits when proving possession of a credential.
But we feel end-to-end encrypted communication is a prerequisite to building systems that compute on encrypted data. So we started with a focus on making end-to-end easy.
For example, the credentials functionality in Ockam is using BBS+ signatures that enable selective disclosure of attested attributes. This has both privacy and bandwidth benefits when proving possession of a credential.
But we feel end-to-end encrypted communication is a prerequisite to building systems that compute on encrypted data. So we started with a focus on making end-to-end easy.
Also, about malicious software at the endpoint ... Ockam Vaults can be backed by Hardware and Vault implementations are pluggable. So Key Storage and Cryptographic Operations can be delegated off the primary the processor.
So far we've built vault implementations for the Microchip ATEEC hardware which is popular in IoT circles. We plan to have Vaults for Cloud HSMs, Cloud Secret Stores, ARM TrustZone etc.
That said, if an attacker in the primary processor of an endpoint. We could protect agains key theft (with vault) ... but its very hard (maybe impossible) to protect against the attacker using the key until they are kicked out somehow.
So far we've built vault implementations for the Microchip ATEEC hardware which is popular in IoT circles. We plan to have Vaults for Cloud HSMs, Cloud Secret Stores, ARM TrustZone etc.
That said, if an attacker in the primary processor of an endpoint. We could protect agains key theft (with vault) ... but its very hard (maybe impossible) to protect against the attacker using the key until they are kicked out somehow.
Looking at the setup code here, this looks a lot like SSL reinvented over Kafka (instead of over TCP), which is not a bad thing, just the mechanism feels a lot like you've automated a tcpdump into the protocol by making the handshake packets persisted in the stream + committed to disks.
This somewhat makes sense if you're going to buy Kafka from a cloud provider without the ability to trust them, but the trust here is not very useful beyond securing transport trust.
E2e for Kafka usually shows up as a mechanism to encrypt each item differently, because that has a bigger use-case in it.
Kafka being an immutable log, there is no way to go back and delete data for a customer who asks for a data deletion or legally mandated GDPR data removals.
Being able to shred the keys is very useful if you have a destination or source attribute based keys - instead of purging data out of the kafka log you can throw away the key.
This somewhat makes sense if you're going to buy Kafka from a cloud provider without the ability to trust them, but the trust here is not very useful beyond securing transport trust.
E2e for Kafka usually shows up as a mechanism to encrypt each item differently, because that has a bigger use-case in it.
Kafka being an immutable log, there is no way to go back and delete data for a customer who asks for a data deletion or legally mandated GDPR data removals.
Being able to shred the keys is very useful if you have a destination or source attribute based keys - instead of purging data out of the kafka log you can throw away the key.
Perfect Forward Secrecy is pretty standard in TLS handshakes these days, so the persistence of messages should not be an issue.
This pattern of burying TLS inside of application payloads is popping up all over the place. People realize that securing the transport isn't the same as securing the application, which often hops over multiple transports.
This pattern of burying TLS inside of application payloads is popping up all over the place. People realize that securing the transport isn't the same as securing the application, which often hops over multiple transports.
"People realize that securing the transport isn't the same as securing the application, which often hops over multiple transports."
Exactly right. Application layer trust, with the transport and security layers abstracted away. This is important, since app developers typically can't control the transports from end to end, or at scale.
Exactly right. Application layer trust, with the transport and security layers abstracted away. This is important, since app developers typically can't control the transports from end to end, or at scale.
Great observations!
Data removal from Kafka for GDPR compliance is trivial in this setting. You just purge your keys. There are also key rotation and revocation protocols that are part of Ockam and we're making them simple to use from our libraries.
The underlying cryptography in Ockam SecureChannels comes from the Noise Protocol Framework (created by one of the authors of Signal Messenger's protocols). The Noise Framework protocols have many published security proofs and is gaining adoption in multiple secure systems. We are proposing an alternative to SSL/TLS but this alternative is well proven/understood in terms of it security guarantees.
Data removal from Kafka for GDPR compliance is trivial in this setting. You just purge your keys. There are also key rotation and revocation protocols that are part of Ockam and we're making them simple to use from our libraries.
The underlying cryptography in Ockam SecureChannels comes from the Noise Protocol Framework (created by one of the authors of Signal Messenger's protocols). The Noise Framework protocols have many published security proofs and is gaining adoption in multiple secure systems. We are proposing an alternative to SSL/TLS but this alternative is well proven/understood in terms of it security guarantees.
What are some use cases for this?
Some examples would be:
1. Alice is the controller of a machine in a factory, Bob is a remote worker controlling the machine from home. (there could be thousands of Alice, one Bob)
2. Alice is an internal system in Enterprise Data Center 1, Bob is a Microsevice in the Cloud
3. Alice is health monitoring device, Bob is a remote doctor's tablet. (there could 100s of thousands of Alice. On of Bob)
4. Alice is a connected car, Bob is a service in the cloud, that uses a third party vendor's managed Kafka to communicate with Alice and wants to guarantee secure software updates / machine learning model updates to Alice (there is a fleet of Alice).
etc.
Any use case where you're using third party services/intermediaries and want to keep the vulnerability surface of your application as small as possible.
1. Alice is the controller of a machine in a factory, Bob is a remote worker controlling the machine from home. (there could be thousands of Alice, one Bob)
2. Alice is an internal system in Enterprise Data Center 1, Bob is a Microsevice in the Cloud
3. Alice is health monitoring device, Bob is a remote doctor's tablet. (there could 100s of thousands of Alice. On of Bob)
4. Alice is a connected car, Bob is a service in the cloud, that uses a third party vendor's managed Kafka to communicate with Alice and wants to guarantee secure software updates / machine learning model updates to Alice (there is a fleet of Alice).
etc.
Any use case where you're using third party services/intermediaries and want to keep the vulnerability surface of your application as small as possible.
Instead of bidirectional messages using two topics, could you use messages in a single topic encrypted with something like PGP, S/MIME or JWE?
Great question!
We're thinking about Ockam Routing, Transports, Streams and SecureChannels as composable building blocks for end-to-end encryption. This allows us to provide end-to-end encryption along a route that may look like this:
Alice <=> Bluetooth hop <=> TCP hop <=> Kafka <=> TCP hop <=> Bluetooth hop <=> Bob
Data between Alice and Bob is end to end encrypted. This guide has some detail about how that works, if you're interested https://github.com/ockam-network/ockam/tree/develop/document...
Currently One Ockam Stream is backed by one Kafka topic. This is simpler, more composable and easier to think about so we started with this approach. In theory there could be a Bidirectional Stream that is backed by one topic, but we haven't focused on that yet.
Re: "PGP, S/MIME or JWE etc."
A secure channel protocols like Ockam SecureChannel or TLS are more comprehensive and have more security guarantees than simple encryption primitives like PGP or JWE. They provide guarantees like forward secrecy and resistance to Key Compromise Impersonation among other things
We're thinking about Ockam Routing, Transports, Streams and SecureChannels as composable building blocks for end-to-end encryption. This allows us to provide end-to-end encryption along a route that may look like this:
Alice <=> Bluetooth hop <=> TCP hop <=> Kafka <=> TCP hop <=> Bluetooth hop <=> Bob
Data between Alice and Bob is end to end encrypted. This guide has some detail about how that works, if you're interested https://github.com/ockam-network/ockam/tree/develop/document...
Currently One Ockam Stream is backed by one Kafka topic. This is simpler, more composable and easier to think about so we started with this approach. In theory there could be a Bidirectional Stream that is backed by one topic, but we haven't focused on that yet.
Re: "PGP, S/MIME or JWE etc."
A secure channel protocols like Ockam SecureChannel or TLS are more comprehensive and have more security guarantees than simple encryption primitives like PGP or JWE. They provide guarantees like forward secrecy and resistance to Key Compromise Impersonation among other things
What are some use cases / problems this solves?
Some examples would be:
1. Alice is the controller of a machine in a factory, Bob is a remote worker controlling the machine from home. (there could be thousands of Alice, one Bob)
2. Alice is an internal system in Enterprise Data Center 1, Bob is a Microsevice in the Cloud
3. Alice is health monitoring device, Bob is a remote doctor's tablet. (there could be 100s of thousands of Alice. On of Bob)
4. Alice is a connected car, Bob is a service in the cloud, that uses a third party vendor's managed Kafka to communicate with Alice and wants to guarantee secure software updates / machine learning model updates to Alice (there is a fleet of Alice).
etc.
Any use case where you're using third party services/intermediaries and want to keep the vulnerability surface of your application as small as possible.
2. Alice is an internal system in Enterprise Data Center 1, Bob is a Microsevice in the Cloud
3. Alice is health monitoring device, Bob is a remote doctor's tablet. (there could be 100s of thousands of Alice. On of Bob)
4. Alice is a connected car, Bob is a service in the cloud, that uses a third party vendor's managed Kafka to communicate with Alice and wants to guarantee secure software updates / machine learning model updates to Alice (there is a fleet of Alice).
etc.
Any use case where you're using third party services/intermediaries and want to keep the vulnerability surface of your application as small as possible.
The old school method is "if it's in the network it's trusted". Obviously (in retrospect) this is terrible.
The new thing is Zero Trust, where you have point to point authentication. Ockam gives you this at an application layer, which is critical if your application serves APIs across trust boundaries (ie: "this API can be called by user group A, this one from B).
Further, Ockam also enables a sort of tunneled trust where you can have applications A, B, and C, and B can be untrusted, and A can communicate with C across B. This unlocks a bunch of things like encrypted delegation (C can know that B has been delegated permissions from A).
Really cool stuff. Excited to see it evolve.