Mirrord – mirror production traffic to your development environment(github.com)
github.com
Mirrord – mirror production traffic to your development environment
https://github.com/metalbear-co/mirrord
43 comments
I don't see how you can use this for any applications which have state that's not common with the source of the request. When I was a SRE we used to do these things called "dark launches" which were a type of canary analysis where we'd mirror production traffic onto a build we wanted to promote. If the error rate breached the current error rate, the build was kicked back. It was a really nice system, and this reminds me of that.
The full use case we want to have is developer using staging as their development environment, sharing it with others. The developers would launch their locally developed services using mirrord, getting traffic from the staging environment and also having access to the same resources as the pod in the staging environment (env vars, file system, outgoing traffic) while being able to control each service with settings (use remote/local for each)
Can you give a few sentences on why someone would want to do this?
So there are a bunch of use cases, but the main one we think is during day to day development. mirrord can give you access to “cloud conditions” while you code, without having to go through CI and actually deploying, which a. makes your feedback loops a lot tighter and b. reduces the likelihood you’ll deploy something broken to the cloud (i.e. staging/production) environment and break it for other developers in the organization.
We wrote about it a bit more here if you’re interested: https://metalbear.co/blog/reintroducing-mirrord/
How can you process production requests without the matching data? Processing a user with UUID X taking action Y depends on those things existing in the DB.
I can't tell if this is brilliant or insane. Brilliant in that I see the potential of catching bugs. Insane in that you'd need to be really really really sure that the environment the requests are diverted to can't change production data. Including sending out e-mails or other communciations.
I can't tell if this is brilliant or insane. Brilliant in that I see the potential of catching bugs. Insane in that you'd need to be really really really sure that the environment the requests are diverted to can't change production data. Including sending out e-mails or other communciations.
So right now we focus on helping devs develop using their staging environments, but we do plan to get to production later on.
As Eyal described here (https://news.ycombinator.com/item?id=31634831) next features would be enabling network access, env var access and file system to the remote cluster as the impersonated pod. After enabling "primitive" access we would also provide "protection" features as of what you can really send/use/change.
How does this compare to Telepresence? https://www.telepresence.io/
First of all, mirrord runs on the process level. Telepresence either runs on your local machine, changing its network behavior or in a container making development less smooth than running a local process.
Second, mirrord's default is to duplicate traffic and not intercept it, so useful when you don't want to disrupt the cloud environment, want to work concurrently with other developers on the same service, etc.
Why would I use this instead of Istio mirroring?
AFAIK Istio mirroring is an infrastructure tooling for devops to mirror traffic between pods in k8s whilst mirrord mirrors traffic to your local ran service without deploying it or building it as a container. The first helps devops achieve traffic mirroring while mirrord helps backend engineers work on their apps faster and better
A trick I've used to do something similar in the past is to setup detailed API logs that are designed to be replayed elsewhere.
This involves more than just standard web access logs, since you also need to capture any POST bodies too in order to be able to replay them later against a test environment.
I wrote more about this here: https://simonwillison.net/2021/Apr/12/porting-vaccinateca-to...
This involves more than just standard web access logs, since you also need to capture any POST bodies too in order to be able to replay them later against a test environment.
I wrote more about this here: https://simonwillison.net/2021/Apr/12/porting-vaccinateca-to...
This could have GDPR and other legal issues.
Example: if names/emails are in use
Example: if names/emails are in use
This looks really neat. Haven't taken much time to look in depth, it it took me a while to find info on whether this was k8s or podman related. Maybe the docs could make it a little more obvious. (for those wondering it's k8s)
Would mirrord actually work with podman? It would be pretty neat to setup something akin to a canary deployment
Would mirrord actually work with podman? It would be pretty neat to setup something akin to a canary deployment
It's literally in the first sentence of the README:
> mirrord lets you easily mirror traffic from your Kubernetes cluster to your development environment.
That said, cool application, although I can see a lot of compliance and/or data protection problems when production data is leaving the (hopefully protected) production environment.
> mirrord lets you easily mirror traffic from your Kubernetes cluster to your development environment.
That said, cool application, although I can see a lot of compliance and/or data protection problems when production data is leaving the (hopefully protected) production environment.
This is a real concern for PCI, PII, and SOX services. If you have good service inventory you could gate this to only work for approved apps, but some live rewriting of form data could also be helpful. Faker on demand :)
Thanks for the feedback! One of the creators here :)
Our initial thought is to create a solution similar to what Lyft has done (https://eng.lyft.com/scaling-productivity-on-microservices-a...) to enable using staging as a dev environment and sharing it with many developers at once.
We do want people to use at us production later on, but right now we know there might be compliance/security issues.
Hey, one of the creators or mirrord here.
The technology behind mirrord is actually agnostic - what it does is wrap a local process and "proxy" it's API calls to a remote environment and remote traffic is also mirrored from there.
We tailored the solution for K8S to provide seamless experience, but if there's a need for another orchestration solution, we can consider adding support :)
Forgive my ignorance, web et al is not my specialty, but how does this work with HTTPS traffic? Do you install a root cert in the dev environment like Wireshark to be able to mirror agnostically? Or does there need to be application config (DNS change, internal routing changes) to make this work? If the latter, what's the value add here?
Our agent sniffs traffic in the same network namespace as the pod whose incoming traffic we mirror, so in most cases that would be after the HTTPS traffic is decrypted (by e.g. a load balancer, or a service mesh)
What would the benefits of mirrord be over something like the NGINX Ingress Controller with mirroring enabled?
Right now the focus of mirrord is for back end engineers, so the use case is backend engineers wanting to work on real traffic and real environment being able to do so without help of or change of infrastructure
The new Kubernetes gateway specification also supports mirroring out of the box.
I used goreplay in an earlier life, is this similar?
Somewhat similar. mirrord works live only (currently) but doesn't just replay the traffic, it wraps your process and duplicates connection to your local service as if it's running in the remote environment. We'll soon add filesystem access, environment variables and outgoing traffic enabling developers to work seamless on a remote environment.
Please don't copy production data to computers you connect to coffee shop WiFi.
Yeah, this feels like a data breach waiting to happen. Every security questionnaire I’ve ever filled out wants to know prod, staging, and dev are entirely separate.
Does mirrord not allow to connect via an encrypted protocol such as HTTPS?
If you copy prod data to a computer that's a public connection then any compromised app can exfiltrate the data. Keep data secure all the time, including when it's on a local machine. Even if the machine isn't compromised, someone could just mug you for it and then you have to start reporting the data breach. It would be... messy.
To be honest, while I imagine there are times when access to prod traffic might be useful, I suspect even having a tool like this would be an instant ISO27001 or ISO27002 audit fail.
To be honest, while I imagine there are times when access to prod traffic might be useful, I suspect even having a tool like this would be an instant ISO27001 or ISO27002 audit fail.
How does the attack vector that you describe differ from being connected to your office or home network? Once this kind of malicious app that you describe is running on your machine, it doesn’t matter any longer what kind of network you’re connected to.
Coffee shop WiFi is shorthand for "any connection that isn't locked down with all unusual ports blocked in both directions, common ports blocked for outbound connections, and monitored for unusual activity." Having a compromised machine shouldn't be the end of the world; you should have layers of security that all have to fail for something bad to happen.
If your approach to security is "I really hope this machine doesn't get compromised" then you're not doing a good enough job.
Also, as bmm6o says, keeping prod data in your secure, pentested prod environment avoids the problem in the first place.
If your approach to security is "I really hope this machine doesn't get compromised" then you're not doing a good enough job.
Also, as bmm6o says, keeping prod data in your secure, pentested prod environment avoids the problem in the first place.
Thanks, I’m now getting the point, too.
[deleted]
This looks really cool! Any plans to support rerouting traffic entirely, rather than just mirroring?
Yes! This is actually in development at the moment so should be out really soon. Along that same line, we're also planning to support routing of outgoing traffic back to the cloud environment. We're obviously thinking of both those features as more suitable for use with a staging environment rather than production.
oh wow, super curious to check it out once its released!
How does this work IRL for side-effectful services.for example, if somebody posts an order and it mirrors to my dev service, does that also hit my dev DB? Then if there is a live request to access a record detail page, does that expect my dev DB to have it? I’m just trying to wrap my brain about teams use tools like this in practice. Is it mostly for load testing or a better form of E2E?
Up to you: you’re going to be able to configure what works locally vs what works with the cloud environment. So for example, you could configure all read functionality (incoming traffic, DB reads) to work with the cloud, and all write functionality to work locally.
Further down the line, we’re planning to support Copy On Write for databases - so you’d read from the cloud, write locally, then read the records you wrote locally (and records you haven’t written, from the cloud).
Disclaimer: we’re still pretty early in the roadmap, so currently only support mirroring incoming traffic. But we’re moving fast :)
I would generally use this for e2e testing since it's too expensive to run a 100% mirror of a large production environment. Well, depends on your scale and whether your production costs are a significant factor in your total expenses - I guess if your production costs are tiny compared to revenue/dev time spent debugging, you could justify a full mirror. For something like an eshop where revenue >> production costs you could probably justify cloning out production multiple times.
Sampling is another useful tool but I don't see any sampling in the API for this. If you do sample though you could end up with inconsistent or divergent state if you just sampled a flat % of traffic, so in some contexts it makes sense for sampling to be determined per-user/tenant/etc. rather than per-request.
This is pod based so you could easily slice by pod. If you have a pod with a lot less traffic, more impactful traffic, or that is harder to debug (example: pod that periodically aggregates and exports metrics but you believe it's dropping data sometimes) you could just enroll that and have some kind of dev shard of a portion of the stack.
I could also see this being very useful if you believe it to be too big a liability to record actual user requests (or implement something special to scrub out sensitive stuff) or let devs peek too close. You have a way of investigating some classes of issues without necessarily looking at sensitive data or requiring extensive back and forth in a support case.
Sampling is another useful tool but I don't see any sampling in the API for this. If you do sample though you could end up with inconsistent or divergent state if you just sampled a flat % of traffic, so in some contexts it makes sense for sampling to be determined per-user/tenant/etc. rather than per-request.
This is pod based so you could easily slice by pod. If you have a pod with a lot less traffic, more impactful traffic, or that is harder to debug (example: pod that periodically aggregates and exports metrics but you believe it's dropping data sometimes) you could just enroll that and have some kind of dev shard of a portion of the stack.
I could also see this being very useful if you believe it to be too big a liability to record actual user requests (or implement something special to scrub out sensitive stuff) or let devs peek too close. You have a way of investigating some classes of issues without necessarily looking at sensitive data or requiring extensive back and forth in a support case.
Don't forget that you need to disable CSRF checks, because the randomly generated tokens from prod vs dev would be different.