one thing i have seen with testcontainers (been a user for a few years) is the ergonomic SDKs that they have
especially in languages like golang, it makes spinning containers up/down, accessing the ports (eg: a mongodb container for some e2e test flow) super trivial - its like a nicety layer on top of vanilla docker (w/ the cost of including their sdk in your test build process)
yes, 100% can be done using docker directly or docker rest api (and def doesn't make sense to migrate if you have already made an investment in an in-house framework that doesn't require much upkeep)
its not necessarily about the privacy layer per se. the workflow i was ideating over is as follows:
1. spin up a production-equivalent database (eg: mysql container instead of prod RDS)
2. point a process/binary (maybe a simple container) to:
-- source db (RDS)
-- sink db (mysql container)
-- transformation function (that may use gen AI, etc) to seed sink db with synthetic/anonymized data [there may be some parallel process to enable testing of this transformation function]
3. profit (use this for dev etc)
Key over here would be speed in step (2) if the entire pipeline were to run end-to-end on-demand.
do you have some examples of using SDV to achieve this? highly possible that there's already something in the docs that I have missed
do you have some thoughts on how sdv-dev type projects can be used to start populating, say, a database (eg: mysql running in a container)
i've looked into this space a bunch (eg: Gretel, Tonic, etc) and there doesn't seem to be a good solution that works end-to-end
Privacy Dynamics is quite cool but ideally I'd like something super lightweight that can get pointed to a source db of some sort and then write to a sink (maybe applying a transformation layer in the middle)
yea - we had that too (good for understanding how laptop tooling worked, and what areas were starting to show latencies and therefore, needed to be worked on)
yea, great for small projects but no good when you're trying to expand into enterprise capabilities -- I have to get one tool for dev, have that config diverge from CI, and then from staging
then i have to hire a large devops team to manage it all -- super inefficient
This is really not true - a poor implementation shouldn't dictate the direction of the "remote development" space. Sure, IT and security have their requirements, but primary requirement is to make the developers happy and provide them with reproducible dev environments.
I used to work at Uber and what we ended up doing with devpod (https://www.uber.com/blog/devpod-improving-developer-product...) was to enable the popular IDEs to connect to these remote environments - all the dotfiles etc etc were persisted so it literally felt like the local IDE, just way faster. Admittedly, it costs a bunch of money to build internally, but there's a path to having people be happy with dev environments.
(we collected data on what IDEs to prioritize based on surveys)
taking the purchase pathway of:
1. need/demand
2. shortlisting
3. testing/evaluation
4. final decision
it seems like the dev teams has (and derivatively, the developers have) say in at least the testing/eval and the final decision stages. as cloud native becomes more mainstream, i'm sure we can see how this influence will affect each stage of the purchase pathway.
We’ve thought about this quite deeply at DevZero. Yes, the iOS developer experience can be quite hampered on the local env and requires users to frequently get beefy local laptops/machines. I saw this first hand at Uber from when crowdstrike was rolled out broadly.
We have support for AWS-based Mac VMs on DevZero but we don’t find our customers having their biggest issues related to iOS dev yet (we also target enterprise cos that have a vast diversity of tools, mostly backend and front end)
we’re not seeing cloud costs be too terrible at DevZero - with proper hibernation/suspension, cloud costs are ~$50-60/mo in the worst case
admittedly, we target only enterprise companies where the cost of loss of dev efficiencies is much higher
Say net cost to company for an engineer is $100k-$200k+. Even a net 10% savings over a year means $10k-$20k+ vs a $600-1k/yr investment (in worst case).
Security posture is also significantly improved, which admittedly is harder to assign a $ value to
fwiw, we’re solving this at DevZero with having multiple AZs - given that our platform centrally manages remote compute (hibernation etc), this is working great at making sure latencies are <20ms for users
Re: if a computer can run scripts in a pod, can’t you just run them locally?
It’s a construct of the order of operations.
One core issue with local is the variety of OS’ and local build tools that would fundamentally mess with the centralized scripts. Getting company-wide setup scripts to work on top of existing laptop config was a continuous challenge. Hence, having a consistent baseline (OS flavor, system-level packages) on top of which the company-wide “setup script” is added followed by “developer-customizations” seems to work great.
Central teams can manage the first couple of steps and individual user-specific configuration can be managed much better in a decentralized manner.
[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber]
I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback!
How we're looking at the space:
- IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers
- the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against.
The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here.
For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management*" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc).
When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode".
Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev.
Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring).
*Re: workload management, we added support for k8s** generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc).
**Say you have helm to deploy containers/pods to prod. We take those charts (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber]
I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback!
How we're looking at the space:
- IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers
- the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against.
The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here.
For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc).
When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode".
Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev.
Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring).
Re: workload management, we added support for k8s* generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc).
*Say you have helm to deploy containers/pods to prod. We look at that (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
things like terraform are great for the initial setup, but there is a lack for tooling around recurring management -- although, docker does help the story for having ephemeral environments quite a bit
it would be great to see developer environments be able to consume helm charts etc to create production-like environments but the overall sprawl around the variety of cloud-hosted or other SaaS products that are used in applications nowadays makes that quite hard - imo would be really nice to have a dev-tenancy variant for things like MongoDB, Datadog, etc
we're building cloud-hosted dev environments (choice of VM, container or full k8s) that engineers can connect to using their existing tooling (keep your IDEs, environment config, etc) but code in a production-like environment
from a pure software development perspective, the goal is to not just code in the cloud, but to also get a plethora of benefits around having dependent cloud resources, sampled message queues/databases, etc alongside these ephemeral dev environments
we're still in closed-beta but would love for folks to try it out and provide us some feedback: www.devzero.io