> The issue isn't tooling, it's hardware resources and in some cases licencing.
Hardware resources are definitely an issue. That's why we generally recommend using remote development environments, which aren't as resource-constrained as the local dev machine. Making that comparably smooth to the local dev experience (e.g. for live reloading of services without rebuilding containers) needs some clever tooling (which is partly the reason we're building our product).
With production-like remote dev environments, you get the same capabilities as your CI environment, but can run test suites ad-hoc (and without having to spin them up and tearing them down for every test run).
There's no fundamental reason why CI environments should have capabilities that individual dev environments can't have—it's all a matter of automation in the end.
> The real challenges, like determining seed data etc is too project specific to be abstracted away.
Very much agree with that! The generic stuff (dependencies, parallel processing, waiting for things to spin up etc.) should be taken care of by the tooling, but without constraining the project-specific stuff (which is highly individual).
Related to this topic: When running integration/e2e tests, setting up the environment (all the required services, data stores etc.) in the right sequence, loading them with test data and so forth can be thorny to automate.
Good automation around preparing/provisioning the testing environment is a necessary companion to the testing tools/frameworks themselves.
Most commonly, fully-capable testing environments aren't available during the inner loop of development (where the dev setup can usually only run unit tests or integration tests for 1-2 services + a database).
Because of this, people tend to rely solely on their CI pipelines to run integ/e2e tests, which can slow things down a lot when one of those tests fails (since the write/run/debug loop has to go through the CI pipeline).
As an industry, I think we should start taking automation and developer productivity more seriously—not least when it comes to writing and debugging tests for complex distributed systems. The more we can lower the marginal cost of writing and running tests, the more effective our test suites will become over time.
Shameless plug: My company (https://garden.io/) is developing a framework and toolchain to bring the full capabilities of a CI pipeline to the inner loop of development, so that developers can efficiently run all/any test suites (including integ/e2e tests) in their personal dev environments.
We do this by capturing the full dependency graph (builds, deploys, tests, DB seeding etc.) of the system in a way that can power CI, preview environments and inner-loop development.
Sounds like Garden Core could be a great fit here.
The motivation behind Garden was that, like you, we had built our own home-grown kubernetes dev environments, but felt like there should be a polished, general-purpose framework + tool for this sort of thing.
Garden supports in-cluster building, using buildkit or kaniko.
This way, you don't need to have Docker or k8s running on your dev machine as you're working.
It also automates the process of redeploying services and re-running tests as you're coding (since it leverages the build/deploy/test dependencies in your stack).
We also provide hot reloading of running services, which brings a similarly fast feedback loop as with local dev.
The idea is to have a dev environment that has the same capabilities as the CI environment, and to be able to run any/all of your tests without having to go through your CI system (which generally involves a lot more waiting).
Hardware resources are definitely an issue. That's why we generally recommend using remote development environments, which aren't as resource-constrained as the local dev machine. Making that comparably smooth to the local dev experience (e.g. for live reloading of services without rebuilding containers) needs some clever tooling (which is partly the reason we're building our product).
With production-like remote dev environments, you get the same capabilities as your CI environment, but can run test suites ad-hoc (and without having to spin them up and tearing them down for every test run).
There's no fundamental reason why CI environments should have capabilities that individual dev environments can't have—it's all a matter of automation in the end.
> The real challenges, like determining seed data etc is too project specific to be abstracted away.
Very much agree with that! The generic stuff (dependencies, parallel processing, waiting for things to spin up etc.) should be taken care of by the tooling, but without constraining the project-specific stuff (which is highly individual).