Launch HN: Gallery (YC S21) – On-demand environments on any cloud provider
22 comments
> We realized that a better approach would be to abstract away the specifics of each environment by using infra-as-code as middleware. Infra-as-code solutions like Terraform, CloudFormation, etc. can represent infrastructure thoroughly in a standardized fashion. The majority of people we talked to were maintaining their infrastructure using Terraform or something similar, and it was easy to build out pipelines for those who didn't by using Terraform under the hood. This was the key to what became Gallery.
The problem you're trying to solve definitely exists, don't get me wrong; but the level of effort required to integrate with your service sounds more like a high-touch consultant-level engagement (definitely sounds enterprise-y and expensive). I'm not sure how you can scale this as a self-serve SaaS business.
The problem you're trying to solve definitely exists, don't get me wrong; but the level of effort required to integrate with your service sounds more like a high-touch consultant-level engagement (definitely sounds enterprise-y and expensive). I'm not sure how you can scale this as a self-serve SaaS business.
You're right; distribution is one of the biggest challenges for a service like this. There's a bit of a tradeoff in my eyes; you can either support a few setups with seamless self-service, or you can support everyone but on a case-by-case basis. We think that the IaC approach bridges the two; our goal at the moment isn't to rush to fully hands-free self-service, but rather to optimize for the range of configurations we're compatible with and how seamlessly we can support clients on those stacks.
Using IaC helps a lot with the former, and getting experience onboarding different clients has helped with the latter - we're now familiar with common hangups (e.g. figuring out the data layer), and we've built processes into the product to get around those (e.g. immediately populating a test DB from a specified staging DB). We've seen progress in how quickly we can get people onto the platform; it now generally only takes a day, and we're bringing full self-service in sight as that duration decreases.
That being said, our broader goal is to hit the larger enterprise market, for which (as you mentioned) a higher-touch approach is warranted.
Using IaC helps a lot with the former, and getting experience onboarding different clients has helped with the latter - we're now familiar with common hangups (e.g. figuring out the data layer), and we've built processes into the product to get around those (e.g. immediately populating a test DB from a specified staging DB). We've seen progress in how quickly we can get people onto the platform; it now generally only takes a day, and we're bringing full self-service in sight as that duration decreases.
That being said, our broader goal is to hit the larger enterprise market, for which (as you mentioned) a higher-touch approach is warranted.
I've seen a couple of these, and I think much of the problem is that I need to duplicate my ops work (scripts to build environments, etc.) somehow for the ephemeral environments to be created. You've evaded that somewhat by using Terraform, but what if I use CloudFormation to deploy my stuff?
I'm wondering if the better way to solve this problem would be to use docker-compose files (or k8s definitions), since we all have them checked in already for use with local development. I really just need a way to fire the local environment up on a server so I can send someone a link to it that will work for a while (and not die when I close my laptop). I don't need the full onslaught of AWS (or GCP services) when I just want to send someone a link and ask them if this CSS looks right (or put the link in a PR), but I need it to work better than a picture.
I'm wondering if the better way to solve this problem would be to use docker-compose files (or k8s definitions), since we all have them checked in already for use with local development. I really just need a way to fire the local environment up on a server so I can send someone a link to it that will work for a while (and not die when I close my laptop). I don't need the full onslaught of AWS (or GCP services) when I just want to send someone a link and ask them if this CSS looks right (or put the link in a PR), but I need it to work better than a picture.
Thanks for your feedback! We've weighed these considerations quite a bit in choosing the right first step for the product.
The docker-compose approach is a valid angle — we’ve seen a couple products built around that workflow. A weakness that led us to our approach is that they generally lack support for projects that involve cloud-specific services. Most of the startups we’ve talked to already have some infra-as-code that describes their cloud infrastructure, and we’re rolling out support for CloudFormation/Pulumi very soon, so customers will be able to bring over infrastructure in any major format. As far as limiting the breadth of services spun up, we've found that removing unnecessary services from the IaC config provides reasonably good results. We plan on adding support for a Docker-compose approach soon, but we felt that starting with IaC would bring us better generalizability out of the gate.
The docker-compose approach is a valid angle — we’ve seen a couple products built around that workflow. A weakness that led us to our approach is that they generally lack support for projects that involve cloud-specific services. Most of the startups we’ve talked to already have some infra-as-code that describes their cloud infrastructure, and we’re rolling out support for CloudFormation/Pulumi very soon, so customers will be able to bring over infrastructure in any major format. As far as limiting the breadth of services spun up, we've found that removing unnecessary services from the IaC config provides reasonably good results. We plan on adding support for a Docker-compose approach soon, but we felt that starting with IaC would bring us better generalizability out of the gate.
Terraform (as you may well have already found out) is a blessing and a curse. It's great for instantiating infrastructure quickly and repeatedly, but what it's not great for is managing it going forward and streaming changes (it seems to break A LOT when there is drift from managed infrastructure vs. real life infrastructure).
Do you guys plan to handle managing instances created via terraform code via Gallery, or is it strictly a tool to instantiate environments and add triggers around it?
Like I'm thinking of cases where you might want to edit your existing env created by Gallery with other stuff that other teams may have worked on and merged onto prod. If I edit the config, there could be a lot breaking changes
Do you guys plan to handle managing instances created via terraform code via Gallery, or is it strictly a tool to instantiate environments and add triggers around it?
Like I'm thinking of cases where you might want to edit your existing env created by Gallery with other stuff that other teams may have worked on and merged onto prod. If I edit the config, there could be a lot breaking changes
Absolutely - managing and syncing the state of IaC tools is core to the value of Gallery. Infrastructure templates can be edited without changing currently-active environments, and this is handled gracefully right now. Propagating template updates to running environments seamlessly is also on our roadmap.
Thanks for the reply. It certainly will be an interesting technical challenge to try and propagate changes efficiently. Speaking from experience (I've worked with Terraform a lot the past 2 years), it can be a tough nut to crack, so good luck!
I would add one more thing (and this is maybe for down the line): There's a use case for duplicating envs beyond just development environments. There's a lot of value in cases where the SaaS product itself requires on-demand environment generation. Where customers of it need staging instances so they can see what config changes are before they merge to production. Obviously in that case, you're treating an actual product like a terraform provider, and the API's of that product as resources. I would explore that down the line because there's a lot of value in a service like that.
FYI, I would love to get involved any way I can to help you guys grow this. Let me know how I can help! mailto:[email protected] if you're interested :)
I would add one more thing (and this is maybe for down the line): There's a use case for duplicating envs beyond just development environments. There's a lot of value in cases where the SaaS product itself requires on-demand environment generation. Where customers of it need staging instances so they can see what config changes are before they merge to production. Obviously in that case, you're treating an actual product like a terraform provider, and the API's of that product as resources. I would explore that down the line because there's a lot of value in a service like that.
FYI, I would love to get involved any way I can to help you guys grow this. Let me know how I can help! mailto:[email protected] if you're interested :)
I honestly don't get the value of this tool.
If I already have a parametrized set of Terraform scripts for creating an environment, what is stopping me from spinning up and tearing down my own ephemeral environments in my CI pipeline? This is what we're doing today and it works pretty well.
What is this tool actually providing? I feel like there's something fundamental I'm misunderstanding.
If I already have a parametrized set of Terraform scripts for creating an environment, what is stopping me from spinning up and tearing down my own ephemeral environments in my CI pipeline? This is what we're doing today and it works pretty well.
What is this tool actually providing? I feel like there's something fundamental I'm misunderstanding.
So if this requires a demo to use, are you writing the TF code manually?
We have reverse terraform support for some services (for example Cloud Run, multiple firebase services, etc.), where we automatically generate the Terraform (if you use services that are not supported, we do offer to manually create the terraform infrastructure as part of the onboarding process).
The reason we have a read-only demo is because of the friction required to set up an account (you have to link your cloud account and repositories); we felt that the read-only account was the easiest way to take a look at the features without needing to go through the account creation process.
The reason we have a read-only demo is because of the friction required to set up an account (you have to link your cloud account and repositories); we felt that the read-only account was the easiest way to take a look at the features without needing to go through the account creation process.
> we automatically generate the Terraform
This is the real product I'd pay for. The networking world has plenty of awesome tools that let you just log in to prod, make the change you need, and an automated system comes along later and dumps the configs to source control and documents the changes. Let engineers move quickly and change stuff in prod, you update the terraform and keep everything synced in dev/test/etc.
This is the real product I'd pay for. The networking world has plenty of awesome tools that let you just log in to prod, make the change you need, and an automated system comes along later and dumps the configs to source control and documents the changes. Let engineers move quickly and change stuff in prod, you update the terraform and keep everything synced in dev/test/etc.
self-removed
I get how tempting it is to hijack someone else's launch thread, especially when you've been working in the same space for a long time, but you're overdoing it. When I looked in on this thread, 3 of 7 comments were yours.
It's maybe fine to post one comment about your competing service—maybe. (I'm not even sure about that, to be honest, but people do it.) Pushing the envelope beyond that is too much.
Btw, this applies to all startup launches, not just YC companies, and we moderate other threads exactly the same way when we see this dynamic. From my perspective it's about the kind of community we want HN to be.
Non-launch threads are different of course. I'm sure your work is very good and you're completely welcome to participate on HN.
I've detached this comment from https://news.ycombinator.com/item?id=29166416.
It's maybe fine to post one comment about your competing service—maybe. (I'm not even sure about that, to be honest, but people do it.) Pushing the envelope beyond that is too much.
Btw, this applies to all startup launches, not just YC companies, and we moderate other threads exactly the same way when we see this dynamic. From my perspective it's about the kind of community we want HN to be.
Non-launch threads are different of course. I'm sure your work is very good and you're completely welcome to participate on HN.
I've detached this comment from https://news.ycombinator.com/item?id=29166416.
Got it. Not my intent to hijack this. I'll delete.
I'm trying to share that much of what is described in the post and discussion is currently being formalized - and there are different interpretations and technical solutions that various teams including what gallery is proposing.
I'm trying to share that much of what is described in the post and discussion is currently being formalized - and there are different interpretations and technical solutions that various teams including what gallery is proposing.
Sorry, are you part of Gallery, or are you answering questions on this Launch from the perspective of a separate, but similar, company?
Hi Jagath and Vignesh,
What you're proposing here sounds like it falls under the Continuous Previews umbrella that we've been collaborating with others to define and refine via the Continuous Previews Manifesto.
Awesome to see others finding the same value proposition in the process. We think there's space for several interpretations of CP and we see value in the implementation you're proposing.
If you're interested in collaborating please reach out.
What you're proposing here sounds like it falls under the Continuous Previews umbrella that we've been collaborating with others to define and refine via the Continuous Previews Manifesto.
Awesome to see others finding the same value proposition in the process. We think there's space for several interpretations of CP and we see value in the implementation you're proposing.
If you're interested in collaborating please reach out.
Reference CP Manifesto https://cpmanifesto.org/
Uh. You should check out Urban Dictionary and see what "CP" means in the context of the internet.
trigger warning, nsfw, dragons etc: https://www.urbandictionary.com/define.php?term=cp
Engineers typically have a fixed number of cloud environments, including production, staging, and maybe some additional pre-production environments. They use these to preview features, share work with other team members, do manual QA/automated testing, and more. As the number of engineers grows, having a fixed number of cloud environments becomes a bottleneck. Teams end up queuing for access to staging and so on. Creating additional environments and keeping them in sync, though, is a major headache, especially when the environments are reasonably complex.
When this problem reaches a boiling point, teams either have to slow down feature development, test on production, or deal with building their own on-demand platform internally. That’s where we come in. We enable on-demand spinup of unlimited environments for quicker development. Engineers use these as environments-per-feature—in parallel—to share previews with QA/other engineers/product people, to create demo environments for clients with features that aren't in production, to run automated testing on ephemeral environments as part of their CI processes, and more.
Vignesh and I are engineers with experience in infra/research at Facebook/Apple/Microsoft. We met as roommates at Caltech. We got into YC with a very different idea (a way for stores to ask customers questions and offer instant recommendations/promotions based on their responses) but soon abandoned it (it was more exciting to us as a technical challenge than to stores as a practical solution) and found ourselves testing different ideas as quickly as we could.
During this period, right before every launch without fail, Vignesh and I would collide on staging: features that worked locally would break on prod infra. We tried to provision our own individual staging environments, but setting them up and keeping them in sync sucked up valuable bandwidth. The gold-standard workflow, in my mind, was the one-click “On-Demand” environment provisioning I’d had at Facebook, wherein I could click a button and instantiate a live feature preview. We found solutions that promised on-demand environment spin-up, but none of them worked for us; they were either incompatible with our stack (mostly built around App Engine), didn't interface with our cloud provider, or required too much overhead and finessing to set up.
This was the seed for our idea. Talking to larger startups, we realized that the inconveniences we faced were just a few of the major pains faced by bigger engineering teams with nascent DevOps; they often had brittle, scattered workflows around managing their environments and growing queues for features. They needed a solution to flexibly create, destroy, and replicate environments, through both ad-hoc and developer triggers like PR’s and CI builds. While there are a number of existing services that aim to simplify DevOps and/or provide easier workflows around environments, engineering teams we spoke to were either unable or unwilling to use them due to incompatibility with their specific setups.
We realized that a better approach would be to abstract away the specifics of each environment by using infra-as-code as middleware. Infra-as-code solutions like Terraform, CloudFormation, etc. can represent infrastructure thoroughly in a standardized fashion. The majority of people we talked to were maintaining their infrastructure using Terraform or something similar, and it was easy to build out pipelines for those who didn't by using Terraform under the hood. This was the key to what became Gallery.
Since every company manages their environments differently (cloud provider, use of K8s or infra-as-code, cloud-managed services, security/privacy layers), creating a product that can work with them all out-of-the-box is a challenge. Unlike other solutions on the market, Gallery isn't a Kubernetes orchestrator that replicates containerized environments; rather, we can represent entire cloud projects, including managed services (e.g. App Engine, Elastic Beanstalk, etc.), spin them up in sandboxed cloud projects, and automate their teardown as well.
To create cloud resources on your GCP/AWS accounts, we use Terraform as a middleware. For select cloud services, we can understand an existing project, generate the Terraform corresponding to it, and use it to spin up your resources. We allow users to link their own Terraform code to allow spin-up of more complex environments as well. When you trigger the creation of a new environment, we spin up a worker that applies the Terraform template and performs any post-deploy actions to seed the newly generated infrastructure (for example, copying over initialization data into databases). We store your Terraform state file and manage tearing down environments when they outlive their TTL, or deletion is triggered by a specific action (e.g. merging a pull request).
Once we spin up infrastructure, we pull application code from your app repositories, and allow you to specify build and deployment commands that target the newly created environment. This lets you use your current deployment scripts on Gallery almost as-is, for fast and straightforward onboarding. We have integrations with Github/Gitlab around the Pull Request/Merge Request flow, letting you create environments that track a branch whenever a PR is created.
We're a SaaS product, and we have pricing tiers based on the number of concurrent environments. We’ve prepared a live demo account that you can play around with as a read-only team member here: http://a.getgallery.co/teams/14. I've also prepared a demo video of how to set up ephemeral environments per-PR in just a couple of minutes (https://www.loom.com/share/26165ea69f0d4b7b974019bdf72e5d11). You can see our docs here: docs.usegallery.com.
Thank you so much for reading! We’d love to hear your thoughts, ideas, and experiences around the problem we’re tackling and the solution we’re proposing!