Hi HN! We're Jake, Jack, and Joe, and we're happy to share what we think is probably the simplest way to scale python code across thousands of computers in the cloud.
Coming from the Biotech world we were frustrated by how surprisingly slow and difficult it can be to iterate on large-scale batch processing pipelines, while contending with the security requirements imposed on us by the FDA.
Simple small changes necessitated rebuilding docker containers, waiting for batch pipelines to redeploy, and waiting for vm's to cold boot, a >5minute per iteration dev cycle, all just so I can see what error my code throws this time, then do it all over again!
Many other tools in the space were either too complicated for our bioinformaticians and data scientists to use, closed-source / managed only (a no-no with medical data), too difficult to setup and manage, or simply too expensive.
This is why we created Burla, an incredibly simple way to run python on thousands of computers, in any container, with any hardware.
It's open-source, can be setup with one command, and is simple enough for beginners to use.
Most importantly, Burla allows developers to iterate quickly on large scale pipelines, changes to code deploy in about 2 seconds, even with thousands of vm's running.
We believe that, in general, whether you're coding locally, or on a cluster of 1000 machines, infrastructure should update and react quickly, like under-a-second quickly. We should be able to iterate at the speed of thought, not at the speed my lambda function, batch workload, ETL-pipeline, or Kubernetes service takes to redeploy. We wrote more out long term goal here: https://docs.burla.dev/about
Coming from the Biotech world we were frustrated by how surprisingly slow and difficult it can be to iterate on large-scale batch processing pipelines, while contending with the security requirements imposed on us by the FDA. Simple small changes necessitated rebuilding docker containers, waiting for batch pipelines to redeploy, and waiting for vm's to cold boot, a >5minute per iteration dev cycle, all just so I can see what error my code throws this time, then do it all over again! Many other tools in the space were either too complicated for our bioinformaticians and data scientists to use, closed-source / managed only (a no-no with medical data), too difficult to setup and manage, or simply too expensive.
This is why we created Burla, an incredibly simple way to run python on thousands of computers, in any container, with any hardware.
It's open-source, can be setup with one command, and is simple enough for beginners to use.
Most importantly, Burla allows developers to iterate quickly on large scale pipelines, changes to code deploy in about 2 seconds, even with thousands of vm's running.
We believe that, in general, whether you're coding locally, or on a cluster of 1000 machines, infrastructure should update and react quickly, like under-a-second quickly. We should be able to iterate at the speed of thought, not at the speed my lambda function, batch workload, ETL-pipeline, or Kubernetes service takes to redeploy. We wrote more out long term goal here: https://docs.burla.dev/about
Github: https://github.com/Burla-Cloud/burla
More info: https://docs.burla.dev