DevOps in Academic Research(mattsegal.dev)
mattsegal.dev
DevOps in Academic Research
https://mattsegal.dev/devops-academic-research.html
10 comments
Great stuff, wonder how he convinced the team to drop the "for free" academic HPC/Slurm mess in favor of costly cloud computing. That is my biggest hurdle working at a university right now.
mhm... having skimmed through the article, the only reason (imho) is not wanting to wait for hpc/slurm to schedule your jobs... all the other stuff, could have been done with university resources. They usually offer some shared drive space and are often forthcoming in supporting special needs (at least from my experience).
Having used both AWS Spot Instances for compute jobs, as well as hpc resources (slurm), the amount of work to getting it run is comparable... Except that the aws stuff quickly gets expensive when needing thousands of cpu hours per "experiment", and money is usually tight at universities ;-). Some ansible scripts automate basically everything away for myself. But using the tool you know is probably not always a bad idea.
Having used both AWS Spot Instances for compute jobs, as well as hpc resources (slurm), the amount of work to getting it run is comparable... Except that the aws stuff quickly gets expensive when needing thousands of cpu hours per "experiment", and money is usually tight at universities ;-). Some ansible scripts automate basically everything away for myself. But using the tool you know is probably not always a bad idea.
Thanks! AWS was costly compared to the free HPC cluster but it was sub $1000 per month and our bottlenecks were person-hours rather than money. We had a lot of work to do. As some have mentioned there wasn't a strong technical argument for switching to AWS, it was more about productivity with those tools. I'd built up a bit of credibility with the team at that point and was like "I can make this better" and my boss was said "ok cool". As I mentioned in the article the team was very open to try new things.
I can see an alternate timeline where I set up Buildkite in AWS to run Slurm jobs on the HPC cluster, rather than using EC2 spot instances. Using Buildkite + S3 were probably the more important infra changes.
I can see an alternate timeline where I set up Buildkite in AWS to run Slurm jobs on the HPC cluster, rather than using EC2 spot instances. Using Buildkite + S3 were probably the more important infra changes.
A lot of companies are donating computing power for COVID-19 research, so it's possible they have a deal of some kind worked out with Amazon.
> That is my biggest hurdle working at a university right now.
We took the opposite approach and assembled our own equipment, which we self host and manage. The cost savings we're significant, and what we lose doing admin and maintenance is saved by having the flexibility and freedom to code and run what we want, when we want.
> That is my biggest hurdle working at a university right now.
We took the opposite approach and assembled our own equipment, which we self host and manage. The cost savings we're significant, and what we lose doing admin and maintenance is saved by having the flexibility and freedom to code and run what we want, when we want.
Onprem doesn’t get a lot of love these days but it’s significantly cheaper.
If you've got the data center for it. Otherwise it's great until you have to upgrade your AC, and punch a hole in the side of the building to get the new one brought in on a crane :-)
I built and ran a 1mw data center for about 7 years, it can be a headache but you can be ahead of the curve if you buy a DC that’s being vacated by someone moving to the cloud.
I paid 400k to buy about 4m worth of equipment from the previous tenant. UPS, STS, busbar distribution, 3n paralleling generators and 20,000 gallons of diesel.
I paid 400k to buy about 4m worth of equipment from the previous tenant. UPS, STS, busbar distribution, 3n paralleling generators and 20,000 gallons of diesel.
> wonder how he convinced the team to drop the "for free" academic HPC/Slurm
Probably because they are working with publicly available data so they don't need to worry about HIPAA/restricted data compliance. One of the main reasons for using the HPC/Slurm cluster is that IT (or the data compliance office) guarantees compliance with restricted data.
Probably because they are working with publicly available data so they don't need to worry about HIPAA/restricted data compliance. One of the main reasons for using the HPC/Slurm cluster is that IT (or the data compliance office) guarantees compliance with restricted data.
Couldn't IT do this with AWS as well? One of the few advantages of AWS are the compliant and auditable infrastructure patterns they provide.
I had a case where using the "for free" academic HPC cluster was so inconvenient that we used only in very few cases when we needed very large amounts of compute, and for any smaller tasks using the commercial cloud solutions saved enough time/salaries so that they were worthwhile even purely from cost perspective. Also, for academic purposes it was possible to get quite large amounts of cloud computing for free, so most users never bothered to even try to approach the academic HPC environment as it had no advantages whatsoever.