I've recently been in a rut and while I still have a lot further to go I've been feeling better over the last couple of days and here's what helped:
- going to therapy
- taking PTO
- going outside
- hanging out with friends
- cooking a meal
- going on a date
- going to LA and getting sun
- reconnecting with old friends
I'm not out yet but I feel like with some time and effort I can be. How did I do it? I think there are really two parts of this - helping others and asking for help. I've always enjoyed the first but never done the second and that was really holding me back.
ML training is not as easily parallelizable as the other problems that have been explored. I'm not familiar with SETI but I know this to be true for folding@home.
As you mentioned, ML training can be parallelized but this requires either model/data parallelism.
Data parallelism means spreading the data over many different compute units and then synchronizing gradients somehow. The heterogeneous nature of @home computing makes this particularly challenging, as you will be limited by the smallest compute unit. I've personally only ever seen data (and model) parallel done on a homogenous compute cluster (i.e. 8x GPUS)
For model parallelism, we split the model across different compute units. However, this means that you need to synchronize the different parts of the model together, which can get very expensive when you do it across the internet. If you have 8xGPUS on one machine, your latency is limited by PCIe instead of TCP/IP in a distributed @home cluster.
But I would say it's not impossible, someone clever could definitely figure it out.
I graduated in 2020 and was considering doing a PhD in NLP as I had already been doing research at a lab. I ended up joining a series A startup, trying to turn research into products.
I don't regret it much, as I was tired of being a broke student. Also I think the problems you run into in industry are more interesting. However, I do think my skills stagnated a bit and I learned more when doing research.
I like this post a lot, even if it's a somewhat contrived example. In particular I like his point about switch statements making it easier to pulled out shared logic vs. polymorphic code.
There's so much emphasis on writing "clean" code (rightly so) that it's nice to hear an opposing viewpoint. I think it's a good reminder to not be dogmatic and that there are many ways to solve a problem, each with their own pros/cons. It's our job to find the best way.
You negotiate around dilution by asking for additional equity grants, like you would for a raise / promotion. But in general dilution is a good problem to have because it means you are raising more money and growing.
Bay -> LA (Go Bruins!) -> NYC