What do Data Scientists use to train models fast?

6 pointsby moridin00711 anni fa7 comments

i'm training a machine learning model using SVM in python and it took aages for it to happen on my local machine. (with 10% of the data that i have) i'm getting a 80-90% correct prediction score on the same subjects data so now want to add in the rest of the data. (11 more subjects)

i thought of offloading it to my ec2 instance but i'm on a budget so cant just take a 30CPU instance.. on top of everything the code just uses 1 CPU at 100% always so i'm not sure about how effective it would be.

what do you guys use to train these models?

7 comments

syllogism11 anni fa

Speed comes from two things: implementation and algorithm. Algorithmically, the way to learn quickly is to use some sort of stochastic gradient method, i.e. learn from examples one-by-one, as opposed to as a batch.

As far as implementation goes, you need dense arrays. A native Python implementation will usually be lists of Python objects, which is very slow.

If you just need an SVM implementation, libsvm is pretty good. I'm assuming you need a non-linear kernel. If you're using a linear kernel then there's not really a difference between SVM and MaxEnt (well, there is but not much).

If your data is very sparse then there aren't many general-purpose implementations that are any good. The scipy.sparse module has some key stuff implemented in pure Python, and doesn't interoperate properly with the rest of the PyData ecosystem. I had to implement my own sparse data structures, in Cython.

facorreia11 anni fa

One approach is to convert the code to use parallelism. For an example of how to do it in Python using joblib see this article: http://blog.dominodatalab.com/simple-parallelization/

Even if you can't afford a 32-core instance, you might get to use 4 cores in your laptop.

syllogism11 anni fa

The problem's pretty obviously the Python implementation...Throwing more cores at it isn't really going to help.

Is it even easy to parallelise SVM training?

moridin00711 anni fa

so the solution would be move from python to go? like python isn't that good for ML algorithms?

syllogism11 anni fa

Python's fine if the heavy lifting is in a C extension. I use Cython for this; others prefer just numpy, maybe something like numba. But you can't just have a Python list of floats.

Java, C++, Scala and Julia are all popular choices. Go is probably fine too, although I know less about it.

Lofkin11 anni fa

Actually list support in numba is WIP

rajacombinator11 anni fa

How much data and how long are you talking about? If it fits in memory, then the slowness is likely due to other coding errors causing a bottleneck, not the SVM training. (Unless you wrote that as well.)