High Expectations for Low Precision at Cern(nextplatform.com)
nextplatform.com
High Expectations for Low Precision at Cern
https://www.nextplatform.com/2017/07/12/high-expectations-low-precision-cern/
14 comments
It mentions GPUs. GPUs generally only offer low-precision floating point arithmetic. GPUs can do many more 32-bit floating point operations per-second than CPUs.
There must be a huge demand for running numerical workloads on faster kit (GPUs), but the attendant loss in precision can make it very difficult to get good results in high-energy particle physics. The idea here might be to develop numerical methods that make up for low-precision arithmetic yet preserve the speed advantage of using GPUs. If so, TFA is very poor indeed.
There must be a huge demand for running numerical workloads on faster kit (GPUs), but the attendant loss in precision can make it very difficult to get good results in high-energy particle physics. The idea here might be to develop numerical methods that make up for low-precision arithmetic yet preserve the speed advantage of using GPUs. If so, TFA is very poor indeed.
Low precision ML is less than 8 bits. 32-bit floating point is the highest precision used.
The Titan and Tesla NVIDIA cards are still built with 64 bit/double precision in mind with "only" a 50% performance reduction compared to FP32 in most of those devices. It is the GeForce gaming cards and the ones being targeted now for ML that are gimping FP64, where it is 1/4th or even 1/16th the speed of FP32.
The vast majority of "real" HPC still requires FP64 (only large industry categorized under HPC that I know is perfectly fine with FP32 is geospatial oil & gas stuff)... a valid LINPACK has a hard FP64 requirement.
The vast majority of "real" HPC still requires FP64 (only large industry categorized under HPC that I know is perfectly fine with FP32 is geospatial oil & gas stuff)... a valid LINPACK has a hard FP64 requirement.
[deleted]
This article confuses me. Machine learning needs less precision (for some reason), but the workloads that CERN performs are scientific computing, not ML. Or, are they doing ML and so need less precision?
> for some reason
When training a model you're typically following a gradient which may be very small so you want high precision. One a model is trained and you're just using it you can generally get by with very low precision, ultimately we're usually talking about placing some input vector in one or more binary categories and the affect of small signal variations higher up in the model generally don't amount to much w.r.t the final categorisations.
When training a model you're typically following a gradient which may be very small so you want high precision. One a model is trained and you're just using it you can generally get by with very low precision, ultimately we're usually talking about placing some input vector in one or more binary categories and the affect of small signal variations higher up in the model generally don't amount to much w.r.t the final categorisations.
The folk theorem of statistical computing [1] says that when you have computational problems, often there’s a problem with your model. So chances are that if your gradient vanishes with 32 bit floats, switching to 64 bit floats won't help much.
[1] http://andrewgelman.com/2008/05/13/the_folk_theore/
[1] http://andrewgelman.com/2008/05/13/the_folk_theore/
Cern does lots of data analysis where measurement precision is not essential at least in evaluation phase. Switching into higher precision may be needed only for the final calculations It's also possible to scale and normalize data to lower precision.
Most likely related:
John L. Gustafson presenting "Beyond Floating Point: Next Generation Computer Arithmetic" https://www.youtube.com/watch?v=aP0Y1uAA-2Y
With papers here: - UNUMS: http://ubiquity.acm.org/article.cfm?id=3001758 - Posits: http://johngustafson.net/pdfs/BeatingFloatingPoint.pdf
John L. Gustafson presenting "Beyond Floating Point: Next Generation Computer Arithmetic" https://www.youtube.com/watch?v=aP0Y1uAA-2Y
With papers here: - UNUMS: http://ubiquity.acm.org/article.cfm?id=3001758 - Posits: http://johngustafson.net/pdfs/BeatingFloatingPoint.pdf
There has been a lot of talk about lower precision and approximate computing over the past couple of years, which I hope will be recognized as a fad with John's (and to a lesser extent mine/REX Computing's) work... Why sacrifice precision/accuracy if you don't have to? Posit's are a great solution for this for a vast number of problems, and the Stanford lecture is great.
The thing about Posits in particular is, depending on the environment (e.g. es=3 for 32 bit Posits as referenced in the Beating Floating Point paper), you can actually have greater accuracy/precision and dynamic range than 64 bit IEEE floats while using fewer bits. The OP article here about CERN is doing something significantly worse, which is just using lower precision and taking the accuracy loss. While this may be acceptable in some applications, I would not want to losing any accuracy in the original data from expensive scientific experiments and using potentially inaccurate results for future calculations and assumptions.
The thing about Posits in particular is, depending on the environment (e.g. es=3 for 32 bit Posits as referenced in the Beating Floating Point paper), you can actually have greater accuracy/precision and dynamic range than 64 bit IEEE floats while using fewer bits. The OP article here about CERN is doing something significantly worse, which is just using lower precision and taking the accuracy loss. While this may be acceptable in some applications, I would not want to losing any accuracy in the original data from expensive scientific experiments and using potentially inaccurate results for future calculations and assumptions.
"last month after running experiments in which particles within the LHC detectors collide about 1 billion times per second, with each collision generating a petabyte of data."
1 billion petabytes per second haha
1 billion petabytes per second haha
I feel like this article is confusing numerical precision with "precision measurements". The latter typically means that the focus has moved from discovery of a given process, to measuring its properties with high precision, where high precision at LHC is often an uncertainty below 1%.
Also in NLP domain a precision reduction may be applied: https://arxiv.org/abs/1706.06363
Seems like the missing bit is talking about how CERN could benefit from low precision? But it's certainly not in the article.