FermiNet: State of the art approx of molecular orbitals(twitter.com)
twitter.com
FermiNet: State of the art approx of molecular orbitals
https://twitter.com/pfau/status/1169981588412817409
18 comments
> In short if I were a VC I would be _very_ keen I’m watching this field. There tremendous value hidden behind this general problem.
As someone who used to be in this very space and even tried to get a startup off the ground based on it, I can tell you with absolute certainty that this will lead absolutely nowhere.
The short of it is that literally no business will accept data that is generated this way until someone shows that every neural net model trained in this way produces solutions that are mathematically equivalent to a validated method.
At best it might be used as a filter step in some pipeline, but that's not going to have much of an effect, and certainly not something on which to bet the success of a startup.
As someone who used to be in this very space and even tried to get a startup off the ground based on it, I can tell you with absolute certainty that this will lead absolutely nowhere.
The short of it is that literally no business will accept data that is generated this way until someone shows that every neural net model trained in this way produces solutions that are mathematically equivalent to a validated method.
At best it might be used as a filter step in some pipeline, but that's not going to have much of an effect, and certainly not something on which to bet the success of a startup.
This approach is guaranteed to give better answers than pretty much anything from standard quantum chemistry. Its limitations are (i) computational cost and (ii) the fact that many material/drug design questions cannot be answered with quantum chemistry only.
The problem is that you don't know what the sources of error are. In methods like density functional theory (DFT), or other post Hartree-Fock calculations, the sources of error are well understood. We know where the predictions break down and we know what can be relied upon.
Methods like this are difficult to verify. You don't know where the weaknesses in the model actually are and you don't know what is reliable. This is an interesting idea but has limited application and, even if the model can be understood well-enough to determine the limitations, this will not replace methods like DFT due to the cost of the calculations.
DFT is imperfect due to the limitations of the functionals and basis-sets but we know what it does well and that is a lot. It is reliable when used by someone that understands the sources of error and how to apply the methodology to the the target system in the most appropriate way.
Methods like this are difficult to verify. You don't know where the weaknesses in the model actually are and you don't know what is reliable. This is an interesting idea but has limited application and, even if the model can be understood well-enough to determine the limitations, this will not replace methods like DFT due to the cost of the calculations.
DFT is imperfect due to the limitations of the functionals and basis-sets but we know what it does well and that is a lot. It is reliable when used by someone that understands the sources of error and how to apply the methodology to the the target system in the most appropriate way.
Are you familiar with quantum Monte Carlo? This is just variational QMC, a well established method, with a neural network as an ansatz. Traditionally QMC uses ad-hoc ansatzes anyway, so this is not different.
Also, I‘ve spent last seven years doing DFT calculations, and although sometimes one can explain the failures, more often than not it‘s just intransparent. QMC is actually in the core of DFT, because QMC calculations on the uniform electron gas have been used to parametrize LDA in DFT.
Also, I‘ve spent last seven years doing DFT calculations, and although sometimes one can explain the failures, more often than not it‘s just intransparent. QMC is actually in the core of DFT, because QMC calculations on the uniform electron gas have been used to parametrize LDA in DFT.
I'm not particularly familiar with QMC, although I do know of it.
I was just adding a comment in support of this not being a particularly revolutionary methodology because, although it may achieve spectacular results for certain systems, the limitations and mistakes of this kind of approach are completely hidden behind an opaque ML methodology of an NN.
ML has a place but NN are notoriously difficult to even grey-box and a black box model doesn't do much to actually advance the field. It certainly doesn't allow for a well-rounded assessment of failures.
As for the limitations of DFT, unless you are referring to convergence issues, I think you are completely wrong to claim that the issues with the method are not well-known and understood. We know precisely where the methodology has limitations and we also know how the functionals have been parametrised and we also know the assumptions / theoretical models upon which they are based and their limits. That is enough information to know where confidence can be placed.
I would also dispute that QMC is in the core of DFT just because it is used to parametrise LDA. As I am sure you are aware, LDA is not used for any reliable modelling. GGAs and Hybrids (And maybe meta-GGAs if we're feeling charitable...) are what make DFT a useful theory. Prior to that the results just sucked for the majority of systems!
I was just adding a comment in support of this not being a particularly revolutionary methodology because, although it may achieve spectacular results for certain systems, the limitations and mistakes of this kind of approach are completely hidden behind an opaque ML methodology of an NN.
ML has a place but NN are notoriously difficult to even grey-box and a black box model doesn't do much to actually advance the field. It certainly doesn't allow for a well-rounded assessment of failures.
As for the limitations of DFT, unless you are referring to convergence issues, I think you are completely wrong to claim that the issues with the method are not well-known and understood. We know precisely where the methodology has limitations and we also know how the functionals have been parametrised and we also know the assumptions / theoretical models upon which they are based and their limits. That is enough information to know where confidence can be placed.
I would also dispute that QMC is in the core of DFT just because it is used to parametrise LDA. As I am sure you are aware, LDA is not used for any reliable modelling. GGAs and Hybrids (And maybe meta-GGAs if we're feeling charitable...) are what make DFT a useful theory. Prior to that the results just sucked for the majority of systems!
I concur that using a neural net as a sub-block of the simulator is probably a bad idea, because NN are approximations and summing them over simulation-time is only going to result in worse approximations.
I'm quite bullish on the other hand for using NN in chemistry. If you use NN at a higher level (for example predicting which reactions will happen and finding pathways), it makes a lot more sense, it's akin having an experienced chemist. A tool like Chematica (database of chemical reactions to find pathways which got bought by Merk) can almost certainly benefit from neural networks where you can try to find similarly-shaped molecules that will have a similar behavior but be cheaper to produce.
I'm also bullish, on the use of all the NN machinery, for the use of simulators. NN frameworks are one way to be able to use efficiently large amount of compute. In fact there are kind of dual problems : In machine learning you try to minimize some energy function while in Hamiltonian Physics simulation you try to keep the energy function constant.
Making approximations in these equations (for example in the way to represent a field, summing in a different order, neglecting zeros), can often result in faster computation for the same accuracy. In fact, if you write the simulator as a Monte-Carlo sum, you can train a NN controller using the trajectory approximation, to do what you want to do, folding the exact simulation computation time inside the training loop of the NN.
I'm quite bullish on the other hand for using NN in chemistry. If you use NN at a higher level (for example predicting which reactions will happen and finding pathways), it makes a lot more sense, it's akin having an experienced chemist. A tool like Chematica (database of chemical reactions to find pathways which got bought by Merk) can almost certainly benefit from neural networks where you can try to find similarly-shaped molecules that will have a similar behavior but be cheaper to produce.
I'm also bullish, on the use of all the NN machinery, for the use of simulators. NN frameworks are one way to be able to use efficiently large amount of compute. In fact there are kind of dual problems : In machine learning you try to minimize some energy function while in Hamiltonian Physics simulation you try to keep the energy function constant.
Making approximations in these equations (for example in the way to represent a field, summing in a different order, neglecting zeros), can often result in faster computation for the same accuracy. In fact, if you write the simulator as a Monte-Carlo sum, you can train a NN controller using the trajectory approximation, to do what you want to do, folding the exact simulation computation time inside the training loop of the NN.
all I'd ask for is a few blind predictions on some reasonably interesting molecules, where no existing method can make an accurate prediction without a very expensive calculation (QM on supercomputers is common, you could easily generate the energies for a bunch of modest molecules from a diverse set).
I say that as somebody who has evaluated VC pitches for O(n) approximations of QM as a startup idea.
I say that as somebody who has evaluated VC pitches for O(n) approximations of QM as a startup idea.
This sounds like a good way to approach it. If there’s a system that is currently impossible to make decent predictions on then any partially accurate result could be beneficial. I’d be suspicious of someone claiming O(n) on QM however. They’d have to be very simple estimates, not sure how useful they’d be.
To your parents comment, it’d depend heavily on the molecules and systems under study and startup goals.
> The short of it is that literally no business will accept data that is generated this way until someone shows that every neural net model trained in this way produces solutions that are mathematically equivalent to a validated method.
This would seem to be wrong approach this early on...
> At best it might be used as a filter step in some pipeline, but that's not going to have much of an effect, and certainly not something on which to bet the success of a startup.
I don’t think a startup based on providing ‘QM simulation as a service’ would work very well and be fraught with issues. However, many industries could make significant usage of a pipeline filtering possible solutions which could be validated experimentally or with more traditional methods.
IMHO, to work with this a startup would need to be a vertically integrated company solving a specific class of problems (say batteries). Even then the current State of the Art still seems a few years off before I’d want to do a startup in the area, though it’s much closer with the recent results.
To your parents comment, it’d depend heavily on the molecules and systems under study and startup goals.
> The short of it is that literally no business will accept data that is generated this way until someone shows that every neural net model trained in this way produces solutions that are mathematically equivalent to a validated method.
This would seem to be wrong approach this early on...
> At best it might be used as a filter step in some pipeline, but that's not going to have much of an effect, and certainly not something on which to bet the success of a startup.
I don’t think a startup based on providing ‘QM simulation as a service’ would work very well and be fraught with issues. However, many industries could make significant usage of a pipeline filtering possible solutions which could be validated experimentally or with more traditional methods.
IMHO, to work with this a startup would need to be a vertically integrated company solving a specific class of problems (say batteries). Even then the current State of the Art still seems a few years off before I’d want to do a startup in the area, though it’s much closer with the recent results.
That seems interesting, mind emailing me at tapabrata(underscore)ghosh(at)vathys(dot)ai?
To me there are two ways to look at this. Either the laws of physics as we have them are too general when compared to what can be encountered in reality (in other words, actual reality is simple enough that a Turing machine approximated by logic circuits can manage well enough at finding a description) or, models found by fitting data are too specialized, ignoring subtleties not captured by loss functions acting on the data they were trained on.
Any physical model gained by fitting data which purports to be faster than those based on a computable approximation of the laws of physics is so constrained. There is room to maneuver however. If the model being replaced has a limited and known domain of applicability due to approximations made for tractability, a fitted model with suffciently large capacity and expressiveness will for sure improve things.
It's just that it's unlikely to be generally applicable without violating what we know about physics, which is why I am skeptical of the latter part of your post.
Any physical model gained by fitting data which purports to be faster than those based on a computable approximation of the laws of physics is so constrained. There is room to maneuver however. If the model being replaced has a limited and known domain of applicability due to approximations made for tractability, a fitted model with suffciently large capacity and expressiveness will for sure improve things.
It's just that it's unlikely to be generally applicable without violating what we know about physics, which is why I am skeptical of the latter part of your post.
>Any physical model gained by fitting data which purports to be faster than those based on a computable approximation of the laws of physics is so constrained.
Constrain the model so that there aren't any superfluids, semiconductors, plasmas, metals or Bose-Einstein condensates and you can still simulate any medicine I know of.
Constrain the model so that there aren't any superfluids, semiconductors, plasmas, metals or Bose-Einstein condensates and you can still simulate any medicine I know of.
Assume you can then simulate the folding of full-sized proteins, too?
Given enough time and a well-considered methodology, yes. I know two groups that work directly upon this and have great results.
Lithium?
Yeah, take metals out of the equation and it probably a lot more likely.
You are still going to need things like: Sodium, Potassium, Magnesium, Iron, etc.
You are still going to need things like: Sodium, Potassium, Magnesium, Iron, etc.
I meant metals in the solid-state physics sense, with the oceans of electrons and band gaps and stuff. You can keep your monatomic ions (which are crucial for biology in many ways).
For those of you who avoid clicking twitter links like me, here's the arxiv link:
https://arxiv.org/abs/1909.02487
For those with a background in ML / AD, but without background in quantum mechanics: there are some important observations that may be useful in other domains in supplementary materials sections B and C of the paper
https://arxiv.org/abs/1909.02487
For those with a background in ML / AD, but without background in quantum mechanics: there are some important observations that may be useful in other domains in supplementary materials sections B and C of the paper
Still running true ab initio QM simulations of a few atoms can take months on a single computer (never had a chance to run simulations on a cluster or GPU). DNN’s however have the ability to find higher dimensional patterns difficult for humans to find but which could significantly speed up QM simulations.
Currently doing QM simulations of chemical reactions for any number of reactions is in feasible but if work like FermiNet could make it feasible for small teams to simulate more complex chemical reactions it could open up an entire field of chemical/industrial processes to startups. As in you could reasonably simulate chemical processes sufficiently to optimize current process or find entirely novel reactions. This would significantly reduce the capital expenses most research in these areas require.
In short if I were a VC I would be _very_ keen I’m watching this field. There tremendous value hidden behind this general problem.