Probabilistic Programming(cs.cornell.edu)
cs.cornell.edu
Probabilistic Programming
https://www.cs.cornell.edu/Courses/cs4110/2016fa/lectures/lecture33.html
29 comments
My feeling is everything you could do this way is either a) trivial anyway, or b) is making use of all the active research on MCMC sampling (which generally gives you DSLs to express a subset of all possible programs, which they are optimized for).
Where in this picture does formulating the problem in a Turing-complete language help anyone? If making the "compiler"/analyser is (for practical/interesting purposes) impossible anyway?
I have spent several years in my PhD trying to map out just one probability distribution and make it computationally tractable. Postulating the existence of a "magic" analyser that work is redundant, which would be nice, but I don't see it ever happening.
Granted it has a billion input variables, but so does any "real-world", non-toy computer program.
TL;DR if the problem fits stabdard MCMC or rejection sampling just do it. If not then a magic performant analyser is going to be hard to build. I hope I am wrong :)
Where in this picture does formulating the problem in a Turing-complete language help anyone? If making the "compiler"/analyser is (for practical/interesting purposes) impossible anyway?
I have spent several years in my PhD trying to map out just one probability distribution and make it computationally tractable. Postulating the existence of a "magic" analyser that work is redundant, which would be nice, but I don't see it ever happening.
Granted it has a billion input variables, but so does any "real-world", non-toy computer program.
TL;DR if the problem fits stabdard MCMC or rejection sampling just do it. If not then a magic performant analyser is going to be hard to build. I hope I am wrong :)
(I'm the author of the lecture notes.) I totally agree, and this is an underappreciated view in the PP community. The hype says "you just write your model; you'll get inference for free," but that obscures that current languages do nothing to help with cooking up an efficient inference algorithm for a particular situation. That's still a hard problem. When the only inference algorithm anyone actually uses is some MCMC variant, I think that's a broken promise.
There's some chance we'll be able to find clever new ways to derive better inference algorithms in specific situations using program analysis, but I wouldn't hold my breath. And it's impossible, of course, for anything like that to work for every possible program in a Turing-complete language.
There's some chance we'll be able to find clever new ways to derive better inference algorithms in specific situations using program analysis, but I wouldn't hold my breath. And it's impossible, of course, for anything like that to work for every possible program in a Turing-complete language.
Thanks for confirming my suspicion. So would it be fair to say that PP is an interface to statistics, so that those who know how to code but not advanced stats can do stats without learning it? (On simple problems)
As it stands, I'd agree with that---but the characterization definitely depends on who you talk to. I might be in the minority among "PP people," since the hype is still going strong and some smart people seem to see more potential than I (and you) do.
If you're interested in a probabilistic prolog, there is ProbLog [0].
[0]: https://dtai.cs.kuleuven.be/problog/
[0]: https://dtai.cs.kuleuven.be/problog/
>> It's more like Prolog: sure, it's a programming language—but it's not the
right tool for writing full-fledged software.
Uh oh. That's dangerous talk.
The problem is that once you start writing the kind of program that Prolog is best at, in Prolog, then you really, _really_ don't want to go back and write the same thing in any other language. Because most of the time it's a huge pain, and you'll probably have to implement Prolog in that other language anyway (and it's going to be a slow and buggy Prolog, but without all the good bits).
This has actually harmed the adoption of Prolog in a roundabout way, I think. Programmers are (or, well, were ... in the distant past) worried they might be stuck with it and unable to write the everyday stuff they need (REST APIs and whatnot).
If PP languages are sold in the same way, as very niche products that are only useful for a specific kind of programming, then I don't think they are ever going to be widely adopted, not even from the people who could benefit from them. Which is to say: eventually, everyone.
Instead of risking being stuck with very valuable software written in a one-trick pony sort of language, much better to take the long road and write it all in a language like C# or Java, where it may look attrocious, but at least it's the same language as the rest of your application (and you can find plenty of warm bodies to throw at it once it starts going south, to boot).
Uh oh. That's dangerous talk.
The problem is that once you start writing the kind of program that Prolog is best at, in Prolog, then you really, _really_ don't want to go back and write the same thing in any other language. Because most of the time it's a huge pain, and you'll probably have to implement Prolog in that other language anyway (and it's going to be a slow and buggy Prolog, but without all the good bits).
This has actually harmed the adoption of Prolog in a roundabout way, I think. Programmers are (or, well, were ... in the distant past) worried they might be stuck with it and unable to write the everyday stuff they need (REST APIs and whatnot).
If PP languages are sold in the same way, as very niche products that are only useful for a specific kind of programming, then I don't think they are ever going to be widely adopted, not even from the people who could benefit from them. Which is to say: eventually, everyone.
Instead of risking being stuck with very valuable software written in a one-trick pony sort of language, much better to take the long road and write it all in a language like C# or Java, where it may look attrocious, but at least it's the same language as the rest of your application (and you can find plenty of warm bodies to throw at it once it starts going south, to boot).
What sort of programs is Prolog good at expressing? I've looked at it a bit, but can't really think of any good use cases.
Anything that needs pattern matching or search.
As a for instance, grammars (in Prolog, a grammar can be expressed as a first order logic theory, to be proved by the compiler; therefore, a program).
As a for instance, grammars (in Prolog, a grammar can be expressed as a first order logic theory, to be proved by the compiler; therefore, a program).
If this tickles your fancy check this out
http://dippl.org
you implement your own universal probabilistic language embedded in js.
http://dippl.org
you implement your own universal probabilistic language embedded in js.
https://projects.csail.mit.edu/church/wiki/Church can be used in the browser via https://probmods.org/play-space.html
This is really cool and would definitely be something I would've loved to study if I had ever went into academic computer science. Shameless plug: a few years ago (actually, more like 6), I wrote a for-fun (zero-instruction) probabilistic VM to mess around with[1][2].
[1] https://code.google.com/archive/p/zeded/
[2] https://github.com/dvx/zeded
[1] https://code.google.com/archive/p/zeded/
[2] https://github.com/dvx/zeded
google says: 404. That's an error. The project zeded was not found.
Heh, I just started doing some searching on probabilistic programming about an hour ago. Funny how that works.
Since it's on subject here - anybody know of any good clojure libraries or resources for probabilistic programming? Anglican definitely looks good, but their intro pages use some kinda strange workflows.
Since it's on subject here - anybody know of any good clojure libraries or resources for probabilistic programming? Anglican definitely looks good, but their intro pages use some kinda strange workflows.
Anglican dev here. We always welcome feedback. Could you explain what you find strange about the workflows?
We taught a summer school on Anglican last August. The materials are available online:
https://bitbucket.org/probprog/ppaml-summer-school-2016 http://www.ccs.neu.edu/home/jwvdm/talks/
We taught a summer school on Anglican last August. The materials are available online:
https://bitbucket.org/probprog/ppaml-summer-school-2016 http://www.ccs.neu.edu/home/jwvdm/talks/
Awesome! I'll definitely check these out.
(Disclaimer: I'm a sysadmin and hobbyist programmer, so my workflow might be a bit different from most.)
The strangeness I saw was mostly in from difficult to follow documentation and the use of gorilla, not necessarily Anglican itseld. As awesome as gorilla is, it's just another new thing to learn. Since I'm already trying to learn Anglican, learning Gorilla and fighting my muscle memory just adds noise to the learning process. It would be nice to have something similar straight within a REPL (similar to how incanter does it with `view`). It's not a problem exclusive to Anglican, though.
That said, the barrier for entry with a lot of Clojure - even the language itself (no, not just its parens) is like that, where frustration is by a thousand cuts.
(After I posted that comment I dug a bit deeper into Anglican and found that I really enjoy it :).)
(Disclaimer: I'm a sysadmin and hobbyist programmer, so my workflow might be a bit different from most.)
The strangeness I saw was mostly in from difficult to follow documentation and the use of gorilla, not necessarily Anglican itseld. As awesome as gorilla is, it's just another new thing to learn. Since I'm already trying to learn Anglican, learning Gorilla and fighting my muscle memory just adds noise to the learning process. It would be nice to have something similar straight within a REPL (similar to how incanter does it with `view`). It's not a problem exclusive to Anglican, though.
That said, the barrier for entry with a lot of Clojure - even the language itself (no, not just its parens) is like that, where frustration is by a thousand cuts.
(After I posted that comment I dug a bit deeper into Anglican and found that I really enjoy it :).)
I think right now for clojure, Anglican is the one furthest along. It took me some effort to get into it, with the documentation being a little bit all over the place, but I got it working for me.
Do remember that Anglican is a partial language in clojure, so you can't use all clojure functions (and I think none of the macros) within anglican's defqueries.
Do remember that Anglican is a partial language in clojure, so you can't use all clojure functions (and I think none of the macros) within anglican's defqueries.
Stan [1] has a modelling dsl, and multiple language interfaces. It also uses a state of the art, Hamiltonian Monte Carlo Markov Chain model which is significantly faster than other MCMCs.
A good introductory tutorial video is [2]
[1] http://mc-stan.org/ [2] https://www.youtube.com/watch?v=uSjsJg8fcwY
[1] http://mc-stan.org/ [2] https://www.youtube.com/watch?v=uSjsJg8fcwY
If you like to try a probabilistic programming language with Lisp/Scheme syntax, there's Church https://projects.csail.mit.edu/church/wiki/Church
Wow this is really interesting and might be really helpful for a problem I'm working on now. I'm wondering though, what is the advantage of using a whole new programming language rather than say, a Python library?
There are packages that are just Python libraries, like PyMC3: http://pymc-devs.github.io/pymc3/ It allows you to specify a Probabilistic Program with a few lines of Python code (x ~ Normal(0, 1) becomes x = pymc3.Normal('x', 0, 1)) and then run automatic inference on whatever model you dream up, it's very powerful. I always found the approach of inventing a new language to specify Probabilistic Programs like Stan does a bit odd. There, you specify the model as a string and can't interact with it or debug if something goes wrong. Disclaimer: I'm a PyMC3 developer.
>I'm wondering though, what is the advantage of using a whole new programming language rather than say, a Python library?
The idea is that if you can write the program, you can sample from the generative model. Most libraries like Stan don't really provide a "Turing-complete" probabilistic language.
Daniel Roy, IIRC, studies precisely why or when you want this kind of "completeness", as well as other topics in PPL theory.
The idea is that if you can write the program, you can sample from the generative model. Most libraries like Stan don't really provide a "Turing-complete" probabilistic language.
Daniel Roy, IIRC, studies precisely why or when you want this kind of "completeness", as well as other topics in PPL theory.
For those who are interested in learning more about Probabilistic Programming:
http://probabilistic-programming.org/wiki/Home
http://probabilistic-programming.org/wiki/Home
I'm aware of Stan - does it cover what probabilistic programming is about or is there more to it?
There's more to it. Stan does models, as akin to statistical models. Workflow: specify model - run - evaluate results. Within that sphere they f.e. don't support all types of variables, but are fast, well-documented and under active development.
Outside of that sphere there is even more: you can make any program probabilistic / learning with something like PyMC that embed PP in a regular environment. Think bots, agents, UIs ...
Outside of that sphere there is even more: you can make any program probabilistic / learning with something like PyMC that embed PP in a regular environment. Think bots, agents, UIs ...
PyMC is roughly equivalent to Stan. I think what you're getting at is more the applications of these statistical models. Stan can be integrated into Python via PyStan.
For example, if your program calls rand(3) as part of the work it's intended to do—as in a cryptographic key generator...
Bangs head into desk repeatedly :-(
(Yes, I know that security wasn't the main point of the post, but still...)
Bangs head into desk repeatedly :-(
(Yes, I know that security wasn't the main point of the post, but still...)
If you like functional programming, probabilistic functional programming might also be interesting for you. I'd recommend reading Practical Probabilistic Programming with Monads [1], and checking out monad-bayes [2] which builds on the ideas discussed in the paper.
[0]: https://github.com/hakaru-dev/hakaru
[1]: http://mlg.eng.cam.ac.uk/pub/pdf/SciGhaGor15.pdf
[2]: https://github.com/adscib/monad-bayes