Future of Statistical Programming(science.smith.edu)
science.smith.edu
Future of Statistical Programming
http://www.science.smith.edu/~amcnamara/FoSP.html
35 comments
It may be useful to take people's actual behaviour into account, instead of starting from the assumption that they must be idiots.
As it happens, no tool has been more successful at making this idea of "everyone being a coder" than the spreadsheet. It's an excellent (pun intended) gateway drug, with low barriers of entry, and a value proposition that was obvious enough to get accountants to spend 5 figures on a Mac when computers were something completely new.
Sure, the language is clunky. But it's apparently enough for most peoples' needs. I can't immediately think of any reasons for this almost moralistic criticism of Excel.
It seems like it would be a weekend project to convert Excel worksheets to sequential code in a universally-readable textfile, plus a separate CSV for the data. That would make sharing and reproducibility just as easy as easy as getting them to switch out their complete toolchain. But I have doubts that unwillingness to share code & data is actually a technical problems, instead of a lack of incentives/embarrassment of sharing ones' awful code/dreams of patents future/privacy issues.
I'm also not convinced that Julia as I have experienced it would be any better for long-term reproducibility. Last time I tried Julia, about every second package had a specific version that it required, and very few of them shared their preferences.
As it happens, no tool has been more successful at making this idea of "everyone being a coder" than the spreadsheet. It's an excellent (pun intended) gateway drug, with low barriers of entry, and a value proposition that was obvious enough to get accountants to spend 5 figures on a Mac when computers were something completely new.
Sure, the language is clunky. But it's apparently enough for most peoples' needs. I can't immediately think of any reasons for this almost moralistic criticism of Excel.
It seems like it would be a weekend project to convert Excel worksheets to sequential code in a universally-readable textfile, plus a separate CSV for the data. That would make sharing and reproducibility just as easy as easy as getting them to switch out their complete toolchain. But I have doubts that unwillingness to share code & data is actually a technical problems, instead of a lack of incentives/embarrassment of sharing ones' awful code/dreams of patents future/privacy issues.
I'm also not convinced that Julia as I have experienced it would be any better for long-term reproducibility. Last time I tried Julia, about every second package had a specific version that it required, and very few of them shared their preferences.
I think there is a lot of reason for moralistic criticism of excel.
> That would make sharing and reproducibility just as easy as easy as getting them to switch out their complete toolchain
Reproducibility is about reading as much as it is getting the right number. Excel hides the operations from the reader, so that there could be a cell X182that has the key operation that you never area able to trace back.
https://www.bloomberg.com/news/articles/2013-04-18/faq-reinh...
But I totally agree that an excel spreadsheet is the most popular and easiest to learn programming language on the planet. That's why I mentioned ipython notebooks above. Excel is great because you can enter something and immediately see your results in front of you, which is something that jupyter does as well. But we definitely need to draw more from excel's UI and workflow if we want people to switch to those tools.
> But I have doubts that unwillingness to share code & data is actually a technical problems, instead of a lack of incentives/embarrassment of sharing ones' awful code/dreams of patents future/privacy issues.
That's true. But at the very least internal use of plain text and git would be immediately helpful for collaboration and project management (branching, code reviews etc.). I think you underestimate how unwieldy a project like this can get very quickly.
> Last time I tried Julia, about every second package had a specific version that it required, and very few of them shared their preferences.
The landscape is still settling down after the 1.0 release. I share the concern about particular version numbers, but notice that basically all packages are on 0.X in SemVer. When the ecosystem makes it to 1.0 as well things will improve dramatically.
> That would make sharing and reproducibility just as easy as easy as getting them to switch out their complete toolchain
Reproducibility is about reading as much as it is getting the right number. Excel hides the operations from the reader, so that there could be a cell X182that has the key operation that you never area able to trace back.
https://www.bloomberg.com/news/articles/2013-04-18/faq-reinh...
But I totally agree that an excel spreadsheet is the most popular and easiest to learn programming language on the planet. That's why I mentioned ipython notebooks above. Excel is great because you can enter something and immediately see your results in front of you, which is something that jupyter does as well. But we definitely need to draw more from excel's UI and workflow if we want people to switch to those tools.
> But I have doubts that unwillingness to share code & data is actually a technical problems, instead of a lack of incentives/embarrassment of sharing ones' awful code/dreams of patents future/privacy issues.
That's true. But at the very least internal use of plain text and git would be immediately helpful for collaboration and project management (branching, code reviews etc.). I think you underestimate how unwieldy a project like this can get very quickly.
> Last time I tried Julia, about every second package had a specific version that it required, and very few of them shared their preferences.
The landscape is still settling down after the 1.0 release. I share the concern about particular version numbers, but notice that basically all packages are on 0.X in SemVer. When the ecosystem makes it to 1.0 as well things will improve dramatically.
I disagree that scripting XOR point-and-click. Why can't we manipulate a dataframe using both approaches?
If the pointing and clicking generates code that can seen and stored, then fine. A reproducibility problem comes from the fact that most point-and-click tools don't. As a computational biologist who uses R I am often frustrated by experimental colleagues who use Excel and often can't remember how they transformed the data months later. I don't necessarily have better memory but I can go into my script and look.
In scripting languages, writing and sharing a new function isn't hard. And, if the language is open source, that encourages the rapid development of a community which implements useful and state-of-the-art tools.
If every tool needs to be available through a GUI, then, as far as I can see, that GUI will either be a burden on creators or so generalized it's no better than tab completion.
If every tool needs to be available through a GUI, then, as far as I can see, that GUI will either be a burden on creators or so generalized it's no better than tab completion.
I'm worried about seeing a script that says
df = read_csv(file)
df <- df %>% mutate(log_income = log(income))
# do manipulations in shiny
Shiny(df)
# click "transform" in the top right hand corner...Point and click is fine for EDA, but it’s inherently manual.
Repeating those manual steps gets boring very quickly, and is also error prone.
You can try and have the GUI generate code, but automatically generated code is awful.
Repeating those manual steps gets boring very quickly, and is also error prone.
You can try and have the GUI generate code, but automatically generated code is awful.
We have had few point&click text-based systems in the past, notably Plan9 and Xerox Alto and only the letter have a complete "user friendly graphic programming environment"... I think a simpler approach like org-mode/org-babel is the best we have now out of the box.
Did you ever tried org-mode with org-babel? If not only see few video about org-mode reproducible research / emacs on YT&c :-) in terms of reproducibility Nix{,OS} and Guix{,SD} offer really nice tools to achieve full-stack reproducibility without the need of publish multi-Mb/Gb archives, only a text file .nix or scheme can do the business.
Also ESS inside Emacs it's another nice working environment, perhaps is not as "natural at first impact" like a spreadsheet but I bet NO ONE after a week of (self) training will found spreadsheet better or even usable.
Also ESS inside Emacs it's another nice working environment, perhaps is not as "natural at first impact" like a spreadsheet but I bet NO ONE after a week of (self) training will found spreadsheet better or even usable.
I think this is missing the forest for the trees. The audience for these tools is people who have never written in a text editor in their life! They literally might not even recognize that some fonts are fixed-width and not know what textedit is.
That is to say, emacs might be over-kill. But literate programming is a pretty powerful concept in general, and rmarkdown and Rstudio (along with simliar tools in Python, Julia, etc.) make this super easy, and I think its a great way to introduce people to programming.
Getting people to use Git is the fight worth having imo, but its still hard. I was talking to someone the other day who didn't like git because tough to have comments about the work: they didn't use code reviews on github to get consensus, just comments in the code.
Plus there is the whole data-management problem. Quilt, SQL connections, and DataDeps for Julia are good solutions, but there isn't a single one answer that we've coalesced behind.
These are all really tough problems. I'm actually really interested in MS's acquisition of github for this reason. Maybe they can put some money towards the obstacles that have prevented mass adoption of scripting and git.
That is to say, emacs might be over-kill. But literate programming is a pretty powerful concept in general, and rmarkdown and Rstudio (along with simliar tools in Python, Julia, etc.) make this super easy, and I think its a great way to introduce people to programming.
Getting people to use Git is the fight worth having imo, but its still hard. I was talking to someone the other day who didn't like git because tough to have comments about the work: they didn't use code reviews on github to get consensus, just comments in the code.
Plus there is the whole data-management problem. Quilt, SQL connections, and DataDeps for Julia are good solutions, but there isn't a single one answer that we've coalesced behind.
These are all really tough problems. I'm actually really interested in MS's acquisition of github for this reason. Maybe they can put some money towards the obstacles that have prevented mass adoption of scripting and git.
IMVHO anyone must know the tools of their trade (I do not know if this make sense in English, hope so, maybe also expressed as "the tools used for doing their job"). If today statistic is done with computers good computers tools must be known. Newcomers of course can't know by genetic, they have to learn and universities must teach them so...
Consider that org-mode itself was designed, written and still maintained by an astronomer, one of the most ancient and widespread completion framework for Emacs (Helm) was written and still maintained by a mounting guide that still be a mounting/climbing guide. Of course they are "exception" but they are simply people who encounter "the right tool" at a certain point in time and start to learn it.
On GitHub, for me as any proprietary platform should be ignored, at maximum used to share git repos, certainly not for PR&c that are proprietary unportable stuff. And that's another thing anyone that use a computer for more than play a game MUST know, from office guys with their email only on webmail and smartphones user with their "valuable" personal data (photos, video etc) to IT pro.
Consider that org-mode itself was designed, written and still maintained by an astronomer, one of the most ancient and widespread completion framework for Emacs (Helm) was written and still maintained by a mounting guide that still be a mounting/climbing guide. Of course they are "exception" but they are simply people who encounter "the right tool" at a certain point in time and start to learn it.
On GitHub, for me as any proprietary platform should be ignored, at maximum used to share git repos, certainly not for PR&c that are proprietary unportable stuff. And that's another thing anyone that use a computer for more than play a game MUST know, from office guys with their email only on webmail and smartphones user with their "valuable" personal data (photos, video etc) to IT pro.
> When I refer to tools for learning statistics, I mean things like applets, TinkerPlots, and Fathom....You can't put TinkerPlots on your resume, and if you actually want to apply the skills you've learned to real data, you need to learn another tool.
I can't really see why this is a problem as stated. You probably used a textbook as well, and you can't put that textbook on your resume either. Just because it's software doesn't make it different.
> So, the first thing that a future tool for statistical programming should do is bridge the gap between learning and doing. I think that a good tool should be able to ease people into programming, using some sort of visual, drag-and-drop interface that exposes novices to the entire trajectory of data analysis (data import, cleaning, new variable creation, plots, summary statistics, models). Then, it should have a way to make the transition to more traditional or textual coding. Ideally, one could look back and forth between the visual representation and the textual one, and a change in one interface would be reflected as a change in the other.
In statics, as programming, it's not clear a drag-and-drop pedagogical interface helps much (believe me, I"ve worked with such systems since I was a kid in the late 70s). This paragraph quoted above, and the part of the article above it, actually have the solution: use a bunch of helper code in R (which is what they use now) but generate the realtime results in an adjacent window. This is the approach used by LOGO back in, yes, the 1970s: you'd type a program (in a real programming language -- basically a Lisp) and see the results immediately. You could go back and modify the program and see those results immediately.
I can't really see why this is a problem as stated. You probably used a textbook as well, and you can't put that textbook on your resume either. Just because it's software doesn't make it different.
> So, the first thing that a future tool for statistical programming should do is bridge the gap between learning and doing. I think that a good tool should be able to ease people into programming, using some sort of visual, drag-and-drop interface that exposes novices to the entire trajectory of data analysis (data import, cleaning, new variable creation, plots, summary statistics, models). Then, it should have a way to make the transition to more traditional or textual coding. Ideally, one could look back and forth between the visual representation and the textual one, and a change in one interface would be reflected as a change in the other.
In statics, as programming, it's not clear a drag-and-drop pedagogical interface helps much (believe me, I"ve worked with such systems since I was a kid in the late 70s). This paragraph quoted above, and the part of the article above it, actually have the solution: use a bunch of helper code in R (which is what they use now) but generate the realtime results in an adjacent window. This is the approach used by LOGO back in, yes, the 1970s: you'd type a program (in a real programming language -- basically a Lisp) and see the results immediately. You could go back and modify the program and see those results immediately.
Today's "notebook UI" seems to be popular (think about Jupyter, Mathematica, SaGe, ...) and in the end they are graphic-sugar and a bit limited org-mode implementation...
On "graphic programming" LOGO was a toy, SmallTalk (actually Squeak) are another story (and the basis of very first modern workstation, the Xerox Alto) :-)
On "graphic programming" LOGO was a toy, SmallTalk (actually Squeak) are another story (and the basis of very first modern workstation, the Xerox Alto) :-)
LOGO was not a toy, it was a means to teach children the basics of differential geometry.
Ok, but try to use it for anything else today...
Do you expect to hammer a nail with a screwdriver?
I use SQLite (just the command-line version is good enough) for statistical programming. Many functions are missing, but I wrote extensions with the needed additional functions, and there are also some useful extensions that can be found in the SQLite source tree but that are not part of SQLite itself, so we can use that too.
Just as a reminder
https://panicz.github.io/pamphlet/
about how R is not such an innovative solution for statistic...
Could you point out where in that pamphlet there are criticisms of R? I see a Scheme tutorial prefaced by a petulant diatribe against programming languages that are not Scheme, and nothing specific about R. I quote, “On the other hand, being a rather seasoned Schemer, I knew that most of these languages were only inferior incarnations of Lisp, and I could find little value in learning a new syntactical disguise for those well-established concepts.
I was very soon to find out that my skepticism was right, and that learning R would require internalizing a lot of unfortunate idiosyncrasies that the authors of the language had chosen for their own reasons, and that I would be constantly stumbling over those incomprehensible limitations and discrep- ancies between common practices and the language developers’ decisions.”
And, “However, the case of the R programming language is particularly interesting, because initially it was just a harmless implementation of Scheme; but then as a result of the irresponsible experiments of mad scientists it mutated into a monster.”
I have a hard time taking this seriously, and it probably isn’t meant to be taken as such. It has practically nothing to do with R so I don’t know why one would bring it up.
And, “However, the case of the R programming language is particularly interesting, because initially it was just a harmless implementation of Scheme; but then as a result of the irresponsible experiments of mad scientists it mutated into a monster.”
I have a hard time taking this seriously, and it probably isn’t meant to be taken as such. It has practically nothing to do with R so I don’t know why one would bring it up.
Maybe adding Greenspun's tenth rule (ref. Wikipedia) or adding the recent "A Programmable Programming Language" [1] or even "Why Lisp" [2]...
Saying "gasoline car sucks" IMO automatically imply that any car using gasoline sucks, or if you prefer "$type car outshine any other car type".
[1] https://cacm.acm.org/magazines/2018/3/225475-a-programmable-...
[2] http://www.gigamonkeys.com/book/introduction-why-lisp.html
Saying "gasoline car sucks" IMO automatically imply that any car using gasoline sucks, or if you prefer "$type car outshine any other car type".
[1] https://cacm.acm.org/magazines/2018/3/225475-a-programmable-...
[2] http://www.gigamonkeys.com/book/introduction-why-lisp.html
The beauty of R (and Python) is the ecosystem, not necessarily the language. Most statisticians want libraries that they can use to do the modeling so that they can focus their time on the data, not the programming.
On python "hy" (programming language) is a lispy-python so can be used with any cython-compatible module, it's not as clear as classic CL/scheme but IMVHO good enough.
Yeah, let’s not reinvent everything that is available in the R ecosystem again for some reason. R might not win a beauty price for best programming language but it does the trick just fine.
An incrementalist approach that I’ve half wondered about is to build an implementation of R on top of Racket. My understanding is that R was originally built on a Scheme runtime. Since that time there’s been a lot of C/C++/Fortran code added in, but maybe too that could be leveraged via Racket’s FFI.
If I have a choice and need to do something reasonably simple or that is well-supported by Racket’s libraries, I like to reach for Racket. If I have a statistical or data munging task, and deployment story isn’t too important, it’s R all the way.
If I have a choice and need to do something reasonably simple or that is well-supported by Racket’s libraries, I like to reach for Racket. If I have a statistical or data munging task, and deployment story isn’t too important, it’s R all the way.
Potentially an easier path I've been thinking about is to take the already well polished Rcpp<->R bridge and extend it one more layer into a guile<->Rcpp<->R bridge. Part of the motivation here is to get better integration for statistical programming into GNU/TeXmacs: an scientific "office suite" built using guile scheme.
I'm curious to understand why is born in the first place since we already have many lisp/scheme then... R itself seem a reinvention of the wheel...
According to an article published in 1996 by Ihaka and Gentleman:
“The work has been heavily influenced by two existing languages-Becker, Chambers, and Wilks' S (1985) and Steel and Sussman's Scheme (1975). We felt that there were strong points in each of these languages and that it would be interesting to see if the strengths could be combined. The resulting language is very similar in appearance to S, but the underlying implementation and semantics are derived from Scheme. In fact, we implemented the language by first writing an interpreter for a Scheme subset and then progressively mutating it to resemble S.”
(https://www.stat.auckland.ac.nz/~ihaka/downloads/R-paper.pdf)
Unfortunately R’s gain was LispStat’s loss: https://www.jstatsoft.org/issue/view/v013
“The work has been heavily influenced by two existing languages-Becker, Chambers, and Wilks' S (1985) and Steel and Sussman's Scheme (1975). We felt that there were strong points in each of these languages and that it would be interesting to see if the strengths could be combined. The resulting language is very similar in appearance to S, but the underlying implementation and semantics are derived from Scheme. In fact, we implemented the language by first writing an interpreter for a Scheme subset and then progressively mutating it to resemble S.”
(https://www.stat.auckland.ac.nz/~ihaka/downloads/R-paper.pdf)
Unfortunately R’s gain was LispStat’s loss: https://www.jstatsoft.org/issue/view/v013
Thanks :-)
Hear tell it was begat of an unholy tryst between PHP and Excel, but I am not judging.
A much less flippant answer it was a work-a-like intended as a successor to a (licensed) stats language 'S'.
[0] https://en.wikipedia.org/wiki/S_(programming_language)
A much less flippant answer it was a work-a-like intended as a successor to a (licensed) stats language 'S'.
[0] https://en.wikipedia.org/wiki/S_(programming_language)
Thanks :-)
The article doesn't mention this, but after some poking around it appears that the interactive demo is written in Morphic, the Smalltalk successor that Alan Kay and colleagues have been working on for the last couple of decades.
[deleted]
First, you need to establish that scripting is far superior to point-and-click interfaces via reproducibility and legibility. This is the fight worth having, as it gets to the core of what it means to do science and share results.
Second, you need to convince them that they don't have to sacrifice the tractability and ease of use of excel. Jupyter notebooks, for all their faults, really sell this idea well. But the first goal should dominate the second. Nteract and shiny are great, but I they are big tools that are difficult to teach beginners to code up. We shouldn't say "use R because you can still use point-and-click interfaces", we should argue against point-and-click interfaces all together in this context.
I also agree with the poster below that running a script again with a new parameter changed is super super easy, and you dont need a GUI to explore data like that.