Living papers are useful vision, but it will take a long way to get there.
Even notebooks still are problematic, for example, this study found that only 25% of Jupyter notebooks could be executed, and of those, only 4% actually reproduced the same results.
One compromise is to evaluate the paper separate from it's artifacts, which are reviewed for availability, reproducibility, and reusability. In software engineering conferences, this is becoming a standard, and while there is a huge burden for reviewers to evaluate these things, I think it does take us in the right direction. So in this case, we also submitted our paper for evaluation for its artifacts.
One interesting distinction is "tutorials" vs. "lessons"... LearnR and datacamp are much better for _lessons_, where you ask a learner to type in some code and check to see if you got it right, with hidden cells with the appropriate test cases backend.
Nice! I like the "link includes" feature. Funny, because we're displaying the terminal results, we've had the opposite problem, where we need to get and process the ansi-escape codes, so we get colors, and proper formatting, etc.
Right now, we're using the notebooks to build better lecture materials + workshops, especially for my DevOps course:
https://github.com/CSC-DevOps/Course
We've seen some interest in having support for live documentation + OneOps (simple runbooks for one-off devops tasks), so we'll probably continue to explore this more.
Even notebooks still are problematic, for example, this study found that only 25% of Jupyter notebooks could be executed, and of those, only 4% actually reproduced the same results.
http://www.ic.uff.br/~leomurta/papers/pimentel2019a.pdf
One compromise is to evaluate the paper separate from it's artifacts, which are reviewed for availability, reproducibility, and reusability. In software engineering conferences, this is becoming a standard, and while there is a huge burden for reviewers to evaluate these things, I think it does take us in the right direction. So in this case, we also submitted our paper for evaluation for its artifacts.