Living papers are useful vision, but it will take a long way to get there.
Even notebooks still are problematic, for example, this study found that only 25% of Jupyter notebooks could be executed, and of those, only 4% actually reproduced the same results.
One compromise is to evaluate the paper separate from it's artifacts, which are reviewed for availability, reproducibility, and reusability. In software engineering conferences, this is becoming a standard, and while there is a huge burden for reviewers to evaluate these things, I think it does take us in the right direction. So in this case, we also submitted our paper for evaluation for its artifacts.
One interesting distinction is "tutorials" vs. "lessons"... LearnR and datacamp are much better for _lessons_, where you ask a learner to type in some code and check to see if you got it right, with hidden cells with the appropriate test cases backend.
Nice! I like the "link includes" feature. Funny, because we're displaying the terminal results, we've had the opposite problem, where we need to get and process the ansi-escape codes, so we get colors, and proper formatting, etc.
Right now, we're using the notebooks to build better lecture materials + workshops, especially for my DevOps course:
https://github.com/CSC-DevOps/Course
We've seen some interest in having support for live documentation + OneOps (simple runbooks for one-off devops tasks), so we'll probably continue to explore this more.
Yes, one use-case is making it easier to setup/teardown clusters for local testing. Two practical scenarios for us: 1) autograding ansible/configuration scripts, 2) CI for instructions/tutorials that involve clusters/devops: https://builds.sr.ht/~ottomatica/job/69644#task-report
Cloud-ready images is an important direction, and on the horizon.
Some limitations in terms of the vms and providers:
* If the size of the initrd is too large, it cannot properly unpack into vm's RAM --- size of RAM must be increased accordingly. We could also change [boot params](https://www.lightofdawn.org/blog/?viewDetailed=00128), or use shared disks, etc.
* For hyperkit, apple's vmnet requires sudo to create a bridge interface on host. We've played with a version that use's vpnkit and port forwarding (like linuxkit/Docker for Mac), but this adds lots of complexity in image, and opted for the simpler approach.
* We would like a better template mechanism for reusing base images and extending. Right now, we support using base image reuse, with extensions through docker buildargs---ideally, we would want something like %include support in Dockerfiles.
* Finally, we're investigating how to make images work well on multiple providers. For example, ubuntu does not play nice with hyperkit out-of-the-box, but works fine for vbox and kvm.
Even notebooks still are problematic, for example, this study found that only 25% of Jupyter notebooks could be executed, and of those, only 4% actually reproduced the same results.
http://www.ic.uff.br/~leomurta/papers/pimentel2019a.pdf
One compromise is to evaluate the paper separate from it's artifacts, which are reviewed for availability, reproducibility, and reusability. In software engineering conferences, this is becoming a standard, and while there is a huge burden for reviewers to evaluate these things, I think it does take us in the right direction. So in this case, we also submitted our paper for evaluation for its artifacts.