The main idea behind Clerk is letting folks use the editors they know & love and not force them to edit code inside a web browser. (This came to us only after working for a few years on nextjournal.com where folks can edit notebooks in a web browser but better late than never :)
In order to keep the feedback loop fast Clerk keeps a dependency graph of all forms and vars in a notebook to figure out what changed between runs on a notebook and only runs those.
Hey, I’m a founder of Nextjournal and one of the authors of clojure-mode.
Highly recommend checking out CodeMirror 6 [1], it’s really well done. On of the main benefits is its blazingly fast incremental parsing system called lezer [2], which is heavily influenced by tree-sitter. The Clojure grammar file is at [3].
I’ve done a small video about this at launch [4] and you can also play with this on Nextjournal [5].
For our notebook format at Nextjournal we use an edn appendix to store metadata like results and environment (docker image). The user-edited content is just plain markdown, while also being runnable reproducibly. See [1] and run it at [2].
Hey HN, I'm one of the founders of Nextjournal. The `cm6=1` query param is a feature flag to enable our new Clojure code editor built on CodeMirror 6 [1] with lezer for fast incremental parsing. You can find the keybindings on [2], be sure to try grow / shrink selection.
I used mmhmm today to record a quick demo [1] and think it works really well for that. I like the looks a lot more than the normal rectangular overlays in Zoom or ScreenFlow.
This is how runkit does it for nodejs and I think it’s working quite well for them.
We (at Nextjournal) tried doing the same for other languages (Python, Julia, R) and felt that it didn’t work nearly as well. You often want to change a cell at the top e.g. to add a new import and it can be quite annoying when long-running dependent cells re-execute automatically. I now think that automatic execution of dependent cells works great when your use case is fast executing cells (see observablehq) but we need to figure out something else for longer running cells. One idea that I haven’t tried yet is only run cells automatically that have executed within a given threshold.
I hear a lot of complaints about hidden state but I think it’s less of a problem in reality. It’s just a lot faster than always rerunning things from a clean slate. Clojure's live programming model [1] works incredibly well by giving the user full control over what should be evaluated. But Clojure's focus on immutability also makes this work really well. I rarely run into issues where I'm still depending on a var that's been removed and then there's still the reloaded workflow [2].
Overall I think notebooks are currently a great improvement for people that would otherwise create plain scripts – working on it is a lot quicker when you have an easy way to just execute parts of it. Plus there's the obvious benefit of interleaving prose and results. That doesn't mean we should not be thinking about addressing the hidden state problem but I think notebooks do add a lot of value nevertheless.
Author here, I did run it a few times but you don’t have to take my word for it, you can rerun the notebook yourself if you sign up for Nextjournal and remix my notebook. Full disclosure: I’m also a Cofounder of Nextjournal.
But I wouldn’t suggest to use Nextjournal for serious benchmarking (yet). We’re running on Google Cloud and it’s not suited for benchmarking unless you pay for a (big) sole tenant instance. In the future we plan to offer dedicated instances for benchmarking.
What you're describing is close to our long-term vision for the platform. Having it be easy to use for non programmers but with the ability to customise things and always "peek under the hood" to see how things are built.
Our GitHub and S3/GCS Components (see it in action in my launch post) are actually just thin layers that execute code from other notebooks and we plan to offer this ability to create custom components like this in the future.
I haven't tried Domino Datalab myself so take this with a grain of salt.
While data science is an obvious use case of literate programming, it's not the only one. I see the fundamental problem that needs to be addressed is one of dependency management. We address this today using Docker. In the future we plan to use a more functional approach most likely based on Nix or Guix. This more principled approach should address both reproducibility and usability (by allowing to compose images and providing much better install times thanks to binary caching).
I haven't really used Domino Datalab but I'm not sure if they allow for the installation of arbitrary system libraries and packages like we do. Check out some out our machine learning samples which run on GPUs: https://nextjournal.com/collection/machine-learning
In the future we also plan to allow in-browser JavaScript execution, this is currently hidden behind a feature flag but we still have an article that uses it in https://nextjournal.com/dubroy/ohm-parsing-made-easy
Yes, we certainly want to support a lot more languages in the future and it should be relatively easy to do as long as there's a Jupyter kernel (which is a lot).
We also have a proof-of-concept PR where you can implement a runtime in a notebook.
I think we'll expand the available languages as soon as we're confident the core product really solid.
Our standard instances at 3,75 GB of Ram and we keep a pool of three idle ones of those around. With the free account you can currently use larger instances of up to 16 GB of Ram and 1 Nvidia K80 GPU for free.
If you sign up for the paid plan which is 99$ per researcher per month you can provision more powerful machines – basically anything that Google Cloud offers.
We currently don't enforce any storage limits.
This is our first iteration of pricing though so I'm pretty sure this will still change over time. We've gotten a lot of feedback from people asking for a cheaper plan.
What most people don't realise however is that you can use most of the features (including private drafts) as it stands now for free. We've also been debating weather we should allow for private drafts on the free plan or take a stance on what open science really means (working in the open from the start) but decided agains this for now.
Curious to hear what others think about this. Do you expect drafts to be private and would it be a violation of those expectations if they were not?
Awesome, happy to see that. Been wanting to play with crux anyway. Nextjournal runs on Clojure and Datomic and we use some of pack.alpha and aero from juxt, so thanks!
Hey, this sounds like something we'd want to look at, I don't think it's inherent to reproducibility. I think we should be truncating the output in this case. Please send us the notebook via the Help button that shows up when you produce an error in a cell, I'll take a look and we'll figure this out.
While Colaboratory is built on top of Jupyter Nextjournal is not.
We do support importing Jupyter notebooks and running Jupyter kernels, we also have our own runtime protocol.
In Jupyter (and hence in Colaboratory) you normally have one runtime that's running both your server code as well as the user code. In Nextjournal there's a separate application called the Runner that's orchestrating the runtimes which currently are docker images.
This allows us to use Nextjournal notebooks to do any kind of installations without the need for a full Jupyter kernel inside the image, something that gets tricky in Jupyter. Once we have a bash shell inside the image, we can do installations.
Our system takes care of only referencing the image sha's everywhere, so everything is immutable and you can't accidentally overwrite anything.
You can also pull those docker images and use them locally.
Any data you upload or results you save (just write to a /results folder) is put into content-addressed storage, so same thing here, you'll never accidentally overwrite a file.
Lastly the document is stored in the database (Datomic) and you can restore any previous state.
Leveraging immutability at all layers of the stack is what enables our "remix" feature, so the ability to quickly and cheaply clone any published notebook and continue where another person left off.
It can take a namespace like https://github.com/nextjournal/clerk/blob/fd79f7504ebffaa279... and turn it into https://storage.googleapis.com/nextjournal-snapshots/clerk/b...
The main idea behind Clerk is letting folks use the editors they know & love and not force them to edit code inside a web browser. (This came to us only after working for a few years on nextjournal.com where folks can edit notebooks in a web browser but better late than never :)
In order to keep the feedback loop fast Clerk keeps a dependency graph of all forms and vars in a notebook to figure out what changed between runs on a notebook and only runs those.
There's no homepage for it yet but Clerk is running a handful of sample notebooks on every commit and building a static website out of them. The latest version is at https://storage.googleapis.com/nextjournal-snapshots/clerk/b...