Show HN: Observable Notebooks(beta.observablehq.com)
beta.observablehq.com
Show HN: Observable Notebooks
https://beta.observablehq.com/
135 comments
Inre your first link, things can get a bit awkward if someone uses javascript features that aren’t supported in your browser. The error message is not especially non-expert-reader-friendly: https://i.imgur.com/Hw9NDPA.png
Any plans to add some kind of transpilation or ...?
Any plans to add some kind of transpilation or ...?
Oh no! That's my bad. I always default to using the spread operators because it's so nice and I live in nice new browsers.
What happens with the image rotation 'jump' between 302 and 303 degrees on the last one? Why does it do that?
Just in case people were wondering, the site seems to have been overwhelmed for the past 10-15 minutes. But there's a YouTube demo of the tech https://news.ycombinator.com/item?id=16275040
I guess an oversimplified description would be that this is like a Jupyter Notebook specifically for JavaScript. Libraries like D3 are pre-loaded and immediately accessible. Am definitely interested in hearing the details about what it is built with and medium to long-term plans for the service.
Note that the Jupyter Notebook service generally requires you to be installing and running Python etc. on your own computer. Jumping into an Observable notebook is as easy as opening your browser and signing in via GIthub
I guess an oversimplified description would be that this is like a Jupyter Notebook specifically for JavaScript. Libraries like D3 are pre-loaded and immediately accessible. Am definitely interested in hearing the details about what it is built with and medium to long-term plans for the service.
Note that the Jupyter Notebook service generally requires you to be installing and running Python etc. on your own computer. Jumping into an Observable notebook is as easy as opening your browser and signing in via GIthub
I wrote more about the motivation for Observable (formerly d3.express) last year: https://medium.com/@mbostock/a-better-way-to-code-2b1d2876a3...
It’s easy to use D3 within Observable, but it’s not pre-loaded or specifically built-in. Any browser library published to npm can be loaded via require (for AMD) or dynamic import (for ES modules). I’ve published a bunch of notebooks with D3 but you’ll find other ones with THREE.js, Vega-Lite, Leaflet and other libraries.
It’s easy to use D3 within Observable, but it’s not pre-loaded or specifically built-in. Any browser library published to npm can be loaded via require (for AMD) or dynamic import (for ES modules). I’ve published a bunch of notebooks with D3 but you’ll find other ones with THREE.js, Vega-Lite, Leaflet and other libraries.
Are the language/syntax differences documented somewhere? It's difficult to tell if something is an obscure early stage ecmascript proposal or a language feature specific to observablehq. For example in https://beta.observablehq.com/@mbostock/introduction-to-note...
import {canvas as flood} with {height} from "@mbostock/randomized-flood-fill"
i = {
let i = 0;
while (true) {
yield ++i;
}
}
We’re still writing more documentation, but there’s a bit more detail here (and I’ll be posting more soon): https://beta.observablehq.com/@mbostock/introduction-to-code
To summarize, the body of a cell is typically either an expression or a block statement, akin to the body of an arrow function. If a cell starts with name = …, then it has a name and can be referenced by other cells. So, the name = part is specific to Observable, but the expression and block statement is normal JavaScript.
The import declaration is similar to a normal static ES import, but the with clause allows you to inject local definitions into the imported notebook.
To summarize, the body of a cell is typically either an expression or a block statement, akin to the body of an arrow function. If a cell starts with name = …, then it has a name and can be referenced by other cells. So, the name = part is specific to Observable, but the expression and block statement is normal JavaScript.
The import declaration is similar to a normal static ES import, but the with clause allows you to inject local definitions into the imported notebook.
There is mybinder.org to run jupyter notebooks online . For example, https://github.com/norvig/pytudes
There's also some (currently) free cloud version by microsoft available on https://notebooks.azure.com in a preview state, but works great already. (Tested python3 only)
There's a reason why you would want to run Python and that's the huge, well supported scientific computing libraries it has.
Absolutely! We all quite like Python, and there's no denying that JavaScript doesn't have the same caliber and range of scientific code (yet!).
I fully expect plenty of people to do some of the data-crunching in Python, Julia, R, and so on, and bring it into Observable to explore and experiment. And I also expect JavaScript's ecosystem to blossom, especially with WebAssembly & WebGL hitting the mainstream.
I fully expect plenty of people to do some of the data-crunching in Python, Julia, R, and so on, and bring it into Observable to explore and experiment. And I also expect JavaScript's ecosystem to blossom, especially with WebAssembly & WebGL hitting the mainstream.
What’s the status of d3 ports to Python? I believe plotly, d3py, and Bokeh all offer some amount of support.
I’d expect more python users to bring d3 in that bring python to JavaScript.
I’d expect more python users to bring d3 in that bring python to JavaScript.
[deleted]
No doubt you're right, but I don't think JS is well suited for scientific computing. At least Python gives you operator overloading. Julia and R are designed for this field. It's better for JS to remain on the presentation side for anything sophisticated.
Forgive me if I read the response wrong but that seems like exactly what the author is saying? Do your computing in Python/R, export a generated CSV, and then present it in Observable?
With the possibility that WebAssembly and WebGL will make the libraries in Python/R available to JS. Which might be alright. You probably won't be able to utilize vectorization with operators.
The point wasn't that there is something wrong with Python, the point was that Jupyter requires local installation whereas Observable doesn't require installation of any kind.
Yep, Python is my language for work and teaching, especially for data projects. Doesn't mean I don't envy the interactivity and visualization potential working in JS.
I was just thinking about a proliferation of Javascript statistical libraries popping up... only an NPM away. And I shuddered a little.
And then it occurred to me that CRAN and NPM have a fair bit in common. Both have a package for anything you might imagine, and both have a mix of amazing and less-than-amazing quality.
And then it occurred to me that CRAN and NPM have a fair bit in common. Both have a package for anything you might imagine, and both have a mix of amazing and less-than-amazing quality.
acosmism(1)
There are a few ways to open Jupyter notebooks in the cloud pretty easily:
https://aws.amazon.com/sagemaker/
https://cloud.google.com/datalab/
https://gryd.us/
https://kyso.io (disclaimer - I'm a founder)
Do you want it to be so easy that you jump into the notebook immediately upon logging in, the hard part is that javascript will run in your browser - but python or R need a kernel running somewhere?
https://aws.amazon.com/sagemaker/
https://cloud.google.com/datalab/
https://gryd.us/
https://kyso.io (disclaimer - I'm a founder)
Do you want it to be so easy that you jump into the notebook immediately upon logging in, the hard part is that javascript will run in your browser - but python or R need a kernel running somewhere?
So it's "databricks for JS" ?
if you're going to be using an online hosted service ("sign in via GitHub") then you don't need to install and run Python to use a Jupyter Notebook...
Please, Observable, put something on the page that says what language this is!
I figured this was like Colaboratory (https://colab.research.google.com) or Azure Notebooks (https://notebooks.azure.com/) or some other hosted-jupyter variant. There was no information about how it should work when I went to the scratchpad, and I could barely tell it was Javascript when picking one of the sample notebooks.
I figured this was like Colaboratory (https://colab.research.google.com) or Azure Notebooks (https://notebooks.azure.com/) or some other hosted-jupyter variant. There was no information about how it should work when I went to the scratchpad, and I could barely tell it was Javascript when picking one of the sample notebooks.
Not only the language, but have at least one line that says what this is. I can guess that it’s a hosted notebook service, but I can’t tell how it’s different from Jupyter Notebook. Is the difference in the language it uses, or are there other differences?
[deleted]
See https://beta.observablehq.com/@mbostock/five-minute-introduc... and the links therein, e.g. the video they made https://www.youtube.com/watch?v=uEmDwflQ3xE
I would also like to understand the licensing and how to host the resulting files myself.
This looks fantastic. The model appears very clean and different than Jupyter in many ways:
- cells can have names (it seems each cell can only export one name that is visible to the rest of the notebook)
- dependent cells auto-update when a source cell changes
- cells can be generators that auto update up to 60 times/sec
- cell values can be tied to UI elements easily
So instead of Jupyter's model where all cells execute in the same namespace, here you have named cells explicitly connected to each other in a graph and executed in topological order (not linear order).
The introduction covers these pretty well: https://beta.observablehq.com/@mbostock/five-minute-introduc...
- cells can have names (it seems each cell can only export one name that is visible to the rest of the notebook)
- dependent cells auto-update when a source cell changes
- cells can be generators that auto update up to 60 times/sec
- cell values can be tied to UI elements easily
So instead of Jupyter's model where all cells execute in the same namespace, here you have named cells explicitly connected to each other in a graph and executed in topological order (not linear order).
The introduction covers these pretty well: https://beta.observablehq.com/@mbostock/five-minute-introduc...
This is really cool and would be a great way to introduce programming concepts in an educational setting. Instant-feedback, inline documentation in a narrative style, and running on a browser really lowers the barrier for entry.
I would have loved something like this when I was learning programming in middle-school decades ago.
Since it comes from the guys behind D3 it's not surprising that much of the visuals come from D3. This is really evocative, but it may limit some of the audience for this tool. I would encourage the developers to take some cues from Apple's swift playgrounds and add visuals and sliders for even some of the simpler language constructs like loops and add some "simple" wrapper functions for drawing elementary shapes. This would definitely be a shift from the intended purpose as I see it right now, but it would have the side-benefit of increasing debugability as well.
Great work - look forward to having the occasion to using it.
I would have loved something like this when I was learning programming in middle-school decades ago.
Since it comes from the guys behind D3 it's not surprising that much of the visuals come from D3. This is really evocative, but it may limit some of the audience for this tool. I would encourage the developers to take some cues from Apple's swift playgrounds and add visuals and sliders for even some of the simpler language constructs like loops and add some "simple" wrapper functions for drawing elementary shapes. This would definitely be a shift from the intended purpose as I see it right now, but it would have the side-benefit of increasing debugability as well.
Great work - look forward to having the occasion to using it.
I would have loved something like this while learning to program as well. I mean ... the Ruby REPL was fun, but this sort of thing is just so much more tactile and visual.
We haven't had time to produce as many demo notebooks as we might have liked, but these notebooks really aren't D3-specific in the slightest. They're a reactive flavor of JavaScript, and anything that produces a value or can be rendered to the DOM will work.
For example, here are a whole mess of Plotly charts: https://beta.observablehq.com/@jashkenas/plotly-js
Here are some (wild) WebGL, Three.js and regl sketches: https://beta.observablehq.com/collection/webgl
And here are some simple native inputs you can use to control your sketches: https://beta.observablehq.com/@jashkenas/inputs
Remember — a "slider" here isn't anything special. It's just a chunk of DOM that produces a number as a value. It can be replaced with anything else you can imagine that produces a number as a value. There's no API there ;)
We haven't had time to produce as many demo notebooks as we might have liked, but these notebooks really aren't D3-specific in the slightest. They're a reactive flavor of JavaScript, and anything that produces a value or can be rendered to the DOM will work.
For example, here are a whole mess of Plotly charts: https://beta.observablehq.com/@jashkenas/plotly-js
Here are some (wild) WebGL, Three.js and regl sketches: https://beta.observablehq.com/collection/webgl
And here are some simple native inputs you can use to control your sketches: https://beta.observablehq.com/@jashkenas/inputs
Remember — a "slider" here isn't anything special. It's just a chunk of DOM that produces a number as a value. It can be replaced with anything else you can imagine that produces a number as a value. There's no API there ;)
Oh that's cool I thought there were special hooks with D3. The native inputs demo seems really compelling especially in a learning environment. I'll definitely play with this and bring it into my next mentoring group meeting. (They'll be so glad to get away from scratch!)
One cool thing that Swift playgrounds let you do is advance program state through loops and other flow-control line-by-line using a slider. I think they have to have cooperation with the compiler/runtime to make this happen. (I'm also not sure how truly useful it is after a few days of learning the basics.)
I suppose you could do something similar with the native inputs but not using native syntax. What I'm thinking would be a massive undertaking I realize - it's cool that this offering generates so many "ooh and what else" ideas.
One cool thing that Swift playgrounds let you do is advance program state through loops and other flow-control line-by-line using a slider. I think they have to have cooperation with the compiler/runtime to make this happen. (I'm also not sure how truly useful it is after a few days of learning the basics.)
I suppose you could do something similar with the native inputs but not using native syntax. What I'm thinking would be a massive undertaking I realize - it's cool that this offering generates so many "ooh and what else" ideas.
Yes, it's absolutely worth exploring further — and we definitely have notions for ways to further expose and make interactive the running state of the notebook.
But for starters: You can do something similar to expose the internal state of loops and flow-control constructs by just using JavaScript's generators.
Here's an extremely simple example that slowly yields the value of a loop's `i` variable:
https://beta.observablehq.com/@jashkenas/flow-control-demo-w...
If you remove the delay, it'll instead yield `i` at 60fps.
But for starters: You can do something similar to expose the internal state of loops and flow-control constructs by just using JavaScript's generators.
Here's an extremely simple example that slowly yields the value of a loop's `i` variable:
https://beta.observablehq.com/@jashkenas/flow-control-demo-w...
If you remove the delay, it'll instead yield `i` at 60fps.
The sliders on https://beta.observablehq.com/@mbostock/clifford-attractor-i... don't work for me, the sliders don't move. If I double-click a cell and edit the value, I get an updated graphic, but the sliders won't move.
Chrome 63.0.3239.132, OS X 10.13.3
Chrome 63.0.3239.132, OS X 10.13.3
I had the same problem and it turned out to be because I had the experimental Strict Site Isolation flag set in Chrome. Try turning that off.
The app itself is well done, but the editing experience is quite lacking...It's 2018 and people don't want to write html/css/js as if writing txt files. https://codesandbox.io can be a good inspiration to learn from. I believe its popularity is a result of using Monaco[0] and supporting Language Servers[1].
I would advise looking into editing/language support early. Auto completion (for DOM API, D3 API, etc) and error checking can be very helpful to the user.
Still, congratulations to the launch (the app looks beautiful and UX is pretty solid)! When I was reading @mbostock's medium post[2] a few months ago I knew something like this will popup.
Anyway, back to building my own thing which focuses on the editing side of playgrounds, which I think is what most online ones (Codepen / JSFiddle) lack.
[0]: https://microsoft.github.io/monaco-editor
[1]: https://microsoft.github.io/language-server-protocol
[2]: https://medium.com/@mbostock/a-better-way-to-code-2b1d2876a3...
I would advise looking into editing/language support early. Auto completion (for DOM API, D3 API, etc) and error checking can be very helpful to the user.
Still, congratulations to the launch (the app looks beautiful and UX is pretty solid)! When I was reading @mbostock's medium post[2] a few months ago I knew something like this will popup.
Anyway, back to building my own thing which focuses on the editing side of playgrounds, which I think is what most online ones (Codepen / JSFiddle) lack.
[0]: https://microsoft.github.io/monaco-editor
[1]: https://microsoft.github.io/language-server-protocol
[2]: https://medium.com/@mbostock/a-better-way-to-code-2b1d2876a3...
This works by executing Javascript code written by the user. Since these notebooks are shareable, and there is also a log-in feature, do any security features need to be put in place to prevent, say, someone linking me to a notebook that sends my login credentials for observablehq.com to a malicious host? That is, is the javascript inside the notebook sandboxed in any way?
I don't fully understand how modern web security works. But "executing arbitrary javascript written by user A on website B viewed by user C" seems worrying.
I don't fully understand how modern web security works. But "executing arbitrary javascript written by user A on website B viewed by user C" seems worrying.
Sure, happy to share a few of the gory details.
Yep, Observable runs user JavaScript code, but that code has no access to your login credentials or anything from the 'application': user code runs in a sandboxed iframe on a separate host - host that doesn't have any cookies or user information. The application communicates with that iframe host through serialized postMessage information only. That's the gist of the 'user code' portion - the rest of the application has additional layers of best-practice security like very strict content security policies and so on that defend from other kinds of attacks.
Yep, Observable runs user JavaScript code, but that code has no access to your login credentials or anything from the 'application': user code runs in a sandboxed iframe on a separate host - host that doesn't have any cookies or user information. The application communicates with that iframe host through serialized postMessage information only. That's the gist of the 'user code' portion - the rest of the application has additional layers of best-practice security like very strict content security policies and so on that defend from other kinds of attacks.
Yes. We use <iframe>s with the sandbox attribute (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/if...) to uh ... sandbox away the running JavaScript.
If you inspect them, you'll see that those frames are hosted on observableusercontent.com instead of observablehq.com.
It's a similar security model to how raw files on GitHub are served from githubusercontent.com.
If you inspect them, you'll see that those frames are hosted on observableusercontent.com instead of observablehq.com.
It's a similar security model to how raw files on GitHub are served from githubusercontent.com.
I wrote a simple notebook:
https://beta.observablehq.com/@ontouchstart/test
to confirm that the code is running in an iframe from host https://static.observableusercontent.com
https://beta.observablehq.com/@ontouchstart/test
to confirm that the code is running in an iframe from host https://static.observableusercontent.com
The homepage copy didn't quite capture me... until I realized it was the new thing by D3's Mike Bostock! Maybe worth name-dropping that somewhere on that page? Though I don't know how many people would respond to that other than myself (hopefully lots of people).
Eh. I respect people more for not namedropping themselves, but maybe I'm alone in this.
(Are you sure it's a good idea to perpetuate the status quo? It's hard to get anyone to pay attention to what you've made unless you're already known. That seems a bit unfair to the next generation, and rather the opposite of oldschool hacker ethos.)
(Are you sure it's a good idea to perpetuate the status quo? It's hard to get anyone to pay attention to what you've made unless you're already known. That seems a bit unfair to the next generation, and rather the opposite of oldschool hacker ethos.)
I totally see where you're coming from - name-dropping like that feels really tacky. I'd shy away from that on my own projects.
The challenge is helping people understand what something is and why it is valuable as quickly as possible. In this case, the fact that the team behind this are Mike Bostock (d3), Jeremy Ashkenas (Backbone, CoffeeScript) and Tom MacWright (Mapbox Studio) feels very relevant to getting me excited about the project.
It's such a strong product that I imagine word of mouth (plus how well it demos) will do that job for them though. This was just my first thought on loading up he page in Mobile Safari (and then realizing what it actually was).
The challenge is helping people understand what something is and why it is valuable as quickly as possible. In this case, the fact that the team behind this are Mike Bostock (d3), Jeremy Ashkenas (Backbone, CoffeeScript) and Tom MacWright (Mapbox Studio) feels very relevant to getting me excited about the project.
It's such a strong product that I imagine word of mouth (plus how well it demos) will do that job for them though. This was just my first thought on loading up he page in Mobile Safari (and then realizing what it actually was).
Yeah honestly that raised my interest considerably.
Echoing the other commentor, I think the name-dropping provides two important pieces of info:
1) I can match my previous knowledge of related projects to get an idea of what this project is trying to be, and likely have that be accurate.
2) in an age where any random half-baked idea has a polished marketing page and lofty ambitions, it's a noise filter for "these people have completed projects that have actually been used". If anything, on a page that doesn't name-drop, I'd trust a bare-bones, nocss landing page more.
1) I can match my previous knowledge of related projects to get an idea of what this project is trying to be, and likely have that be accurate.
2) in an age where any random half-baked idea has a polished marketing page and lofty ambitions, it's a noise filter for "these people have completed projects that have actually been used". If anything, on a page that doesn't name-drop, I'd trust a bare-bones, nocss landing page more.
This is a rare case where it seems useful for me to externalize my inner monologue: "Well, this just looks like Jupyter lite... oh, wait, this is Mike Bostock's project? He's built some very cool visualizations and some really powerful libraries with a tasteful API design... if this is his idea of a useful workflow it's probably well thought-out and executed and worth checking out."
A+.
Feedback: I instantly understood that it was something similar to Jupyter, and my very first mental question was "I hope it's in JS", and this took a surprisingly long few seconds and two page navigations to establish -- I expected the choice of language to be on the splash page.
Anyway, very well done, and it's quite clear to me that JS is going to trounce Python for nearly everything long-term. The JS ecosystem is just an unstoppable freight train.
Feedback: I instantly understood that it was something similar to Jupyter, and my very first mental question was "I hope it's in JS", and this took a surprisingly long few seconds and two page navigations to establish -- I expected the choice of language to be on the splash page.
Anyway, very well done, and it's quite clear to me that JS is going to trounce Python for nearly everything long-term. The JS ecosystem is just an unstoppable freight train.
lol
Their new screencast is awesome too!
"Observable: An Earthquake Globe in Ten Minutes" https://www.youtube.com/watch?v=uEmDwflQ3xE
"Observable: An Earthquake Globe in Ten Minutes" https://www.youtube.com/watch?v=uEmDwflQ3xE
And good place to drop few links, if you'd like to cut to the chase.
The finished notebook from the screencast (which you can fork): https://beta.observablehq.com/@jashkenas/earthquakes
And a fancier version that spins, reuses the canvas element instead of re-rendering it from scratch, projects the quakes as circles on the surface of the sphere, and offers a bunch of other options to configure (like if you'd like to chart the past day, week or month of quakes): https://beta.observablehq.com/@jashkenas/quakespotter-0-1
The finished notebook from the screencast (which you can fork): https://beta.observablehq.com/@jashkenas/earthquakes
And a fancier version that spins, reuses the canvas element instead of re-rendering it from scratch, projects the quakes as circles on the surface of the sphere, and offers a bunch of other options to configure (like if you'd like to chart the past day, week or month of quakes): https://beta.observablehq.com/@jashkenas/quakespotter-0-1
Do you guys have a keyboard shortcuts cheatsheet? I can't figure how to move from one cell to another without clicking or creating a new cell.
Sure thing! When you're on a notebook, click on your user avatar, then help. Walkthrough: https://file-oxhnhlfzte.now.sh/
Most excellent, thanks much
If you hit the '?' key (when you don’t have a code editor focused), a help modal appears that shows all the keyboard shortcuts. Most of the keyboard shortcuts are also described in the introduction here: https://beta.observablehq.com/@mbostock/introduction-to-note...
You can change which cell is focused using Option-Tab ⌥⇥ and Shift-Option-Tab ⇧⌥⇥. You can also click on the asterisk (*) in the left margin to edit any cell.
You can change which cell is focused using Option-Tab ⌥⇥ and Shift-Option-Tab ⇧⌥⇥. You can also click on the asterisk (*) in the left margin to edit any cell.
I didn't know you were involved in this project too. That screencast is astonishing. Congratulations on the launch!
small bug: the earthquake notebook points to `http://www.example.com` instead of the youtube screencast.
Right, we didn't have the URL yet last night. Thanks! Fixed.
This couldn't have come at a more convenient time for me. After years of telling myself I'd learn D3 I have finally started to as well as use it at work. I was using an extension in Jupyter but it was behaving unpredictably, and not being a frontend engineer by trade I found the process of sublime + browser foreign and cumbersome. As a big fan of jupyter and zeppelin I am stoked to see notebooks entering the js data viz domain.
Curious as to know how developers view the notebook paradigm? I feel like there is a sense of pride in working from the command line or vi, so wonder how these UI heavy approaches are received by the js/python/scala community.
Curious as to know how developers view the notebook paradigm? I feel like there is a sense of pride in working from the command line or vi, so wonder how these UI heavy approaches are received by the js/python/scala community.
I learned D3 a year back and am reasonably fluent at it now and I'm not sure I would ever pick this platform to learn D3. The easiest way to learn is simply to have a blank HTML file with your D3 code in it. The reason you want to do this is because you want to take advantage of powerful debugging tools that exist within modern browsers (Chrome dev tools, or equivalent from Firefox). ObservableHQ will simply add a layer in between that will make everything more annoying to debug and slow down experimentation.
I think it is really cool, and I have mountains of respect for Mike for his contributions to the field of data visualization/javascript, but my feeling is that this is more suited for sharing "notebooks" that are interactive and also to potentially allow for collaborative notebook building/demos. In essence, this is a giant evolution over his previous work on bl.ocks.org (https://bl.ocks.org/mbostock)
I think it is really cool, and I have mountains of respect for Mike for his contributions to the field of data visualization/javascript, but my feeling is that this is more suited for sharing "notebooks" that are interactive and also to potentially allow for collaborative notebook building/demos. In essence, this is a giant evolution over his previous work on bl.ocks.org (https://bl.ocks.org/mbostock)
I know many developers who've integrated Jupyter Notebooks in their workflow too. For Python-specific things, it's a great way to replicate bugs, try out a few initial solution paths, and share them with collaborators who can then interact with.
Regarding D3, I haven't tried that out yet, although I've used the `folium` Python package, which generates leaflet.js code you can put on a website. Otherwise, maybe check out [Jupyter Dashboards](https://github.com/jupyter/dashboards), whose interactive widgets seem to be in the same vein as OP.
Regarding D3, I haven't tried that out yet, although I've used the `folium` Python package, which generates leaflet.js code you can put on a website. Otherwise, maybe check out [Jupyter Dashboards](https://github.com/jupyter/dashboards), whose interactive widgets seem to be in the same vein as OP.
The BeakerX extension to Jupyter has good support for D3 visualization of data prepared in Python and other languages. http://BeakerX.com/
Here's a live demo on MyBinder:
https://mybinder.org/v2/gh/twosigma/beakerx/0.12.0?filepath=...
https://mybinder.org/v2/gh/twosigma/beakerx/0.12.0?filepath=...
To me, a netbook like this is similar to a REPL and works well for visualization tasks. You can figure it out in the netbook/REPL and then move it to "real code" once you've got it figured out.
To me the killer app for this is to empower data scientists to create useful, ad-hoc visualization apps for their results. Right now the html / presentation story for Jupyter notebooks is awful.
Mike, Jeremy, and Tom, this looks awesome. I'm looking forward to put it through the paces.
Mike, Jeremy, and Tom, this looks awesome. I'm looking forward to put it through the paces.
>Right now the html / presentation story for Jupyter notebooks is awful.
How so?
Ive found it pretty straightforward to convert matplotlib plots to gif/mp4 for presentation (note: this can be done automatically within the notebook), and plotly is pretty good for truly interactive stuff.
How so?
Ive found it pretty straightforward to convert matplotlib plots to gif/mp4 for presentation (note: this can be done automatically within the notebook), and plotly is pretty good for truly interactive stuff.
I'm specifically referring to the use case of creating explorable apps. Most data scientists I know would love a way of, say, building a map with a few input controls that alter the data or presentation (colors, layers, filtering, aggregation).
I'm currently working on a JS-powered geospatial exploration app that is meant to be run either standalone or embedded into a Jupyter notebook. We often run into questions around how much the embeddable app should be scriptable or how much user control to allow. This would seemingly solve many of our needs.
I'm currently working on a JS-powered geospatial exploration app that is meant to be run either standalone or embedded into a Jupyter notebook. We often run into questions around how much the embeddable app should be scriptable or how much user control to allow. This would seemingly solve many of our needs.
Some years ago I tried (and admittedly failed) to do something similar (a Jupyter notebook as an app) when trying to build a domain specific application for water resource management - we ended up building a "classic" web application via Leaflet and Python at the backend.
The idea is intriguing, however it is non trivial to pass information from JS to Python and vice versa. You can do that, but it messes with Jupyter's event system and breaks the usual notebook workflow. Maybe this has changed for the better, but at that time this stuff (IPython widgets I think) was heavily in flux.
The idea is intriguing, however it is non trivial to pass information from JS to Python and vice versa. You can do that, but it messes with Jupyter's event system and breaks the usual notebook workflow. Maybe this has changed for the better, but at that time this stuff (IPython widgets I think) was heavily in flux.
Have you looked at shiny (R) or dash (python)? I found it pretty easy to
building a map with a few input controls that alter the data or presentation (colors, layers, filtering, aggregation)
Yes, but neither shiny nor python (folium, etc) give us the level of customization or control via an actual JS bridge that this appears to.
That said, I tried to create a demo application and it seems like the require / exports / global situation isn't 100% yet. Any tips from the team on what to do here?
https://beta.observablehq.com/@akre54/deck-gl-test
That said, I tried to create a demo application and it seems like the require / exports / global situation isn't 100% yet. Any tips from the team on what to do here?
https://beta.observablehq.com/@akre54/deck-gl-test
I like the idea, I've even written something similar myself, but for me a few things were not clear.
How would I load in data that is stored on my harddrive (not web addressable) without having to run my own server or go through the file selector popup box each time. User JS can't just read the harddisk (a requirement for web security). If I have to run my own server then a major selling point of this is gone.
I work with large tabled data and very often use vectorised functions using numpy or pandas, working on entire columns with a fantastically simple interface. A = B * x, for arrays B, scalars x returning a new array A. Is there something as beautiful available, working around the lack of operator overloading in JS? I've only seen string abuse like p('A') = p('B*x'). Auh.
How would I load in data that is stored on my harddrive (not web addressable) without having to run my own server or go through the file selector popup box each time. User JS can't just read the harddisk (a requirement for web security). If I have to run my own server then a major selling point of this is gone.
I work with large tabled data and very often use vectorised functions using numpy or pandas, working on entire columns with a fantastically simple interface. A = B * x, for arrays B, scalars x returning a new array A. Is there something as beautiful available, working around the lack of operator overloading in JS? I've only seen string abuse like p('A') = p('B*x'). Auh.
Good question! Mike wrote a notebook about getting data into Observable here: https://beta.observablehq.com/@mbostock/introduction-to-data - the gist is that right now any host that supports CORS does the trick, and the combination of GitHub Gist & RawGit is pretty good for the task. It'd be nice to drag & drop data right in the app, though, so... stay tuned.
And, yep - something like pandas for JavaScript would also be nice... also stay tuned.
And, yep - something like pandas for JavaScript would also be nice... also stay tuned.
FYI: got some 'Oops, an unexpected error occurred.' error message, then the notebook I was testing just disappeared.
So I finally got around to taking a good look at this last night after work. I adapted a single page d3 app that I'd written many years ago to model sous vide cooking.
A few comments:
- Probably needs some more documentation. Maybe a quick reference for the API that's available and a list of keystrokes. I will admit though, I bailed halfway through the tutorials so I could start playing with it.
- I'd like to be able to drag and drop cells (or otherwise rearrange them). I created my page, but then wanted to better organize it. Making new cells and cut/pasting content was annoying.
- The Opt-Return thing is killing me. It executes the cell in Jupyter and splits the cell into two in Observable. I can't think of a reason I'd want to split a cell in the middle. I believe splitting in the middle will always result in two cells with syntax errors. Perhaps just always create and focus a new, empty cell?
- It'd be nice if there was an easy way to delete a cell. Currently, I'm emptying it and then joining with the previous via option-del, which concatenates cells. I believe joining two cells with content will always result in a syntax error, so I'm not sure the utility of that.
- The samples were useful, especially for learning how d3 fits into this. I ended up breaking up a long function (to build the d3 graph) into multiple cells, taking cues from the samples. (I also learned that d3 has changed a little since v3.)
It would also be nice to be able to rename variables (i.e. refactor), but that's not a critical feature.
A few comments:
- Probably needs some more documentation. Maybe a quick reference for the API that's available and a list of keystrokes. I will admit though, I bailed halfway through the tutorials so I could start playing with it.
- I'd like to be able to drag and drop cells (or otherwise rearrange them). I created my page, but then wanted to better organize it. Making new cells and cut/pasting content was annoying.
- The Opt-Return thing is killing me. It executes the cell in Jupyter and splits the cell into two in Observable. I can't think of a reason I'd want to split a cell in the middle. I believe splitting in the middle will always result in two cells with syntax errors. Perhaps just always create and focus a new, empty cell?
- It'd be nice if there was an easy way to delete a cell. Currently, I'm emptying it and then joining with the previous via option-del, which concatenates cells. I believe joining two cells with content will always result in a syntax error, so I'm not sure the utility of that.
- The samples were useful, especially for learning how d3 fits into this. I ended up breaking up a long function (to build the d3 graph) into multiple cells, taking cues from the samples. (I also learned that d3 has changed a little since v3.)
It would also be nice to be able to rename variables (i.e. refactor), but that's not a critical feature.
Glanced over this yesterday, came back to it today when I saw it was by jashkenas, who has done some real innovative work over the years. @jashkenas -- is this your primary project now? Are you building a business around this or is it more for fun/open source?
Would be curious to hear plans to monetize it.
Would be curious to hear plans to monetize it.
Would be cool if there was reason / typescript support.
How is this different than Ipython/Jupyter notebooks? Whys everyone freaking out about this.
A few differences. 1. You don’t have to install anything, so running (or forking) someone else’s notebook is as simple as clicking a link. 2. It’s reactive, which means simpler code and better feedback, particularly when building user interfaces or analyzing realtime data. 3. It runs in the browser so you have direct access to powerful graphics (and GPU computation): Canvas, SVG, WebGL. 4. You can quickly import code for reuse from other notebooks, making a notebook a kind of lightweight library as well as a human-readable document.
> 3. It runs in the browser so you have direct access to powerful graphics (and GPU computation): Canvas, SVG, WebGL
Jupyter is a web server. Notebooks run in the browser. iPython the kernel can run on the command line in addition to the browser.
> 1. You don’t have to install anything, so running (or forking) someone else’s notebook is as simple as clicking a link
Jupyter lets you clone notebooks with the click of a button.
Also, you can run Javascript from a Notebook cell, and there are Python libraries that interface with the JS visualization libraries, like Plotly.
Jupyter is a web server. Notebooks run in the browser. iPython the kernel can run on the command line in addition to the browser.
> 1. You don’t have to install anything, so running (or forking) someone else’s notebook is as simple as clicking a link
Jupyter lets you clone notebooks with the click of a button.
Also, you can run Javascript from a Notebook cell, and there are Python libraries that interface with the JS visualization libraries, like Plotly.
Jupyter Notebooks run in the browser but what examples are there of being able to write Python and have the same kind of access to the DOM and graphics APIs like you have with JavaScript?
ipywidgets for creating interactive html widges & dashboards and Plotly, Bokeh, and nvd3 are 3 examples for visualaition that integrates Python with JS libraries.
Of course JS gives you full flexibility for talking to the DOM and graphics APIs, but web apps are not the main focus of data science. The web is just one possible means to presentation or collaboration.
Of course JS gives you full flexibility for talking to the DOM and graphics APIs, but web apps are not the main focus of data science. The web is just one possible means to presentation or collaboration.
acosmism(1)
Were there objections by collaborators or in your own mind, of whether enough people would abandon Jupyter Notebooks to use Observable?
Or was it just not an issue, because you figured 1) not having to installing anything and 2) reactive would be enough to win converts?
Or was it just not an issue, because you figured 1) not having to installing anything and 2) reactive would be enough to win converts?
basically if you have a bunch of code blocks that depend on each others' results - you don't have to remember which ones to update if you changed one, it will update automatically.
especially useful with data visualization/exploration, but also helps a lot if you have a complex sequence where it's easy to get lost and forget to recompute something. there's a youtube link in the comments here which explains a lot.
especially useful with data visualization/exploration, but also helps a lot if you have a complex sequence where it's easy to get lost and forget to recompute something. there's a youtube link in the comments here which explains a lot.
> you don't have to remember which ones to update if you changed one, it will update automatically.
Although I see how that would enable rapid iteration and make e.g. creating nice visualizations much easier, I don't think it should be mandatory. The user may want to control the state their notebook and variables are in manually.
Although I see how that would enable rapid iteration and make e.g. creating nice visualizations much easier, I don't think it should be mandatory. The user may want to control the state their notebook and variables are in manually.
Isn't that like saying you want to control your execution manually? Basically, this just get rids of the manual refresh button that would be required to sync pieces of dependent computations after code or data changes somewhere. It is as if the graphics were running in immediate mode being refreshed completely on every frame (a common goal for live programming systems).
> The user may want to control the state their notebook and variables are in manually.
Can you share an example where you’d want that behavior? Spreadsheet cells don’t ask you when they update themselves.
Can you share an example where you’d want that behavior? Spreadsheet cells don’t ask you when they update themselves.
Its still lacking some core features that make jupyter easier to use. I went through the tutorials and 10 min earthquake videos.
There's no shortcut for deleting items on the "help" section. No WYISWYG editor for that either. So if you want to delete a bulletpoint you accidentally made, it just shows up as "undefined"
There's not a way to bind your own hotkeys. Some of default hotkeys get overridden by my own set of hotkeys, so this isn't too helpful
I'm not a big fan of the UI/UX. I see where its going, collapsing items you don't want to see. But this gets rather tedious to always do everytime. There should be a default setting that lets you bind whether pins are defaulted or not.
There's no shortcut for deleting items on the "help" section. No WYISWYG editor for that either. So if you want to delete a bulletpoint you accidentally made, it just shows up as "undefined"
There's not a way to bind your own hotkeys. Some of default hotkeys get overridden by my own set of hotkeys, so this isn't too helpful
I'm not a big fan of the UI/UX. I see where its going, collapsing items you don't want to see. But this gets rather tedious to always do everytime. There should be a default setting that lets you bind whether pins are defaulted or not.
Great product! I've been using RunKit (https://runkit.com/home) a bit, what would you say are the key differentiators?
Yeah, a while ago I discovered this slick JS notebook app called Tonic, and could never remember the name until one day I came across this thing called RunKit and I thought that it must be a clone. Nope, name just changed when Striped acquired it.
When I saw this app, I thought Tonic must have gone through another rebranding. Wrong again! This just looks more geared for data visualization and collaboration though. Would be curious to see if Tonic/RunKit (or iPython even) were used as inspiration.
When I saw this app, I thought Tonic must have gone through another rebranding. Wrong again! This just looks more geared for data visualization and collaboration though. Would be curious to see if Tonic/RunKit (or iPython even) were used as inspiration.
This is fantastic & a huge congrats from the Azure Notebooks team!
Legitimately very cool. As someone else mentioned I’d also really like the ability to embed these on another page, and possibly have these run locally offline.
I do have a question: for notebooks linking to outside notebooks: is there anything that prevents breakages? Ie external notebook variable name changes or is completely removed, would parts of my notebook relying on that one then also fail? Is there some kind of public versioning of notebooks so that I can refer to a value at notebook ref,v1?
I do have a question: for notebooks linking to outside notebooks: is there anything that prevents breakages? Ie external notebook variable name changes or is completely removed, would parts of my notebook relying on that one then also fail? Is there some kind of public versioning of notebooks so that I can refer to a value at notebook ref,v1?
That's a fantastic question.
It's not ready yet, but we're hoping to roll out import pinning for notebooks soon.
Each notebook is versioned — every time you make a change and re-run the cell, a version is saved to the server. And every time you publish, that's a publication of a specific version of the notebook.
Our plan is to pin the version of any code you import from outside the notebook, including require(npm-module), import {value} from "other-notebook", and even the version of the Observable standard library that was current when you wrote the notebook.
Hopefully, that should provide a good foundation for robust notebooks that don't break as libraries change over time. Of course, you'll be able to re-pin a new version if you want to upgrade something.
It's not ready yet, but we're hoping to roll out import pinning for notebooks soon.
Each notebook is versioned — every time you make a change and re-run the cell, a version is saved to the server. And every time you publish, that's a publication of a specific version of the notebook.
Our plan is to pin the version of any code you import from outside the notebook, including require(npm-module), import {value} from "other-notebook", and even the version of the Observable standard library that was current when you wrote the notebook.
Hopefully, that should provide a good foundation for robust notebooks that don't break as libraries change over time. Of course, you'll be able to re-pin a new version if you want to upgrade something.
Awesome. Thanks for replying. I'll definitely be watching your team at work :)
Any plans to make the notebooks embeddable in other sites?
Yes, and single interactive cells as well.
A notebook already runs in a sandboxed iframe, and is able to load its dependencies there (libraries, data, values from other notebooks). So we're in quite good shape to be able to do that soon.
A notebook already runs in a sandboxed iframe, and is able to load its dependencies there (libraries, data, values from other notebooks). So we're in quite good shape to be able to do that soon.
Cool! I've been looking for something like this for a while and couldn't have asked for better people to tackle it.
Fantastic.
One final question/request (for which I'd happily pay for!) : it would be awesome to have a path away from Observable's infra if desired. Say I _really_ want to host a particular notebook locally: is something like that planned? I know this is not a trivial feature since notebooks can call other notebooks, but I'd love to develop stuff on Observable knowing that should the worst happen and it doesn't exist anymore, I can run it all locally on my webpage.
One final question/request (for which I'd happily pay for!) : it would be awesome to have a path away from Observable's infra if desired. Say I _really_ want to host a particular notebook locally: is something like that planned? I know this is not a trivial feature since notebooks can call other notebooks, but I'd love to develop stuff on Observable knowing that should the worst happen and it doesn't exist anymore, I can run it all locally on my webpage.
Yes, something like that is in the roadmap.
We’ve been scrambling to get things ready enough for this initial launch, but Mike managed to get the Standard Library open sourced this morning: https://github.com/observablehq/notebook-stdlib
One of the next pieces we'd like to open-source is the Notebook Runtime — All of the JS that you need to take a blank webpage, and host a notebook within it.
After that, there's some file format details to figure out — hopefully a notebook can just be published and consumed as a standard ES Module. But then we'll be most of the way there.
We’ve been scrambling to get things ready enough for this initial launch, but Mike managed to get the Standard Library open sourced this morning: https://github.com/observablehq/notebook-stdlib
One of the next pieces we'd like to open-source is the Notebook Runtime — All of the JS that you need to take a blank webpage, and host a notebook within it.
After that, there's some file format details to figure out — hopefully a notebook can just be published and consumed as a standard ES Module. But then we'll be most of the way there.
This is cool, but I'm annoyed at yet more intentional overlapping of names in the JS ecosystem.
Strikes me that a markdown syntax with small extensions would serve well as a format for this. This seems to invert the default mode from text to code. Is there a reason for this?
Also, where are the notebook stored? Can I get an offline copy? Wasn't obvious from a quick look.
Also, where are the notebook stored? Can I get an offline copy? Wasn't obvious from a quick look.
I'd love to see a notebook environment that could somehow combine Python data science tools with DOM/JavaScript interactive visualization tools like D3. Jupyter and crew do former and this does the latter. What we really want is both. Would obviously be tricky to both implement technically and to design a good experience that lets you swap between the two languages and paradigms. Like you might have run webserver behind the scenes to expose the Python produced data to the D3? Stuff like that. Sounds like an interesting challenge. Because I'm generous though, I'd like to give this idea away for free to anyone willing to execute on it. Just joking! Devil in the details. More accurately, I'd just be pumped if someone had the energy to tackle this beast. Also, I'd definitely pay for a hybrid tool like this.
Really useful tool to iterate and riff on ideas.
The visualizations especially good as a feature to promote it.
im sure im not alone id love to create pretty charts and would wow a customer.. But cant justify spending the cycles to fully learn d3 etc..
The visualizations especially good as a feature to promote it.
im sure im not alone id love to create pretty charts and would wow a customer.. But cant justify spending the cycles to fully learn d3 etc..
Nice work, it looks pretty slick. You have a right to be proud of your work.
But....
IMO, the project name is a disaster.
If successful it will muddy the water on searches for RxJS (and other Rx) observables.
Please do like the Amber.js team did when they learned of a name comflict and rebrand as Ember.js. That would be awesome.
But....
IMO, the project name is a disaster.
If successful it will muddy the water on searches for RxJS (and other Rx) observables.
Please do like the Amber.js team did when they learned of a name comflict and rebrand as Ember.js. That would be awesome.
[deleted]
It's amusing that their markdown example is not reactive.
so... what IS observable?
I like it, but I'm not sure I have a user case for it.
Sorry, I can only see "Oops, an unexpected error occurred."
Nick Strayer shows how t-SNE is similar (equivalent?) to a force-directed graph layout: https://beta.observablehq.com/@nstrayer/physics-based-t-sne
Jim Bumgardner published a tutorial that starts with drawing circles with canvas, then fibonacci spirals, then colorful sunflower seeds: https://beta.observablehq.com/@jbum/circles-spirals-and-sunf...
Kamil Slowikowski drew a lovely Barnsley Fern: https://beta.observablehq.com/@slowkow/barnsley-fern
Alan Palazzolo did a Mandelbrot fractal with randomized coloring: https://beta.observablehq.com/@zzolo/mandlebrot-set
Justin Palmer uses PROJ4JS to transform and scale a local map: https://beta.observablehq.com/@caged/local-map-projection-wi...