Announcing .NET Jupyter Notebooks(hanselman.com)
hanselman.com
Announcing .NET Jupyter Notebooks
https://www.hanselman.com/blog/AnnouncingNETJupyterNotebooks.aspx
82 comments
Mike Bostock is a beast - this d3.js example from 2013 still blows my mind:
https://archive.nytimes.com/www.nytimes.com/interactive/2012...
I haven’t gotten properly stuck into observable - in what ways is it better than using a jupyter notebook?
I haven’t gotten properly stuck into observable - in what ways is it better than using a jupyter notebook?
I don't know about "better" as I don't have a lot of experience with either—I've only dabbled or used them to prototype something small. The major differentiator is the JS implementation and ability to import libraries from sibling notebooks and other sources.
But yeah I'd have to agree that Bostock is a beast.
But yeah I'd have to agree that Bostock is a beast.
I like Observable for sharing JS tutorials - was just updating https://observablehq.com/@lmeyerov/manipulating-flat-arrays-... today.
Observable does some things quite well for that use case:
-- Easier dependencies: no need for 'npm install', just 'require(...)'
-- URL publishing
-- Collaborative merge flow
However, Jupyter made some good decisions that make it win over Observable for our day-to-day data work:
-- Manual reexecution vs. automatic: when working with big data, outside APIs, etc., Observable's automatic reexecution is a non-starter
-- Fully open source, embeddable, and successful history of non-vc funding: Jupyter is organized and provided in a way companies ( who aren't Amazon ;-)) can rally around, evidenced by contributions by Bloomberg etc.
-- Access to underlying unix/windows env and multiple environments (multiple Python versions, ...). As much as I wish Observable's choice of JS could provide a viable data environment, and I've personally invested in making it so and the path to making it first-class is clear, we need way more gov/google/nvidia/etc. support.
Ultimately, for not-too-sensitive collaborative data work, we use Google Colab, then offline and sensitive commercial work via Jupyter (Graphistry ships with it preloaded for GPU dataframe & GPU visual graph analytics goodness!), and if we did more in JS tutorial land, Observable would be my first choice.
Observable does some things quite well for that use case:
-- Easier dependencies: no need for 'npm install', just 'require(...)'
-- URL publishing
-- Collaborative merge flow
However, Jupyter made some good decisions that make it win over Observable for our day-to-day data work:
-- Manual reexecution vs. automatic: when working with big data, outside APIs, etc., Observable's automatic reexecution is a non-starter
-- Fully open source, embeddable, and successful history of non-vc funding: Jupyter is organized and provided in a way companies ( who aren't Amazon ;-)) can rally around, evidenced by contributions by Bloomberg etc.
-- Access to underlying unix/windows env and multiple environments (multiple Python versions, ...). As much as I wish Observable's choice of JS could provide a viable data environment, and I've personally invested in making it so and the path to making it first-class is clear, we need way more gov/google/nvidia/etc. support.
Ultimately, for not-too-sensitive collaborative data work, we use Google Colab, then offline and sensitive commercial work via Jupyter (Graphistry ships with it preloaded for GPU dataframe & GPU visual graph analytics goodness!), and if we did more in JS tutorial land, Observable would be my first choice.
Very cool, thanks for the detailed reply - your JS tutorial looks great :)
Yeah automatic re-execution sounds like a weird design choice. Think I'll stick with Jupyter notebooks for now...
Yeah automatic re-execution sounds like a weird design choice. Think I'll stick with Jupyter notebooks for now...
> and ability to import libraries from sibling notebooks
The Jupyter documentation includes an example for importing other notebooks here: https://jupyter-notebook.readthedocs.io/en/stable/examples/N...
The Jupyter documentation includes an example for importing other notebooks here: https://jupyter-notebook.readthedocs.io/en/stable/examples/N...
completely agree, that thing is sick!
while you're browsing: this is equally great :)
https://archive.nytimes.com/www.nytimes.com/interactive/2013...
while you're browsing: this is equally great :)
https://archive.nytimes.com/www.nytimes.com/interactive/2013...
Yeah I remember seeing that one a few years ago - it's also super cool :)
Especially crazy that these work as well and quickly on mobile!
Especially crazy that these work as well and quickly on mobile!
No self hosting? What do you mean?
The (reactive JavaScript) Observable runtime is open source, and every notebook is available compiled as an ES Module.
You can embed notebooks in their entirety on any webpage, or just grab the embed code for an individual cell, if the notebook produces a single visualization.
For the details in all their nitty gritty, see: https://observablehq.com/@observablehq/downloading-and-embed...
The (reactive JavaScript) Observable runtime is open source, and every notebook is available compiled as an ES Module.
You can embed notebooks in their entirety on any webpage, or just grab the embed code for an individual cell, if the notebook produces a single visualization.
For the details in all their nitty gritty, see: https://observablehq.com/@observablehq/downloading-and-embed...
Does that now include the ability to create and edit, or still just run? Having to rely on a third party to create and edit could be problematic for some purposes.
A comment from May last year [1] says "But that won’t include the editable notebook interface or the rest of observablehq.com - unlike Jupyter, we’re really aiming to create a community and cross-pollination of concepts and code rather than individual installations. We might eventually launch something that enables offline editing, but that’s a bit further out in the future."
[1] https://talk.observablehq.com/t/noob-can-you-run-your-own-ob...
A comment from May last year [1] says "But that won’t include the editable notebook interface or the rest of observablehq.com - unlike Jupyter, we’re really aiming to create a community and cross-pollination of concepts and code rather than individual installations. We might eventually launch something that enables offline editing, but that’s a bit further out in the future."
[1] https://talk.observablehq.com/t/noob-can-you-run-your-own-ob...
[deleted]
Also polynote.
> Polynote is an experimental polyglot notebook environment. Currently, it supports Scala and Python (with or without Spark), SQL, and Vega.
https://github.com/polynote/polynote
> Polynote is an experimental polyglot notebook environment. Currently, it supports Scala and Python (with or without Spark), SQL, and Vega.
https://github.com/polynote/polynote
I've been tinkering with Data-Forge Notebook.
https://www.data-forge-notebook.com/
It supports javascript and typescript relatively new but capable.
It supports javascript and typescript relatively new but capable.
Nextjournal's another one in this space, more focused on reproducibility and polyglot support.
https://nextjournal.com
https://nextjournal.com
Even though I love Python I don't really like Jupyter, seems like with a dev environment like VS or VSCode I can't quite find the right use case.
With C# it might be a different story, since the development cycle is fundamentally edit-compile-run an environment like Jupyter might give me an extra platform for in-between testing. I already use the Interactive built into VS quite a bit to figure out, for example, just the right format string for Datetime.Now.ToString() or stuff like that.
With C# it might be a different story, since the development cycle is fundamentally edit-compile-run an environment like Jupyter might give me an extra platform for in-between testing. I already use the Interactive built into VS quite a bit to figure out, for example, just the right format string for Datetime.Now.ToString() or stuff like that.
The benefit of Jupyter notebooks really is about being able to share chunks of code and write narrative around it.
It isn’t really aiming to replace a dev environment like VSCode.
We’re seeing a rise of notebooks in particular because they encourage writing a lot of plain text around your code, and that’s really good both for teaching, and promoting good documentation skills for learners.
It isn’t really aiming to replace a dev environment like VSCode.
We’re seeing a rise of notebooks in particular because they encourage writing a lot of plain text around your code, and that’s really good both for teaching, and promoting good documentation skills for learners.
>share chunks of code and write narrative around it.
>encourage writing a lot of plain text around your code
Is this targeted at non techies? Teachers?
Can someone provide some good example of this? Not having used it much trying to understand what the advantages are over good quality documentation/tutorial pages on say msdn.
>encourage writing a lot of plain text around your code
Is this targeted at non techies? Teachers?
Can someone provide some good example of this? Not having used it much trying to understand what the advantages are over good quality documentation/tutorial pages on say msdn.
Jupyter came from the scientific community (originally the IPython Notebook - IPython itself was made by a physics grad student).
The use case is basically a scientific journal. You collect data. Analyze it. Make plots. Write notes (e.g. with equations, or diagrams). All in one place.
Then you can share the notebook with anyone else. If they have similar data (in the same format), they can execute the notebook on their data and get the plots all in one place.
It's great for data oriented work. It was never meant as an IDE replacement, nor as a general purpose development environment.
The use case is basically a scientific journal. You collect data. Analyze it. Make plots. Write notes (e.g. with equations, or diagrams). All in one place.
Then you can share the notebook with anyone else. If they have similar data (in the same format), they can execute the notebook on their data and get the plots all in one place.
It's great for data oriented work. It was never meant as an IDE replacement, nor as a general purpose development environment.
Honest question, does anyone ever reuse the notebooks they receive with their data or is it just a matter of you could if you wanted but we don't ?
Yes.
In a previous job I had set up a notebook to analyze some of the models our team produced - it was essentially a Q/A notebook that generated data from our models, algorithmically looked for unphysicalities in our models, and plotted any it found.
The rest of my team used it for the models they were working on. The alternative/old way was just too painful (lots of manual steps).
But although reuse by others is a touted feature, it's not really that important. It's incredibly useful for one's own workflow. Think of the convoluted Excel sheets people often have at engineering companies. They get new data, copy and paste it into Excel, and get new plots. This is no different.
In a previous job I had set up a notebook to analyze some of the models our team produced - it was essentially a Q/A notebook that generated data from our models, algorithmically looked for unphysicalities in our models, and plotted any it found.
The rest of my team used it for the models they were working on. The alternative/old way was just too painful (lots of manual steps).
But although reuse by others is a touted feature, it's not really that important. It's incredibly useful for one's own workflow. Think of the convoluted Excel sheets people often have at engineering companies. They get new data, copy and paste it into Excel, and get new plots. This is no different.
> Can someone provide some good example of this? Not having used it much trying to understand what the advantages are...
Sure! Here are some ways I use notebooks:
Imagine any scenario in which you'd rather make a video to demonstrate how something works. For a lot of those cases, a notebook is a great way to show how the code works and how it all comes together.
Another set of use cases is you've done some research, need to call a few api's, post process the results, and share that with your team or have it ready for later to do something similar. You could think of this as a "Super Postman". For example, you need to run some api's in a loop, filter the results, and accumulate some things and print totals at the end.
Another use case is you're troubleshooting an issue, and want to keep notes of what you've tried, what the results were, and be ready to go back and run things again.
Sure! Here are some ways I use notebooks:
Imagine any scenario in which you'd rather make a video to demonstrate how something works. For a lot of those cases, a notebook is a great way to show how the code works and how it all comes together.
Another set of use cases is you've done some research, need to call a few api's, post process the results, and share that with your team or have it ready for later to do something similar. You could think of this as a "Super Postman". For example, you need to run some api's in a loop, filter the results, and accumulate some things and print totals at the end.
Another use case is you're troubleshooting an issue, and want to keep notes of what you've tried, what the results were, and be ready to go back and run things again.
It solves the problem of scientists that have done experiments, but have no way to share and have their experiments and have them validated using the same procedures and programs. At least that's the way I understand it.
You can even have a notebook run in a cluster.
http://ipython.org/ipython-doc/dev/parallel/
You can even have a notebook run in a cluster.
http://ipython.org/ipython-doc/dev/parallel/
> You can even have a notebook run in a cluster.
Well, that's exactly what Databricks does.
Well, that's exactly what Databricks does.
This explains the advantages very well imo
https://en.wikipedia.org/wiki/Literate_programming#Contrast_...
https://en.wikipedia.org/wiki/Literate_programming#Contrast_...
I do financial data analysis and Jupyter Notebooks allows me to run scripts while explaining my findings/arguments about the data within the same space.
It's a convenient way to present both data and narrative to whoever might need to see it.
I also like being able to run snippets when writing scripts, at least when it comes to building/testing. I'm more on the data science side than the dev side though, so that's probably why Jupyter Notebooks appeal to me more than other IDEs.
It's a convenient way to present both data and narrative to whoever might need to see it.
I also like being able to run snippets when writing scripts, at least when it comes to building/testing. I'm more on the data science side than the dev side though, so that's probably why Jupyter Notebooks appeal to me more than other IDEs.
That reminded me of Literate Programming and sure enough the wikipedia article for that on mentions Jupyter.
Try F# feels like Python in most regards but ends up having type inference.
Edit: if you think “oh that’s nice, just like Java/C++”, HECK NO. It goes beyond this, where complex modeling can easily be represented and things like pattern matching and other features not known in other langs
Edit: if you think “oh that’s nice, just like Java/C++”, HECK NO. It goes beyond this, where complex modeling can easily be represented and things like pattern matching and other features not known in other langs
I really wish F# were the dominant language in the AI glue space instead of Python :| Typed array/tensor shapes? Yes please.
Hello! We're taking a look at improving these kinds of programming scenarios in F# for the next version of the language. Since F# is first-class in the jupyter experience, one of those hurdles is already done!
That said, improvements to the language can only go so far. We're trying to figure out what a good set of libraries for the whole range of ML tasks looks like for F# and .NET.
That said, improvements to the language can only go so far. We're trying to figure out what a good set of libraries for the whole range of ML tasks looks like for F# and .NET.
Continue the good work :) I use mostly C# out of habit but F# is my favorite in terms of expressiveness. I'm planning a series of educational videos in French about the language because it doesn't have the exposure it merits.
I've been out of the scene for a bit but last I looked Miguel had created tensorflow bindings for .Net. With some magic require incantations I was able to get that into F# and it actually worked.. Looks like there is SciSharp now and that project looks like it's heading in the right direction.
The key, IMHO, is going to be getting some influential people and projects on F# generating code and blog content. The user experience of getting setup for them and the audience they reach needs to be streamlined. From there if it JustWorks™ F# should do the rest of the selling :)
And to digress, because I always mention this when talking about F#, it deserves a batteries included web framework story as good as Elixir's Phoenix.
The key, IMHO, is going to be getting some influential people and projects on F# generating code and blog content. The user experience of getting setup for them and the audience they reach needs to be streamlined. From there if it JustWorks™ F# should do the rest of the selling :)
And to digress, because I always mention this when talking about F#, it deserves a batteries included web framework story as good as Elixir's Phoenix.
> And to digress, because I always mention this when talking about F#, it deserves a batteries included web framework story as good as Elixir's Phoenix.
I'd check out Saturn for that: https://saturnframework.org/
I'd check out Saturn for that: https://saturnframework.org/
I was thinking of writing the bindings to Pytorch directly to C++ for F# now that would be the killer thing to bring on to .net . It’d be like writing As close as writing Python but order of magnitude improvement for performance. But that’s a lot of work and I don’t do much ml work lol so meh Someone should do it
With the caveat of being Windows only, you could have a look into C++/CLI or C++/WinRT.
The thing that I find really daunting with Jupyter is the set of APIs that it seems I need to learn in order to get it to do anything cool, like for example visualizing data or a function, or making an interactive widget that does the same. I don’t remember the same kind of pain when playing around with Mathematica. I really want to like it and learn it — I really like interactive programming. It just never makes it high enough on my brain space list to seem worth it.
Since C# is my daily language, this is pretty cool though. I just really wish that I had a use-case strong enough for it to overcome the initial pains of learning it and the ongoing mental maintenance of staying usefully fluent in yet another set of libraries.
Since C# is my daily language, this is pretty cool though. I just really wish that I had a use-case strong enough for it to overcome the initial pains of learning it and the ongoing mental maintenance of staying usefully fluent in yet another set of libraries.
Agreed on the bewildering number of disparate APIs that you need to bash together in a Jupyter notebook. Data wrangling, plotting, machine-learning, logging, writing to file/DB... It doesn't help that there are often multiple competing packages that solve basically the same problem in different ways, ensuring that whatever choice you make will be fraught with anxiety about not "doing it right."
That being said, I've found the only way I can get interactive programming to stick in my mind is to pick a sample dataset, wonder some open-ended questions about it, and then answer those questions in a Jupyter notebook with the help of Google, documenting your thought process as you go. A great starting point to find real-world sample data is the UCI dataset repository: https://archive.ics.uci.edu/ml/datasets.php
That being said, I've found the only way I can get interactive programming to stick in my mind is to pick a sample dataset, wonder some open-ended questions about it, and then answer those questions in a Jupyter notebook with the help of Google, documenting your thought process as you go. A great starting point to find real-world sample data is the UCI dataset repository: https://archive.ics.uci.edu/ml/datasets.php
My use case is algorithm prototyping/development. It lets me mostly put the code aside and concentrate and quickly iterate on the algorithm. It's just a fancy, persistent, reproducible, REPL.
C# has supported interactive scripting for years now since Roslyn came about. Even VS has a C# Interactive window.
Actually, C# had a REPL even before Roslyn: The Mono project shipped one about 4 years prior.
C# had a pseudo REPL already back then, the immediate window available during debugging sessions.
True, but what made the Mono REPL particularly remarkable was that it launched with much of the functionality of a notebook, with things like inline graphics.
There are approaches to combine the two. See for example this article about Jupyter and vscode [1].
And to ”close the loop”, you can of course access vscode using browser with Visual Studio Online [2].
[1] https://code.visualstudio.com/docs/python/jupyter-support
[2] https://visualstudio.microsoft.com/services/visual-studio-on...
And to ”close the loop”, you can of course access vscode using browser with Visual Studio Online [2].
[1] https://code.visualstudio.com/docs/python/jupyter-support
[2] https://visualstudio.microsoft.com/services/visual-studio-on...
> I already use the Interactive built into VS quite a bit to figure out, for example, just the right format string for Datetime.Now.ToString() or stuff like that.
This is me_irl
This is me_irl
what I end up doing is having both the dev env and jupyter open at the same time - doing dev the normal way and then using a jupyter notebook for q&d manual validation and plots.
like others said already, jupyter isn't a development environment and trying to use it like one is asking for trouble, but it is a great REPL on some awesome steroids.
like others said already, jupyter isn't a development environment and trying to use it like one is asking for trouble, but it is a great REPL on some awesome steroids.
I set up Jupyter for my wider team to use, I thought it would be popular but nearly everyone chose to use PyCharm instead.
I love .NET and I love Jupyter. I don't know how well they will combine though. I feel like the lack of Pandas and flexible typing of Python will make it a lot less useful.
When you use it with F#, you get pretty close to the flexible typing feel of Python :)
I haven't tried it, but what I'm envisioning is LinqPad with cells, including Markdown cells. And that sounds like a nice tool to me.
Even better if it works well with Deedle and FSharp.Charting. But I'm not going to demand everything on release day.
Even better if it works well with Deedle and FSharp.Charting. But I'm not going to demand everything on release day.
It currently works with Deedle and XPlot.Plotly (which I personally far prefer to FSharp.Charting). We're also adding XPlot.GooglePlots support for rendering the charts inline.
I feel like Deedle is dead unfortunately, although I see there has been a bit of commit action recently.
There are a couple of efforts to build .NET wrappers around Pandas, or to reimplement Pandas in .NET; see http://scisharp.github.io
There are a couple of efforts to build .NET wrappers around Pandas, or to reimplement Pandas in .NET; see http://scisharp.github.io
Hello. I am the current maintainer of Deedle. Recently, I have been focused on use cases of MathNet interop. Deedle.Math was released last week. pandas-like matrix dot functions are available now. Lots of linear algebra, stats and finance related functions are added too. More finance domain functions and docs improvements are on my to-do list.
I am really excited about the new Data Frame from corefxlab to complement what Deedle lacks. Since Deedle's original core engineering team has left and moved on, some core technical debt are beyond my capability to address. But Deedle is still the best data frame library in .Net ecosystem for now if you are able to get around some of its odd edges.
I am really excited about the new Data Frame from corefxlab to complement what Deedle lacks. Since Deedle's original core engineering team has left and moved on, some core technical debt are beyond my capability to address. But Deedle is still the best data frame library in .Net ecosystem for now if you are able to get around some of its odd edges.
Hello! Deedle is actively maintained, as is the XPlot library and a few others under the fslaborg organization on GitHub. However, it's not without its flaws and addressing them in the best way possible would be a breaking change. Additionally, the .NET team is working on a core DataFrame API that could service these needs here.
I use the almond scala kernel for jupyter and I think its awesome.
Trying in my limited spare time to learn Scala and use it here and there on the job for script like stuff of our Java sim and simple analysis. I found almond a bit slow in compiling cells and then found BeakerX. Seemed much faster. Have any thoughts between the two (if you've used BeakerX)
Was not aware of BeakerX but happy to find out about it. Eager to try it out now.
This is great news. Jupyter has become my default tool for prototyping code. I keep trying other platforms that should theoretically have the same features, but I just find Jupyter much more pleasant to use.
How do you prevent jupyter getting lost in a sea of browser tabs ? Do you use it in a separate one ?
Exactly, one window for Jupyter and sometimes Tensorboard, one for normal browsing.
I’d love to see some nontrivial side by side examples of F# notebooks compared to python.
Hey there, I work on F# at Microsoft and helped put together the initial set of samples for F#. You can access them here: https://mybinder.org/v2/gh/dotnet/try/master?urlpath=lab
What kinds of nontrivial samples would you be interested in seeing?
What kinds of nontrivial samples would you be interested in seeing?
Not the OP but I enjoyed the "Introduction to F#" notebook there. Interactivity let me see what the error messages are like for eaxmple replacing "negate" with "42" in the pipeline example.
(hint to Jupyter newbies: alt-enter re-evaluates the code block after editing it)
(hint to Jupyter newbies: alt-enter re-evaluates the code block after editing it)
On the introduction to F# sample, there is a comment
The best way to work with DUs is pattern matching. Using the previously-defined type definitions, we can model withdrawing money from an account:
Thanks, fixing that here: https://github.com/dotnet/try/pull/638
Jupyter is such a cool tool, not only for data scientists! Give it a try, guys :D
Are there any plans for implementing code completion/intellisense?
Yes, absolutely.
Integration of ML.NET and https://github.com/SciSharp would be amazing!
Now if only .NET had arguably the world's greatest IDE!
Not my cup of tea as I am more multi platform. Windows only I suppose. F# interesting but still.
.NET core is open source and has worked on windows, mac os, and linux for a while now. I started working on a front end for a C# .net core project recently which is being developed at a primarily Windows shop and I got it running on Ubuntu in 15 minutes. I would have died before picking a Microsoft stack for a project 5 years ago, but the back end for my next project will be .net core. The developer experience and tooling are second to none these days. You really should give it another look.
I use jupyter notebooks with F#/.NET primarily on macOS and Linux :)
.net core is cross platform, so this should work on Linux and Mac.
Not only that, .net core is now essentially the only active branch of .net. The windows only .net framework is becoming legacy.
Kind of. .NET Framework will be the XP of .NET until Microsoft provides an actual migration path to many of the APIs and frameworks that haven't made it into .NET Core.
Plus some of those APIs that made the cut into .NET Core, like the UI ones, are Windows only.
Plus some of those APIs that made the cut into .NET Core, like the UI ones, are Windows only.
There are also some critical tooling gaps that mean .Net Core is still broken for scenarios that worked a decade ago (FSI package management with F#, specifically).
FSI has never had package management. Are you referring to something else?
Sorry for the loose term. Not package management a la NuGet, but how packages and their references are handled (ie managed), so that they're not available and operative in FSI on .Net Core in VS.
Things that work when compiled, worked about a decade ago, but die on FSI preventing multiple data interaction & scripting scenarios. Ref: https://github.com/dotnet/fsharp/issues/3309 , https://github.com/dotnet/fsharp/pull/5850 , https://github.com/fsprojects/IfSharp/issues/206 , etc etc
Specifically, referencing a library that uses an FSharp.Data type provider, and calling that library in FSI in Visual Studio.
Updated to the latest and greatest, 5 y.o. code looks like this when we execute it:
> System.MissingMethodException: Method not found: 'System.String FSharp.Data.Http.RequestString(System.String ....
Things that work when compiled, worked about a decade ago, but die on FSI preventing multiple data interaction & scripting scenarios. Ref: https://github.com/dotnet/fsharp/issues/3309 , https://github.com/dotnet/fsharp/pull/5850 , https://github.com/fsprojects/IfSharp/issues/206 , etc etc
Specifically, referencing a library that uses an FSharp.Data type provider, and calling that library in FSI in Visual Studio.
Updated to the latest and greatest, 5 y.o. code looks like this when we execute it:
> System.MissingMethodException: Method not found: 'System.String FSharp.Data.Http.RequestString(System.String ....
What about vb.net?
I think the only real drawback is no self hosting.
https://observablehq.com/
A pretty healthy example:
https://observablehq.com/@rreusser/2d-n-body-gravity-with-po...