Laboratory Notebook Skills [pdf](dur.ac.uk)
dur.ac.uk
Laboratory Notebook Skills [pdf]
https://www.dur.ac.uk/resources/physics/students/labs/skills/notebookskills.pdf
23 comments
Have we reach a point where there are similar guides for Jupyter notebooks? Or just for the domain of data science / exploratory statistics
Like, conceptually, what should I keep, and how should I organise my notebook(s)?
Most guides are focused on what you can do (Here's how you draw a graph, here's how you load a kernel), I'd rather read about what I should store in my notebooks, and how many I should have.
Like, conceptually, what should I keep, and how should I organise my notebook(s)?
Most guides are focused on what you can do (Here's how you draw a graph, here's how you load a kernel), I'd rather read about what I should store in my notebooks, and how many I should have.
I agree. From personal experience, here are some organizing principles I find useful for exploratory data analysis:
1. A single Jupyter notebook should either tackle a single question, or use collapsible headings (https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...) to organize separate questions.
2. The Jupyter notebook should be written such that all cells are executable in order, and it should produce exactly the same output every time (unless the input has changed) for reproducibility. The entire notebook should be executable on the order of seconds - if it's taking longer, this is usually a sign that plotting should be scripted instead, or the data needs to be subsetted.
3. The jupyter notebook should have a clearly annotated input file / folder of data (typically pre-processed using scripts). I usually include this in the title. The notebook must have a creation/last modified date (fortunately this is automatic, but it is crucial when the input data can change over time)
4. Observations, general conclusions, preliminary answers to exploratory questions, etc from data analysis should be written in the notebook in comments or Markdown. This is for your future self.
1. A single Jupyter notebook should either tackle a single question, or use collapsible headings (https://jupyter-contrib-nbextensions.readthedocs.io/en/lates...) to organize separate questions.
2. The Jupyter notebook should be written such that all cells are executable in order, and it should produce exactly the same output every time (unless the input has changed) for reproducibility. The entire notebook should be executable on the order of seconds - if it's taking longer, this is usually a sign that plotting should be scripted instead, or the data needs to be subsetted.
3. The jupyter notebook should have a clearly annotated input file / folder of data (typically pre-processed using scripts). I usually include this in the title. The notebook must have a creation/last modified date (fortunately this is automatic, but it is crucial when the input data can change over time)
4. Observations, general conclusions, preliminary answers to exploratory questions, etc from data analysis should be written in the notebook in comments or Markdown. This is for your future self.
This is something we think a lot about at Kyso (https://kyso.io/for-teams) - my startup - we let teams manage analytics knowledge basically by letting teams share Jupyter notebooks - we've spoken to lots of people about how the organise them and are writing a guide really soon that sounds like what you want.
- Keep all your teams notebooks in one place (not on laptops)
- Reuse them liberally (the %run command is excellent and often better than creating python libraries)
- Most important of all is make them readable documents not just list of coding commands, that means you should title things clearly, explain what you going to do.
- Label everything, axis labels are needed on almost every chart, and also make sure you put a title and description for every chart.
- Separating your project into processing and presentation notebooks is really helpful. So one notebook for data prep and another for plots and explanations.
- Keep all your teams notebooks in one place (not on laptops)
- Reuse them liberally (the %run command is excellent and often better than creating python libraries)
- Most important of all is make them readable documents not just list of coding commands, that means you should title things clearly, explain what you going to do.
- Label everything, axis labels are needed on almost every chart, and also make sure you put a title and description for every chart.
- Separating your project into processing and presentation notebooks is really helpful. So one notebook for data prep and another for plots and explanations.
> Reuse them liberally (the %run command is excellent and often better than creating python libraries)
Does that mean you run other notebooks from within notebooks?
Can you ping me or something when the guide is out?
Does that mean you run other notebooks from within notebooks?
Can you ping me or something when the guide is out?
Happy to, but I've no way to ping you?
Everyone's focusing on charts and notebook types; I'd like to highlight something I find particularly important: reporting uncertainties. I strongly agree with recommendations in this article, and personally I don't think a work qualifies as professional if it reports measurements without providing uncertainties. And no, I don't trust it's correctly rounded off (no more digits than justified by certainty level) - too many people don't care or know they should care, and too many areas of endeavor are poisoned by marketing.
A trivial every-day example of why that matters: a person steps on scales twice in a span of few days, and notices 1 kg of gain, and panics. The reaction is completely unwarranted, given that a cheap household electronic scale will be accurate to +/- 0.5kg in this range, and random daily variability of weight is somewhere around +/- 2 kg. They may as well have lost their average weight over that span, but it's hidden under random and systemic errors.
A trivial every-day example of why that matters: a person steps on scales twice in a span of few days, and notices 1 kg of gain, and panics. The reaction is completely unwarranted, given that a cheap household electronic scale will be accurate to +/- 0.5kg in this range, and random daily variability of weight is somewhere around +/- 2 kg. They may as well have lost their average weight over that span, but it's hidden under random and systemic errors.
My version of this, which i maintain to this day, is a git repo of org-mode documents with excecutable inline snippets of code and figures.
It is append-only; i don't delete stuff, usually, i just modify the heading if it's obsolete. (it's easier to read ~text~ than it is to dig through commits)
It is append-only; i don't delete stuff, usually, i just modify the heading if it's obsolete. (it's easier to read ~text~ than it is to dig through commits)
"Draw your figures in Excel" No no no no no no no! Do not draw your figures in Excel. Yes, they can be drawn quicly, but Excel figures suck, are a pain to customize and will introduce bad habits. Learn to draw figures in R using ggplot2.
Pure opinion until you explain why Excel figures suck and which bad habits it introduces?
Not the person to whom you were responding.
I think Excel charts can be quite good. But my impression is that ggplot2 is much more flexible in what you can do with it. It's also more difficult to learn, but that's to be expected.
I think Excel charts can be quite good. But my impression is that ggplot2 is much more flexible in what you can do with it. It's also more difficult to learn, but that's to be expected.
Learn to draw figures in R using ggplot2.
R + ggplot2 was the road I took and taught my students. There are other good tools indeed.
Or use Microcal Origin instead if you are uncomfortable with programming.
> The Level 1 Lab book is a bound A4 notebook
I would add that it should be acid-free paper, very strong binding (sewed+glued) and have graph paper. The acid-free paper can give notebook 20+ year life span, strong binding is important as you may frequently force notebook open with heavy objects, and finally graph paper style allows you to draw complex diagrams easily. Here's notebook I've liked: https://www.amazon.com/gp/product/B071NWSB7P
I would add that it should be acid-free paper, very strong binding (sewed+glued) and have graph paper. The acid-free paper can give notebook 20+ year life span, strong binding is important as you may frequently force notebook open with heavy objects, and finally graph paper style allows you to draw complex diagrams easily. Here's notebook I've liked: https://www.amazon.com/gp/product/B071NWSB7P
Why do they recommend a paper record that can't be backed up and nobody can verify what it said when? Why not a digital file that can be backed up and with a published checksum that can be used to verify what the state was at a given point in time.
Digital notebooks (Electronic Lab Notebooks --- ELNs) exist, but haven't caught on yet.
Many researchers often need to be able to note down arbitrary diagrams, not just text, in real time, which pretty much means low-latency tablet with stylus. This became feasible in just the last decade or so.
Many ELNs do implement published hashes for verification ("trusted timestamping").
Many researchers often need to be able to note down arbitrary diagrams, not just text, in real time, which pretty much means low-latency tablet with stylus. This became feasible in just the last decade or so.
Many ELNs do implement published hashes for verification ("trusted timestamping").
I personally understand better my own ideas (it souds weird i know) when i draw graphics and diagrams, that can be done quickly and easily on paper.
I think it's still true that at certain legal moments a paper record is (all but) required.
would love to hear input from people with both lab and software dev experience to compare and contrast the two notetaking experiences
These are excellent suggestions and are taught in science classes. Except for the use of Excel. Yes, Excel is easy, but its flexibility will lead to sloppiness, and drawing figures sucks. Excels statistical functions are also wrong in some cases. For data analysis, learn R or Python, period. If you have lots of data, learn to use SQLite in addition. The learning curve is steep, but well worth it.
Source: I have a decade of experience in science, and some 5 years in software development.
Source: I have a decade of experience in science, and some 5 years in software development.
So I had the opposite experience-- learned to do all my data processing in undergrad+grad school for physics using Python. Moved out to my first industry job developing simulations and learned that everyone (other scientists, management, etc) would rather me process results in excel (unless we were working on a database scale, in which case we used postgres). I actually had to learn excel properly for the first time for this job.
I'm now at another large university-affiliated research lab and excel is king here as well, though I can get away with using Matlab generated plots in my slides when I'm working solo. People still don't like python for some reason.
I had a similar experience with paper writing-- in academics it was conventional to do everything in LaTeX from my first lab courses in freshman year. In both workplaces, we've just been using word.
And it's not that people don't know python/latex here-- we've just apparently developed a culture of using these matlab/excel/word tools instead.
I'm now at another large university-affiliated research lab and excel is king here as well, though I can get away with using Matlab generated plots in my slides when I'm working solo. People still don't like python for some reason.
I had a similar experience with paper writing-- in academics it was conventional to do everything in LaTeX from my first lab courses in freshman year. In both workplaces, we've just been using word.
And it's not that people don't know python/latex here-- we've just apparently developed a culture of using these matlab/excel/word tools instead.
I appreciate codifying it in a document, but most of the rules listed are taught (at least implicitly) in middle school.
I think I will be coming up with workflows and checkboxes soon for organizing a experiment and easily integrating results of the experiments I am running from disparate software.
I think its very easy to incorporate the stuff in this post, and create a new template in org mode for this. And its so flexible that it will fit all usecases.
Emacs is too awesome.