How a PhD astrophysicist thinks about data(hex.tech)
hex.tech
How a PhD astrophysicist thinks about data
https://hex.tech/blog/friends-of-data/melodie-kao-1/
31 comments
Sequence read archive has been such a boon for developing reproducible biological pipelines without having to worry about data. A paper references a dataset by ID and I can use it as input for my pipelines and keep the raw data locally only for as long as its needed to generate analysis within the running pipeline. I can even set threshold levels of how much local or cloud compute resources should be used at a time if I didn't want to exhaust my systems with one job.
Do people ever make a torrent of their open data and link to it?
Not for these size datasets. Torrents are just too small and unreliable, except for the very most popular items. Slightly out of mainstream movies, which are small and likely more popular than data for an obscure science experiment, are nearly impossible to find seeders for.
So I'd guess no seeders want to host tens to thousands of terabyte torrents, and then thousands to millions of those for all the different datasets grabbed by all the science projects all over the world. The odds of being able to download one of these on demand is just about zero.
So I'd guess no seeders want to host tens to thousands of terabyte torrents, and then thousands to millions of those for all the different datasets grabbed by all the science projects all over the world. The odds of being able to download one of these on demand is just about zero.
In my experience this is very rare, but that's my plan going forward. I can't guarantee that seed will stay up if I'm not paying attention to it though, so in practice it still means emailing the corresponding author. And actual transfer may still require sending hard drives through the mail. But publishing a torrent at least indicates willingness and preparation to share data, which is worth something.
For anyone interested in what an astophysicist gets up to, I'd recommend Dr Becky
https://www.youtube.com/watch?v=XW_qIqLhPkI
and all her other YouTube vids.
https://www.youtube.com/watch?v=XW_qIqLhPkI
and all her other YouTube vids.
> How a PhD astrophysicist thinks about data
I kinda feel this should be "How a PhD astrophysicist thinks"
I kinda feel this should be "How a PhD astrophysicist thinks"
this is an unnecessarily negative comment but astrophysicists get too much credit because of their cool sounding title. I have worked/collabed with plenty and their math/programming/modeling skills are not better than the average compsci,physics phd.
I did my masters in astrophysics, specialised from being a general physicist for my bachelors.
I was reading the piece, waiting for the punchline of what kind of unholy beast of a workstation she was using, and wasn’t disappointed.
Thing is, physicists, heck, scientists in general, are not programmers. Fortran 77 and python are pretty much the only shows in town - and your usual data crunching script will be huge, procedural, in log time, and will eat mountains of memory while making disks thrash as hard as humanly possible.
For instance, I helped out a postdoc who I shared a lab with with his ephemeris calculator - it’d take in a series of fits images, and it’d tell you the ephemeris of whatever object you chose - by editing the source, and putting in the x/y of the object in the first frame.
Thing was, it took all night to do this for a single object from half a dozen frames. Most of the time was spent opening and closing each file to read each pixel, and then stuffing those pixels into a gigantic array, and writing that array to text files, and then re-reading those files, and then doing matrix multiplication and all sorts of amazingly baroque stuff that must have seemed like a good idea at the time. He was running it on a monster (for the time) of a workstation, with 64GB of ram and several TB of storage.
I banged together an app in C++ for him with a basic tcl/tk gui, and what had taken him a day of setup and a night of processing and an ungodly machine instead took about as long as it took for him to click on an object and click “go” - on my creaking laptop with 128mb of ram and a 1.4gb hard drive.
This was far from singular - after this, I found myself being “the guy” to talk to about your slow scripts - and that was basically every script in the department.
So no, not better, considerably worse, and I can’t help but think that having a more cross-disciplinary approach to science (embed tech people!) would yield benefits across the board.
I was reading the piece, waiting for the punchline of what kind of unholy beast of a workstation she was using, and wasn’t disappointed.
Thing is, physicists, heck, scientists in general, are not programmers. Fortran 77 and python are pretty much the only shows in town - and your usual data crunching script will be huge, procedural, in log time, and will eat mountains of memory while making disks thrash as hard as humanly possible.
For instance, I helped out a postdoc who I shared a lab with with his ephemeris calculator - it’d take in a series of fits images, and it’d tell you the ephemeris of whatever object you chose - by editing the source, and putting in the x/y of the object in the first frame.
Thing was, it took all night to do this for a single object from half a dozen frames. Most of the time was spent opening and closing each file to read each pixel, and then stuffing those pixels into a gigantic array, and writing that array to text files, and then re-reading those files, and then doing matrix multiplication and all sorts of amazingly baroque stuff that must have seemed like a good idea at the time. He was running it on a monster (for the time) of a workstation, with 64GB of ram and several TB of storage.
I banged together an app in C++ for him with a basic tcl/tk gui, and what had taken him a day of setup and a night of processing and an ungodly machine instead took about as long as it took for him to click on an object and click “go” - on my creaking laptop with 128mb of ram and a 1.4gb hard drive.
This was far from singular - after this, I found myself being “the guy” to talk to about your slow scripts - and that was basically every script in the department.
So no, not better, considerably worse, and I can’t help but think that having a more cross-disciplinary approach to science (embed tech people!) would yield benefits across the board.
I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
> But they generally have no working model of how a physical computer actually implements those things.
To be fair, how many blog posts have been written over the years on programmers doing ridiculous things with SQL, or network IO, or file IO, etc. etc.? If trained programmers struggle with these things, I'm willing to give non-programmers some slack.
To be fair, how many blog posts have been written over the years on programmers doing ridiculous things with SQL, or network IO, or file IO, etc. etc.? If trained programmers struggle with these things, I'm willing to give non-programmers some slack.
you are describing my job security. I've had a career spanning 25 years speeding up other peoples scripts.
For example once $LARGE_BIOTECH retired its inhouse supercomputer (A TruCluster composed of multiple GS1280s, the epitome of classical UNIX power) and replaced it with Linux and NFS. The principal engineer noticed his pipeline never finished on Linux but how no idea where to start.
Within five minutes I noticed that GNU/Linux sort uses $TMPDIR for large sorts (multipass). They had pointed TMPDIR at an NFS mount and were doing a multipass sort over NFS. On TruCluster, tmp pointed to an ultrafast cluster diskl.
For example once $LARGE_BIOTECH retired its inhouse supercomputer (A TruCluster composed of multiple GS1280s, the epitome of classical UNIX power) and replaced it with Linux and NFS. The principal engineer noticed his pipeline never finished on Linux but how no idea where to start.
Within five minutes I noticed that GNU/Linux sort uses $TMPDIR for large sorts (multipass). They had pointed TMPDIR at an NFS mount and were doing a multipass sort over NFS. On TruCluster, tmp pointed to an ultrafast cluster diskl.
I obtained a PhD in {astrophysics, theoretical physics, quantum physics, computational physics} a few years ago. Choose one of these keywords: Which one is most catchy? I second you, astrophysics just sounds cool in most cases, whereas quantum physics is obviously much of an advantage if you talk to quantum computing (and AI) people.
The bottom line is: In this kind of science, you are used to "big data", to massively parallel computing, to computation and statistics on various levels of abstraction. But the engineering skills outcome greatly vary, because from the physics perspective, computers are just a tool for getting the job done.
For instance, big data in astrophysics is quite different from big data in accounting. Complexity in astrophysics programming is also very different from complexity in the banking buisness. People tend to get arrogant due to their years-long experience, but in the end all they have is just years-long experience in that particular domain, let it be buisness or physics.
The bottom line is: In this kind of science, you are used to "big data", to massively parallel computing, to computation and statistics on various levels of abstraction. But the engineering skills outcome greatly vary, because from the physics perspective, computers are just a tool for getting the job done.
For instance, big data in astrophysics is quite different from big data in accounting. Complexity in astrophysics programming is also very different from complexity in the banking buisness. People tend to get arrogant due to their years-long experience, but in the end all they have is just years-long experience in that particular domain, let it be buisness or physics.
Not to mention programming isn't a sport.
I don't care who has the best paint brush technique. What is interesting is the art and how to creatively solve an interesting problem. There is no shortage of people who can masterfully paint absolutely uncreative shit.
This article though didn't really get deep enough to learn much from.
I don't care who has the best paint brush technique. What is interesting is the art and how to creatively solve an interesting problem. There is no shortage of people who can masterfully paint absolutely uncreative shit.
This article though didn't really get deep enough to learn much from.
How different is astrophysics from quantum physics? I would think quantum physics has a wide range of applicability over astrophysics when it cones to the variety of fields the research can impact, but this is just a guess based on my limited understanding.
They are the essentially the same and radically different.
That is to say, if you have to ask, be skeptical of any terse answer.
That is to say, if you have to ask, be skeptical of any terse answer.
> this is an unnecessarily negative comment
I'd argue this is simply an unnecessary comment altogether.
I'd argue this is simply an unnecessary comment altogether.
Aren't they all?
I don't think it's too negative. The actual complexity is high for the modeling, but very few of the abstractions are any worse than other many body simulations, unless you're studying specifically black holes or other theoretical edge phenomena, in which case it becomes hard to use the term "astro" instead of just theoretical physicist. Arguably magnetohydrodynamics has some difficult components, but it's not specific to cosmology.
Clickbait titles revolve around sounds and language though, so I don't know if there's a way to combat what's profitable and not strictly speaking misleading.
Clickbait titles revolve around sounds and language though, so I don't know if there's a way to combat what's profitable and not strictly speaking misleading.
…who is giving them too much credit exactly? Doesn’t seem to be this piece?
> this is an unnecessarily negative comment but astrophysicists get too much credit because of their cool sounding title.
I thought it was the oldest joke in the book that at the start of the academic year, physicists pronounce to all and sundry they're superior to all non-physicists. They then they spend the rest of the academic year fighting over what the most superior branch of physics is.
As they say, the best jokes have an element of truth to them...
I thought it was the oldest joke in the book that at the start of the academic year, physicists pronounce to all and sundry they're superior to all non-physicists. They then they spend the rest of the academic year fighting over what the most superior branch of physics is.
As they say, the best jokes have an element of truth to them...
Is there any other field that relies on multiple huge arrays of sensors, distributed around the world, that generate huge amounts of data non-stop? Data that is not only fed into processing pipelines but needs to be explored? Meteorology maybe. Something else?
She actually explains that in the article:
"one object might be anywhere between roughly 50 gigabytes to maybe a terabyte. By the time I'm done reducing that data and imaging it, it'll probably have roughly tripled to quadrupled in size. I tend to have larger surveys, so by the time I'm done I might have several tens of terabytes of data."
> their math/programming/modeling skills are not better than the average compsci,physics phd.
No, but the data they handle regularly is just much, much larger than in many other fields.
She actually explains that in the article:
"one object might be anywhere between roughly 50 gigabytes to maybe a terabyte. By the time I'm done reducing that data and imaging it, it'll probably have roughly tripled to quadrupled in size. I tend to have larger surveys, so by the time I'm done I might have several tens of terabytes of data."
> their math/programming/modeling skills are not better than the average compsci,physics phd.
No, but the data they handle regularly is just much, much larger than in many other fields.
Earth observation satellites produce an even larger amount of data. It is stored forever in publicly accessible places (a large amount of it, for free).
> Earth observation satellites produce an even larger amount of data.
Do they? I don't know and I am really curious: Do you have any source for this?
> It is stored forever in publicly accessible places (a large amount of it, for free).
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Do they? I don't know and I am really curious: Do you have any source for this?
> It is stored forever in publicly accessible places (a large amount of it, for free).
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Sure they do. Even if you only count optical satellites, each sensor produces an image of about 40.000 x 40.000 pixels every few seconds. There are hundreds of satellites doing that and sending the images to earth. All of these data is archived indefinitely and widely accessible.
> How to get the data from its source into your number crunchers?
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.
> How to get the data from its source into your number crunchers?
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.
genome sequencers are sensors distributed around the work producing 15PB of intermediate data a day. If desired (IE, smebody had the capital and operating) it could be many multiples of that.
Astro is one of the areas where compsci and physics PhDs can come in and be helpful. The underlying data model of the universe is far more amenable to data processing than biological data. I was a biologist who switched to astro for a while and I was totally blown away at how "easy" it was, especially the data processing. The scientists were typical in their data analytics ability (roughly the same level of ability across many fields).
Once I visited the astro folks at Caltech and they were light years ahead of everybody else in every tchnical dimension. But that's CAltech for you.
Once I visited the astro folks at Caltech and they were light years ahead of everybody else in every tchnical dimension. But that's CAltech for you.
It would seem that the interesting modern astrophysics theories were created, bottom up, from fundamental physics and math. Not by looking at the sky. My view of astrophysics is that they catalogue the sky and know interesting party facts but I wouldn't think a real genius would be an astrophysicist as their specialty. It also kind of bugs me that they can't really perform experiments through the telescope which kind of brings into question what is the actual definition of science, which I think is actually much looser than what we are taught in school. Sorry to join the astrophysics bashing train..
Exactly. I care about how a PhD astrophysicist thinks about data just as much I care how a biologist, psychologist, or sociologist researcher thinks about data.
That is, not so much.
At the PhD and research level the data types and questions are so specific that the analyses often are bespoke (in academia, the more bespoke the analysis, the more you can sell its novelty). PhD-level data analyses have very little relevance outside of its own field.
There is nothing profound to be had here, even if they have the prestigious title of "PhD astrophysicist".
That is, not so much.
At the PhD and research level the data types and questions are so specific that the analyses often are bespoke (in academia, the more bespoke the analysis, the more you can sell its novelty). PhD-level data analyses have very little relevance outside of its own field.
There is nothing profound to be had here, even if they have the prestigious title of "PhD astrophysicist".
It's certainly interesting when we stop to think about the elevated value we place on so many different job titles, viewing people as inherently super intelligent and failing to allow room for them to be humans as fallible as the next.
I think that depends on the phd. In biology at least, a lot of the data types and questions seem specific and bespoke, but they are really not. Its often just tabular data you are working with, and you are generating the same models as any other data scientist generates to find significance in tabular data. The only difference is in interpreting the significance of the model output, but the tooling is often the same as in a lot of fields. This is why PhD computational biologists have no problems pivoting into all sorts of distally related industries from pure biology, to data science, to computer science, ad tech, or even management consulting.
Hosting is a big challenge for open science. Whenever I see someone calling for open data, I feel half enthusiasm for the movement to cooperate more and half dread at the prospect of trying to comply. I used OSF to publish one of my datasets, but they instituted storage limits that would prevent doing the same in the future. And exposure to thousands of dollars in surprise fees from personal archival in cloud hosts isn't acceptable. Which leaves the status quo of email the author for data, hope they respond, as disappointingly the best option.
Open code is trivial to provide in comparison, but also less useful.