Why Does the DICOM Standard Exist?(innolitics.com)
innolitics.com
Why Does the DICOM Standard Exist?
https://innolitics.com/articles/dicom-i-facilitating-interoperability/
56 comments
One thing that always annoyed me was the time stamps
So much this. Now your example can still be countered somewhat by taking notes and writing down which scan was taken when. This is usually common practice in research.
But it can get worse when trying to integrate integrating MRI scanner software with external systems which do get a correct timestamp (or at least, different one) and which are used for research experiments like presenting stimuli and recording behaviour all together with getting MRI data.
Scanner software usually doesn't provide any means of starting it except for clicking in the UI so that means the people doing such experiment have to start 2 or more pieces of software manually (and hope they don't forget one). Now if one session lasts 10 minutes and it's not uncommon to do test sessions (with no scan) or test scans (with no corresponding experiment) or make a mistake (start scan with wrong parameters, abort, start again) and a complete test takes a couple of hours, imagine the fun of linking all of that session data together correctly based on just timestamps. I don't even know how much time has been lost or many mistakes were made (and possibly published) because of that. But it won't be 0, it's just too easy to mess up.
Of course some of this is out of the manufacturers contro
Only partly. Some of these systems are always connected to the internet anway so then at least try to make sure NTP works.
So much this. Now your example can still be countered somewhat by taking notes and writing down which scan was taken when. This is usually common practice in research.
But it can get worse when trying to integrate integrating MRI scanner software with external systems which do get a correct timestamp (or at least, different one) and which are used for research experiments like presenting stimuli and recording behaviour all together with getting MRI data.
Scanner software usually doesn't provide any means of starting it except for clicking in the UI so that means the people doing such experiment have to start 2 or more pieces of software manually (and hope they don't forget one). Now if one session lasts 10 minutes and it's not uncommon to do test sessions (with no scan) or test scans (with no corresponding experiment) or make a mistake (start scan with wrong parameters, abort, start again) and a complete test takes a couple of hours, imagine the fun of linking all of that session data together correctly based on just timestamps. I don't even know how much time has been lost or many mistakes were made (and possibly published) because of that. But it won't be 0, it's just too easy to mess up.
Of course some of this is out of the manufacturers contro
Only partly. Some of these systems are always connected to the internet anway so then at least try to make sure NTP works.
Do you know what software is usually used to get fMRI timestamps? My last position was in an fMRI lab, and we had this in-house piece of Windows 98 software that did real-time recording down to <10ms, but it was a huge codebase to maintain and pretty much impossible for me to migrate it to Windows 10. The output file was pretty hard to decode too. I've always wondered what else is out there.
Not 100% sure what you mean with 'fMRI timestamps' but since you say 'recording' I assume it's the TTL/optical trigger produced at the start of each volume? And since you mention windows 98 and < 10mSec the in-house software is maybe using the parallal port? From former experience with such software: that's usually a mess, especially on later versions of Windows where direct access to such port was being ruled out.
Anyway the principle is you use an A/D converter to digitize the signal then save it (either the complete signal or just timestamps of rising edge, the latter can also be done directly on some A/D cards). In the system I'm currently working on I'm using National Instruments PCI-X cards their C SDK to get the data but that's just because I need I/O channels for other things as well and because it has to include online processing of data. Otherwise you could just save the data at a nice and slow rate using Labview or Matlab or ... (pretty sure there are open source solutions as well). And there are many alternatives to National instruments as well of course. All woth resolution down to a couple of nanoseconds if needed. But note that it's still not trivial to get an accurate 'time of day' timestamp from it (should that be needed): at the recording site there's no way of knowing exactly when the data acquisition starts. And there can be clock drift.
Anyway the principle is you use an A/D converter to digitize the signal then save it (either the complete signal or just timestamps of rising edge, the latter can also be done directly on some A/D cards). In the system I'm currently working on I'm using National Instruments PCI-X cards their C SDK to get the data but that's just because I need I/O channels for other things as well and because it has to include online processing of data. Otherwise you could just save the data at a nice and slow rate using Labview or Matlab or ... (pretty sure there are open source solutions as well). And there are many alternatives to National instruments as well of course. All woth resolution down to a couple of nanoseconds if needed. But note that it's still not trivial to get an accurate 'time of day' timestamp from it (should that be needed): at the recording site there's no way of knowing exactly when the data acquisition starts. And there can be clock drift.
Yes, that's it. Sorry it's been a while, and yes it was using the parallel port. I wish I had thought to try modern PCIe cards and different software like Matlab, that would probably have been much easier.
[deleted]
This is why I hate DICOM. DICOM itself is fine, it's just that every device manufacturer mucks it up. I remember once working with a device that claimed to meet a certain supplement. Got some sample data from a real world device, implemented the standard in our system, and started playing with it, but every image I got back was just a garbled mess. After tons of attempts, I eventually got in contact with a developer at the manufacturer who told me they encrypt the image data so that it can only be used by their supporting software...
I think I know which manufacturer you are talking about, and this is super shady. They also go around in conferences talking about how "the data belongs to the patient" and how they "support standards".
Maybe now that the EU has started mandating data interoperability this could improve, but I have a feeling they will just find a way to tick that box without actually offering any diagnostically relevant data.
Maybe now that the EU has started mandating data interoperability this could improve, but I have a feeling they will just find a way to tick that box without actually offering any diagnostically relevant data.
All manufacturers do this indeed. The “good” data is in the private fields that can only be read by the manufacturer’s proprietary software. The fields that are open for everyone usually have a lower quality version of the actual data. The latter is usually good enough for a small preview, but not good enough for proper diagnostic usage.
What sort of machines were you using?
My experience with research MRI/CT scanners is that the actual image data is usually where it's supposed to be.
My experience with research MRI/CT scanners is that the actual image data is usually where it's supposed to be.
This was an ultrasound machine. Philips would first have you import the data in their own QLab software and from there export a copy (which misses some of the data from the private fields). With GE reading the data was completely impossible.
Ugh. That may be one of the things on my post-pandemic todo list. Well, thanks for the heads up!
Is that even legal?
I don't think it is, at least in the US. What most people forget about HIPAA is that the P stands for "Portability". The entire point behind it was that someone should be able to take their records from one hospital, doctor's office, or imaging clinic and then bring it to another one without having to go through another scan. DICOM was the industry's response to those portability requirements that our government put into place.
Are you sure about that timeline?
DICOM 3 (which was the first version using the DICOM name) dates back to 1993. HIPAA was signed in 1996.
I thought the “portability” part was meant to cover changes in jobs and life circumstances.
DICOM 3 (which was the first version using the DICOM name) dates back to 1993. HIPAA was signed in 1996.
I thought the “portability” part was meant to cover changes in jobs and life circumstances.
I suppose it could be false advertising to say you support a standard but only in a way that's meaningful to your software? As a 3rd party developer, it's hard to demand another company build something into their API, based on standards or otherwise. The best you can do is throw around your market share and lean on your mutual unhappy customers to make it happen.
This is one of those times I feel like the cell networks (or other network) should broadcast an accurate time without the need for an account/SIM and a cheap chip to receive it. I would say there's a lot of medical facilities that don't want to connect their equipment to the internet for time information and for good reason.
This already exists: https://en.wikipedia.org/wiki/Time_signal#Radio_time_sources
Edit: a page more specific to this use case is probably https://en.wikipedia.org/wiki/Radio_clock
Edit: a page more specific to this use case is probably https://en.wikipedia.org/wiki/Radio_clock
It's called GPS :-)
You can use a GPS as a time source for an NTP server (on your airwalled network).
You can use a GPS as a time source for an NTP server (on your airwalled network).
Unfortunately the GPS signal is generally too weak to be used indoors - especially in a larger building like a hospital.
Do you need to be more than 1km underground, or away from outside walls? Just put the GPS-receiver on the far end of an RS485 serial connection.
First google hit: https://www.worldtimesolutions.com/products/gps_rs485_time_s...
First google hit: https://www.worldtimesolutions.com/products/gps_rs485_time_s...
That'd depend on the hospital building out an RS485 infrastructure to support the device, though, which is a nontrivial ask.
Piggybacking off of a cell network is honestly probably the right solution. It doesn't have quite as high resolution as GPS, but that's hardly necessary for the sort of timestamps we're discussing -- even a resolution of a minute or two would be a huge improvement on "hope the tech who installed the equipment set the time correctly".
Piggybacking off of a cell network is honestly probably the right solution. It doesn't have quite as high resolution as GPS, but that's hardly necessary for the sort of timestamps we're discussing -- even a resolution of a minute or two would be a huge improvement on "hope the tech who installed the equipment set the time correctly".
Remember that anything that is harder than setting the date and time is unlikely to be done in practice, since we're looking for something easier than that. Routing cabling out of a room seems unlikely.
You put your ntp stratum one where it can receive the signal
Why not simply rely on GPS time? Seems like a stable and universal time source that could be used anywhere.
Dicom gets a lot of flak, but as you say, the thing that makes it bad is shitty implementations and shitty vendors. After a while, you realize there aren't any nonshitty vendors in the medical field.
And then you count your blessings that at least dicom gives you the beginning of interop.
And then you count your blessings that at least dicom gives you the beginning of interop.
My software gig largely involved converting DICOM files into the format used by the company. I very very very much can relate with all this. Basically almost every client needed a custom script to correct the issues in their use of DICOM
Pretty much all of the medical instrument software I've ever seen is awful. I'd always imagined that that was because it was written by EEs. No less scary for that, though.
I think some of the frustration towards DICOM is misattributed. Yes, there are many instances of the standard being implemented poorly, and yes it is not always the most convenient having to index into your metadata using hex digits. But at least there IS a well documented, universally adopted standard that captures the most important information relevant to medical images right there next to the pixels. If this weren't the case, I think it would be much harder to have reliable interoperability between the highly complex + diverse types of systems that need to talk to each other in order to conduct an imaging workflow. There is a reason it exists and is so widely adopted -- and if you dig in to the details, a lot of thought has been put into its design. I highly recommend the book https://www.amazon.com/Digital-Imaging-Communications-Medici..., which believe it or not is a very entertaining read on the subject.
I used to work with DICOM about 7 years ago. It was a complete shitshow. Most of the important data was in "private fields". Imagine all websites using CSS, but they all use different vendor-specific attributes like "-moz-border-radius: 5". Might as well use a proprietary format at that point.
In the pre-DICOM days, I made money by making video capture cards. Back then, the only standard to get images from CT and MRI machines was the composite video on the 75 ohm coax going to the operator's console screen. Laser cameras (medical image film printers) worked the same way.
Actually that's not entirely true: there was also a 3M / Kodak defacto standard using a parallel RS-485 bus.
Actually that's not entirely true: there was also a 3M / Kodak defacto standard using a parallel RS-485 bus.
I'll just leave this right here: https://www.ihe.net/
I worked on DICOM images sent to Blackberry apps in 2006 or so. It was a pain then. Assume not much has changed.
DICOM hasn't changed a lot, but thankfully there are lots of tools available now, and vendors have stopped using proprietary formats.
megiddo(3)
Seeing those diagrams brought back swarms of memories from 20+ years ago working on UML diagrams on HL7 committees. Ewww.
FWIW, Innolitics has a better data dictionary (information object definitions) for DICOM than the official standards website. https://dicom.innolitics.com/ciods
DICOM has its issues, but it’s much better than HL7.
DICOM has its issues, but it’s much better than HL7.
Hey, I'm actually one of the engineers at Innolitics working on improving the DICOM Browser, and we just finished a relatively large update to it.
Thanks for linking the site! If you have any suggestions or encountered any bugs on the site, please feel free to create an issue in the repo: https://github.com/innolitics/dicom-standard/issues
Thanks for linking the site! If you have any suggestions or encountered any bugs on the site, please feel free to create an issue in the repo: https://github.com/innolitics/dicom-standard/issues
Nice work! It is the primary DICOM resource for me and my team. Good to see that it's being actively maintained!
@korijn I'm glad you find our DICOM Standard Browser to be useful!
You (and others reading this post) may be interested to know that the JSON data the powers the browser is extracted using a set of open-sourced Python scripts:
https://github.com/innolitics/dicom-standard
You (and others reading this post) may be interested to know that the JSON data the powers the browser is extracted using a set of open-sourced Python scripts:
https://github.com/innolitics/dicom-standard
Reminder to not put these things on the Internet (~2,600 results):
https://beta.shodan.io/search?query=dicom+server+response
We're continuing to see hospitals and other healthcare-related organizations expose services that fall through the cracks due to the relative obscurity of these types of protocols in IT.
https://beta.shodan.io/search?query=dicom+server+response
We're continuing to see hospitals and other healthcare-related organizations expose services that fall through the cracks due to the relative obscurity of these types of protocols in IT.
Hospitals tend to have piss-poor security. Their IT budget is usually low, they have one or two sysadmins who are eternally busy fixing things that users have messed up. Pentests are slowly becoming more accepted as a requirement, mostly because the GDPR has come into effect. And those inevitably turn up large lists of criticals.
Typically, gear is left with all the settings at 'default', things are opened up to the net for temporary access which is then never removed again, it's possible for the whole hospital to have only one LAN segment and so on. Ransomware, directed attacks to gain admin privileges and substantial violations of patient confidentiality have all been recorded with great regularity.
I'm assuming that if I have to go into the hospital again for whatever reason that that data will be on the street before I leave the building.
Typically, gear is left with all the settings at 'default', things are opened up to the net for temporary access which is then never removed again, it's possible for the whole hospital to have only one LAN segment and so on. Ransomware, directed attacks to gain admin privileges and substantial violations of patient confidentiality have all been recorded with great regularity.
I'm assuming that if I have to go into the hospital again for whatever reason that that data will be on the street before I leave the building.
As standards come DICOM is actually pretty good. Sure, manufacturers will do their damnest to mess it up but interop tends to be surprisingly good because - just like with HTML - people tend to write for the lowest common denominator on reception. A typical PACS system will talk to a large number of machines all producing different output for different specialists to look at.
I've looked at about 10 companies using DICOM over the last couple of years and interop was typically the least of their worries. Think about the level of complexity here: anything from single frame BW at high res (say an X-ray) to time series of volumetric scans (for instance CAT).
The smarter companies will use one of the better FOSS libraries to implement the standard, the not-so-smart ones will try to roll their own, burn a ton of resources and will end up playing catch-up.
I've looked at about 10 companies using DICOM over the last couple of years and interop was typically the least of their worries. Think about the level of complexity here: anything from single frame BW at high res (say an X-ray) to time series of volumetric scans (for instance CAT).
The smarter companies will use one of the better FOSS libraries to implement the standard, the not-so-smart ones will try to roll their own, burn a ton of resources and will end up playing catch-up.
Be glad for DICOM. In digital pathology we don't even have that. Every manufacturer has their own format. We don't even have proper OS libraries, there is a LGPL library that hasn't been updated for 2 years and GPL one that's up to date but slow as molasses.
EDIT: yes, I meant OpenSlide and BioFormats. BioFormats is in unoptimized Java. Oh yeah, and DICOM for pathology exists and some software tools use it exclusively but none of the manufacturers do. As the only standard some organization press for standardizing on it - forcing all vendors to provide convertors. But those are often slow, buggy and produce huge files.
EDIT: yes, I meant OpenSlide and BioFormats. BioFormats is in unoptimized Java. Oh yeah, and DICOM for pathology exists and some software tools use it exclusively but none of the manufacturers do. As the only standard some organization press for standardizing on it - forcing all vendors to provide convertors. But those are often slow, buggy and produce huge files.
I am curious what digital pathology libraries you are referring to (by any chance, is one of them https://www.openmicroscopy.org/bio-formats/ ?)
Also, I am sure you are aware the DICOM folks have been trying to get digital pathology pulled into the standard. I am not sure how much success they have had, but David Clunie (the current editor of the standard and a very friendly and open-minded person) had a funny set of power point slides about this topic. I recommend taking a look just for the humor:
https://www.dclunie.com/papers/HIMA_2017_DICOMWSI_Clunie.pdf
Also, I am sure you are aware the DICOM folks have been trying to get digital pathology pulled into the standard. I am not sure how much success they have had, but David Clunie (the current editor of the standard and a very friendly and open-minded person) had a funny set of power point slides about this topic. I recommend taking a look just for the humor:
https://www.dclunie.com/papers/HIMA_2017_DICOMWSI_Clunie.pdf
What do you mean by OS libraries here? Open source? Have you profiled the library to see where it is slow?
I work with DICOM and HL7 and these things bring me to daydream about becoming a farmer and never looking back... Public healthcare IT is not where you want to be.
It's like TIFF, or like USB Storage.
Sometimes when it doesn't really matter what is chosen, one powerful entity can just pick and it doesn't matter. Even if maybe option B is very slightly better, clarity that we're all doing A is arguably better than some do A and some B and all suffer.
Unicode / ISO 10646-1 is an example like that. Some people don't like Han Unification, some people think UTF-16 reserved codepoints is an abomination, but mostly everybody is happy to live in a world where Unicode and ISO-10646 are effectively the same thing seen from different angles, not competing standards (Yes that could have happened)
Other times though there is no single entity powerful enough and willing to insist on one way forward, and no natural agreement on whether to do A or B. Instead parties that want to do A, and parties that want to do B just agree either is fine. The result is "standards" like USB Storage, TIFF or DICOM that leave far too much as unresolved choices.
Example: Some scanner manufacturers agreeing TIFF had designed scanners that start from the top-left of the eventual document, others started top-right or even bottom-right. If TIFF said "Images are top-to-bottom" the bottom-right scanners become expensive because they need a huge RAM buffer. Vice versa if TIFF says "Images are bottom-to-top". So... the actual TIFF standard says either can be true, you just flip a tag in the header to declare which way up the image is. Decoders have to carry all the burden of dealing with this, or be incompatible with some images to the confusion of users. ("Why is this picture upside down?")
These standards are marginally better than nobody agreeing anything, but only marginally. They're like that mediocre fusion restaurant you end up at when half the group wants Chinese and half wants Indian. Nobody likes this (hypothetical) fusion restaurant, so now nobody is happy, is that really better? I guess it's an improvement on spending the evening arguing pointlessly, because at least you can eat now.
Sometimes when it doesn't really matter what is chosen, one powerful entity can just pick and it doesn't matter. Even if maybe option B is very slightly better, clarity that we're all doing A is arguably better than some do A and some B and all suffer.
Unicode / ISO 10646-1 is an example like that. Some people don't like Han Unification, some people think UTF-16 reserved codepoints is an abomination, but mostly everybody is happy to live in a world where Unicode and ISO-10646 are effectively the same thing seen from different angles, not competing standards (Yes that could have happened)
Other times though there is no single entity powerful enough and willing to insist on one way forward, and no natural agreement on whether to do A or B. Instead parties that want to do A, and parties that want to do B just agree either is fine. The result is "standards" like USB Storage, TIFF or DICOM that leave far too much as unresolved choices.
Example: Some scanner manufacturers agreeing TIFF had designed scanners that start from the top-left of the eventual document, others started top-right or even bottom-right. If TIFF said "Images are top-to-bottom" the bottom-right scanners become expensive because they need a huge RAM buffer. Vice versa if TIFF says "Images are bottom-to-top". So... the actual TIFF standard says either can be true, you just flip a tag in the header to declare which way up the image is. Decoders have to carry all the burden of dealing with this, or be incompatible with some images to the confusion of users. ("Why is this picture upside down?")
These standards are marginally better than nobody agreeing anything, but only marginally. They're like that mediocre fusion restaurant you end up at when half the group wants Chinese and half wants Indian. Nobody likes this (hypothetical) fusion restaurant, so now nobody is happy, is that really better? I guess it's an improvement on spending the evening arguing pointlessly, because at least you can eat now.
The main issue I've found is the manufacturer's implementation. They range from okay to barely meeting the minimum standards.
One thing that always annoyed me was the time stamps. They're mandated by the standard, so there's always a timestamp, but it's often useless. For example, it seems some scanners don't get their time set, over the network or otherwise, _ever_, so you might have a bunch of scans performed a few years ago on scanner A with an accurate timestamp of say 2017, but in a new scan from last week that was taken by scanner B, which hasn't had its clock configured properly, it reports a date closer to the start of the unix epoch rather than 2020.
Of course some of this is out of the manufacturers control, but temporal data is critical to analyzing changes to patient over time. I think this is one area where the DICOM standard couldn't be a bit more strict with a lot of positive impact.