Algorithm Removes Water from Underwater Images(scientificamerican.com)
scientificamerican.com
Algorithm Removes Water from Underwater Images
https://www.scientificamerican.com/video/seeing-through-the-sea/
110 comments
"Sea-thru" is such a great name for this. Also I'm always glad to see things that aren't "we threw a neural net at it and it looks pretty good"
The company name ("SeaErra") is pretty amusing too :)
Soundtrack: https://youtu.be/NWB02uS2Gow
that's fascinating, and a lot more involved than what I expected (my guess was something more like off-the-shelf color correction)
The palette cards they carry around remind me of white-balancing.
If I understand things correctly it's not entirely different to white balancing either, just that the difference between white balancing a monitor or camera is that you don't really have to correct for the scattering of light through air because it's negligible at the distances you're likely to work with. The trick here – again, if I understand correctly – is that this method also finds the light scattering/absorption characteristics and corrects for those as well. It might seem obvious, but I think it's more involving than it seems, because light scatters differently in different environments and I think – again, I might be wrong, this is just my understanding – this method accounts for that by using multiple images at different distances to gather the parameters to find the scattering and absorption characteristics. This is what makes it physically accurate and not just "make these colors mostly look good" which is really quite cool and useful!
If I remember correctly, the percentage of light that passes through a column of water exponentially decays with distance, like stacking a bunch of fixed-percentage optical filters on top of each other. They’re using parallax shift between multiple images to generate a depth map, so they have a measure of the amount of water obscuring each pixel.
I’m not sure how they’re finding the optical properties of the water; there’s something about using the darkest pixels in the image for calibration that I didn’t quite follow.
I’m not sure how they’re finding the optical properties of the water; there’s something about using the darkest pixels in the image for calibration that I didn’t quite follow.
It's actually exactly that. Her set of skills is to do that actually accurately with setpoints for both distance to the objects as well for different wavelengths simulataneously.
On the first example corrected image reveals some details in shadows but also burns out some highlights.
So it seems like it's just MVS > depth map + recovery of haze characteristics by sampling a color reference at a bunch of distances. That seems blindingly obvious unless I missed something.
First of all, you're wrong. "It works without a color chart, or any information about the optical qualities or depth of the water body."
Secondly, if you think it's "blindingly obvious", read the paper linked in the parent's comment. Even a quick skim will show you're wrong.
In conclusion: yes, you missed something.
Secondly, if you think it's "blindingly obvious", read the paper linked in the parent's comment. Even a quick skim will show you're wrong.
In conclusion: yes, you missed something.
Don't tell me, you could have done it yourself in your basement in a couple of hours? Well done.
You missed something: the huge amounts of effort required to make it work and produce acceptable results
And that no one had done it before (apparently).
> Sea-thru is protected by a registered patent owned by Carmel Ltd the Economic Co. of the Haifa University and its subsidiary SeaErra
Looks like they are going to monetise this technology at some point given the disclaimer at the bottom of the page. This is not wrong. But it feels like a PR exercise dressed up as something academic which is a little creepy.
Looks like they are going to monetise this technology at some point given the disclaimer at the bottom of the page. This is not wrong. But it feels like a PR exercise dressed up as something academic which is a little creepy.
It is not unusual in academia to patent all innovations with potential commercial applications. At least in Canada, universities typically have innovation centres whose main job is encouraging and helping professors, graduate students, and other researchers patent their innovations and commercialize them (e.g. by licensing the patent). It is not sinister, it is normal procedure in academia.
And the funding for the research that allowed for this invention was paid for by? The inventor? Or the government?
[deleted]
Really?
Some professors have “hard money” jobs where the university covers most/all of their salary; startup packages that are meant to help you get a grant are pretty common, as are fellowships or TAships for students.
However, I don’t think most universities cover much of the actual research expenses.
As for the patent, most places offer a split with the inventor, and may not patent everything; they have a right of first refusal though.
Some professors have “hard money” jobs where the university covers most/all of their salary; startup packages that are meant to help you get a grant are pretty common, as are fellowships or TAships for students.
However, I don’t think most universities cover much of the actual research expenses.
As for the patent, most places offer a split with the inventor, and may not patent everything; they have a right of first refusal though.
If the university is publicly funded, then it is the public who actually paid for the research.
Yes, this is ridiculous especially considering that a patent gives the holder a monopoly, stifling further innovation, which is exactly the opposite of the purpose of academia.
Typically, if it is in collaboration with private sector (i.e. a company is providing the funding), said company will retain the intellectual property produced by the research. If it is government funds, university (itself a government organization) will have partial ownership of the intellectual property. So the government pays for research, the researcher does research. The researcher and the government (via university) own the fruits of the research together.
It is also notable that in Canada (almost?) all large research universities are public institutions: University of Toronto, McGill, Queen's, University of British Columbia, University of Alberta, etc. I imagine the calculus would have been different if the university was itself a private corporation.
It is also notable that in Canada (almost?) all large research universities are public institutions: University of Toronto, McGill, Queen's, University of British Columbia, University of Alberta, etc. I imagine the calculus would have been different if the university was itself a private corporation.
All universities seems to be patenting the hell out of their research these days.
> This is not wrong.
It is if the research was paid for by the public.
It is if the research was paid for by the public.
If a Canadian university invents a new image filter that will make fifty million dollars for companies whose products include image filters, but all such companies are foreign (instagram, apple, adobe, facebook...) why should the Canadian taxpayer give it to them for free?
It's not like Instagram are going to pass the cost of licensing on to users.
It's not like Instagram are going to pass the cost of licensing on to users.
If you have a mechanism for making the patent freely available to Canadians only that's fine.
But if the taxpayers who funded the research have to license the technology to use it, that's just robbery.
But if the taxpayers who funded the research have to license the technology to use it, that's just robbery.
I'm not sure taxpayers are covering all the costs. When I was a kid I used to golf at a golf course that was owned by the county. You still had to pay to play there. People that don't play golf pay very little for the course to exist, but they are still paying something.
Maybe this is "robbery" (which is not what that word means), but you still get the benefits of living near open space (no car-owners living there, trees turning CO2 into oxygen, bees pollinating your flowers), etc. All in all, it doesn't seem too bad. (It is $26 a round, btw.)
Research universities fill a similar role. They are good to have around, even if you can't monetize their research for free. There are other benefits.
Maybe this is "robbery" (which is not what that word means), but you still get the benefits of living near open space (no car-owners living there, trees turning CO2 into oxygen, bees pollinating your flowers), etc. All in all, it doesn't seem too bad. (It is $26 a round, btw.)
Research universities fill a similar role. They are good to have around, even if you can't monetize their research for free. There are other benefits.
Obvious difference between the golf course and research results, which justifies different treatment, is that the golf course is rival good.
> Research universities fill a similar role. They are good to have around, even if you can't monetize their research for free. There are other benefits.
I did not claim that the downside of granting patents to universities is so large that universities should not exist at all. Only that it would be better and more just to not grant them patents based on publicly funded research.
> Research universities fill a similar role. They are good to have around, even if you can't monetize their research for free. There are other benefits.
I did not claim that the downside of granting patents to universities is so large that universities should not exist at all. Only that it would be better and more just to not grant them patents based on publicly funded research.
I agree with your general argument, but golf courses are such an unfortunate example. They're the one use of public space that would be many orders of magnitude better for the community if they were used completely differently. A park would be a good start, but there are so many more.
When you have a colour chart for reference and known distances between the camera and the chart, it's not difficult to colour correct. They might have a better interpretation of the underwater imaging phenomenon, which anyone can reverse engineer and use (from the published work). Patent's are just for reputation.
> When you have a colour chart for reference and known distances between the camera and the chart, it's not difficult to colour correct.
I think the whole point of that project is that yo udon't need a chart.
I think the whole point of that project is that yo udon't need a chart.
I hate to say it, but that was actually a really poorly edited and produced video. It spent way too long on b-roll and did a really poor job framing the problem.
I would have strongly preferred static images in the article and an interview video buried below.
I would have strongly preferred static images in the article and an interview video buried below.
The video let them show a 30 second ad which monetizes much better than static ads.
I agree they should have gotten to the punch line and show results rather than the doctor swimming.
I agree they should have gotten to the punch line and show results rather than the doctor swimming.
Not a single image in the article. An article about images. What a shame.
Scuba divers already use post-processing in Lightroom or apps like Dive+ (http://dive.plus/). It will be interesting to see if this becomes popular in that community. The results are pretty good already with Dive+.
Looks a lot stronger than the Dive+ images I've seen (and created)
This would be a great application for deep learning. Use the authors' method to generate a lot of uncorrected-corrected pairs. Or, use a graphics engine to render realistic underwater scenes with and without water color. Then use a convolutional neural network to learn to mimic the transformation. Then any photographer can apply the learned filter without a color card or depth information.
Edit: already been done: https://arxiv.org/abs/1702.07392
Edit: already been done: https://arxiv.org/abs/1702.07392
You should take a look at the paper itself which is linked above. It does not require a color card in all cases, and the only information required is the depth, which can be obtained from a series of photos instead if required.
What you want is a few lasers of known wavelengths (RGB?) pointing at known angles such that, from the camera's perspective, they appear as lines (ie they aren't perpendicular to the plane of the photo sensor).
A calibration image(s) can be made before each shot. Possibly the resulting image correction can be integrated in to the camera too.
The laser wavelengths are a substitute for the color chart. The laser angle means you get a reading at each distance (ie in the image, each point on the laser "line", corresponds to a distance.)
A calibration image(s) can be made before each shot. Possibly the resulting image correction can be integrated in to the camera too.
The laser wavelengths are a substitute for the color chart. The laser angle means you get a reading at each distance (ie in the image, each point on the laser "line", corresponds to a distance.)
Her algorithm doesn't require calibration. The color chart in many of the photos is to demonstrate the effect, or was just habit when she took them.
True, but the method would totally benefit from calibration, because not all water is made the same - some has different densities of particulates in.
Also I imagine that depth would be a significant factor
That seems like a bad idea, because lasers are monochromatic.
Sure, a red laser looks red. But just because you only see 50% of the brightness of the light of a red laser doesn't mean the absorption of all red colors will be 50%. Seems a lot better to use some very mixed source of light.
Sure, a red laser looks red. But just because you only see 50% of the brightness of the light of a red laser doesn't mean the absorption of all red colors will be 50%. Seems a lot better to use some very mixed source of light.
laser beams are only visible if they reflect off something, so in a lot of situations the camera just won't see them except as points where they hit an object.
[deleted]
How is this different from the colour chart idea? If we know how some actual KNOWN RGB pixels look in a particular setup, we can apply the same filter across the image. Right?
The amount of color shift depends on how much water is between the object and the camera, so you need to have a depth map to recover the true colors. You can see how it compares to naive color transforms in the paper.
> so you need to have a depth map to recover the true colors
Isn't the blue shift of a known color already a measure of the amount of water between the object and the camera- and therefore its distance? Knowing the true color of a fish, a seaweed or the sand isn't already enough to infer distances and color-correct?
Isn't the blue shift of a known color already a measure of the amount of water between the object and the camera- and therefore its distance? Knowing the true color of a fish, a seaweed or the sand isn't already enough to infer distances and color-correct?
Yes, but how do you know the true color of a fish, seaweed, or sand? That's the unknown part!
> how do you know the true color of a fish, seaweed, or sand
I am pretty sure that in most cases those colours are well known.
I am pretty sure that in most cases those colours are well known.
I, personally, have picked up sand, shells, even seaweed from the ocean, and lifted it out of the water and looked at it with my eyes. I have released this technique to the world, patent free.
I've only skimmed the paper, but it's more complicated than that. There's both an light attenuation effect (the "blue shift" you're talking about), as well as a separate back-scatter effect, which use different coefficients based on depth. It seems to me that the hard part is figuring out a function to color correct based on both these factors for a large range of depths using a limited number of reference points (as well as getting an accurate depth map, of course).
Not everything in the image is at the same distance from the camera as the color chart. You still need depth information for the other parts.
I'm sorry, but I don't get it. I mean, if what you want is an accurate color reconstruction, for scientific purpose, then yes, you probably need depth information. If what you want is simply to colorise an image inferring the correct colors from a mix of residual color, blue shift, and a general knowledge about the object itself (the color of a seaweed, a starfish, etc.) then you can probably do it pretty well from a single image. There are neural networks that can colorise a b&w picture with zero color data available(1), and in this case you just want to enhance existing colors, so it should be much easier. Of course you'll get a representation with no claim of accuracy, but it should be fine for many purposes.
(1) often with pretty bad results, but consider the variety of scenes and object above water compared to the average underwater scenario.
(1) often with pretty bad results, but consider the variety of scenes and object above water compared to the average underwater scenario.
The difference is that in this case the same colour will look different depending on how far away it is.
With regular colour correction things have a slightly different colour regardless of how far away they are from the camera, so the task is way simpler.
With regular colour correction things have a slightly different colour regardless of how far away they are from the camera, so the task is way simpler.
To add to that, as I understand it Sea-Thru calculates what colour should be where based on depth information, even if the original colour information is 'lost'. This makes it more than a simple filter you apply.
What if I now get a bunch of images before & after and train a neural network on this to "learn" the mapping. Who would the neural network belong to?
This looks really cool. She described how she takes photos that include her color chart, and I'm wondering if that's actually necessary to calibrate the process, or if that was just done for the purposes of developing it.
The researcher that authored the paper answered a few questions on Reddit last year [0]. She explained that the color chart isn’t necessary for every photo.
[0] https://www.reddit.com/r/videos/comments/dvts2j/this_researc...
[0] https://www.reddit.com/r/videos/comments/dvts2j/this_researc...
The change in the shades of the color chart (due to lighting) is probably used as a tolerance in the algorithm.
It's necessary for calibration at shooting time. It's a very similar technique to using, say, a ColorChecker Passport or a SpiderCHEKR with their associated profile-creation software for still photography, the main difference being that attenuation underwater is much more variable that lighting conditions on dry land. You need to accommodate attenuation of both the incident light (light hitting the subject has to get there through local water conditions) and light reflected from the subject at different depths at different times of the day and year, and with different "water" compositions (minerals, particulates, etc.).
Coincidentally: the researcher's name, Derya, translates to "sea" or "ocean" in her native Turkish. I wonder if that's something that might have inspired her as part of this.
I read this and wonder if the technique can be applied to space. Too bad we can’t take photos of a closing distance of something 50 million light years away.
I wonder if the algorithm would become better if not only did author did the swim closer but also took pictures of different path distances and angles simultaneously and map images together. Maybe it will reveal some of the editing work on hazy objects took a little more liberty and will produce more accurate images.
Great read!
I wonder if the algorithm would become better if not only did author did the swim closer but also took pictures of different path distances and angles simultaneously and map images together. Maybe it will reveal some of the editing work on hazy objects took a little more liberty and will produce more accurate images.
Great read!
Perhaps a real-time version of this can be embedded into the diving goggles(AR) for murk-free dives.
Currently it’s only photoS and requires a carefully placed colour chart to sync up the colours.
I don’t think that last bit is automatable with just glasses.
I don’t think that last bit is automatable with just glasses.
The project website shows a video example and says that a color chart is not required.
https://www.deryaakkaynak.com/sea-thru
https://www.deryaakkaynak.com/sea-thru
I don’t get how it’s not a photoshop though, it’s just a really specific photoshop. See the stuff how it would be on land... but it wouldn’t be like that on land at all. This is no more real or fake than any other filter applied to pixels.
It's "not a photoshop" in that it's not someone manipulating the image until it looks good. It's an algorithm that (according to the video) uses precise physical modeling to achieve the correct color, regardless of aesthetic preferences.
does the method construct some model of the physical environment or does it just modify the color of each pixel based on the color chart (color calibration/correction)?
> Sea-thru is a physics-based algorithm using the Akkaynak-Treibitz underwater image formation model (CVPR 2018). It does not use neural networks and was not trained on any dataset. It works without a color chart, or any information about the optical qualities or depth of the water body.
https://www.deryaakkaynak.com/sea-thru
https://www.deryaakkaynak.com/sea-thru
Why would it matter if it is a "photoshop" or not?
We already use filters all the time on normal camera photos (e.g. Low Light ML on pixels). As long as it's correcting the colour for us to be able to assess it better, and it's accuracy is reasonably high, than it is gravy.
We already use filters all the time on normal camera photos (e.g. Low Light ML on pixels). As long as it's correcting the colour for us to be able to assess it better, and it's accuracy is reasonably high, than it is gravy.
That’s a question for the maker of the video who thought it was worth differentiating.
The moment you take a picture, your camera or phone already does tons of editing. Like really a lot! I assume, you wouldn’t consider that “photoshopping”. Where do you draw the line?
For me, haze removal, water removal are like white balance. To me it’s not manipulation.
For me, haze removal, water removal are like white balance. To me it’s not manipulation.
This is a quote from the video, the host took time to explain how it was not the same as “photoshopping” when it really is, not much different from your phone processing the image as you’ve mentioned.
Did you watch the video where she demonstrated how the technology works and how she says it differs from photoshopping? She pretty clearly explains the difference.
It’s a computer changing colored pixels, it’s marketing fluff to try and say it’s not the same. Especially the concept that these things will really look that way on dry land, I can assure you they would not.
Again, did you watch the video? The technology is different. Yes, it involves a computer changing colored pixels but that's only one part of it. The calibration process is something you cannot do in Photoshop. Photoshop does not place physical color calibration charts in the sea and calculate the color profiles for you based on that chart.
It would be like that without water, that's the whole point of the algorithm: based on physics, it computes the (best known approximation to) way the scene would look with no water being present.
That is, it's supposed to make the photo be like the one you would take if you were to lift a part of the seafloor onto a boat and photograph it.
That is, it's supposed to make the photo be like the one you would take if you were to lift a part of the seafloor onto a boat and photograph it.
Not really: it doesn’t account for specular reflections. It’s more like if you took the objects out of the water and then used a polarising filter to remove the highlights.
You think that stuff would really look that way without water though? I expect many things would lose color quite rapidly.
Its just semantics. Its a filter that finds color information closer to the reality and ignores aesthetics.
I agree with your view. An algorithm is manipulating the image and similar if a person used photoshop for manipulating an image. The process being more automated than a person doesn’t matter because both situations are altering the pixels for an altered outcome.
> The process being more automated than a person doesn’t matter because
It does matter because it's infinitely quicker to do it that way. It's like saying a modern calculator and an abacus are the same because they let you add/sub/&c numbers
It does matter because it's infinitely quicker to do it that way. It's like saying a modern calculator and an abacus are the same because they let you add/sub/&c numbers
I’m not saying I agree with the GPs point but I do think you’re also missing their point. To use your example, it would matter if the question was asking people to do mental arithmetic.
The issue isn’t whether a machine or human is making the edits though. It’s really a question of what is the actual accurate image. Unfortunately that’s probably more of a philosophical rather than technical one. For example differing camera lenses will add their own distortions of “truth”. We also have different perceptions of colour from person to person. So how do you define “accurate”?
In my personal opinion what really matters is whether the subject is fair. By that I mean: are the filters changing the context or altering the perceived facts of the image? Or are the filters just an artistic choice. This filter definitely falls into the latter since the subject is still very obviously under water albeit presented with greater clarity. You’re not changing the context of the image but rather the presentation. So it’s hard to argue that as being “photoshopped” (in the context that term is typically used).
The issue isn’t whether a machine or human is making the edits though. It’s really a question of what is the actual accurate image. Unfortunately that’s probably more of a philosophical rather than technical one. For example differing camera lenses will add their own distortions of “truth”. We also have different perceptions of colour from person to person. So how do you define “accurate”?
In my personal opinion what really matters is whether the subject is fair. By that I mean: are the filters changing the context or altering the perceived facts of the image? Or are the filters just an artistic choice. This filter definitely falls into the latter since the subject is still very obviously under water albeit presented with greater clarity. You’re not changing the context of the image but rather the presentation. So it’s hard to argue that as being “photoshopped” (in the context that term is typically used).
Exactly. The reason behind the configuration of the pixel changing filter is the only thing that is different. But it doesn’t stop there, the concept that an aesthetic choice is more or less real than a “just like it would be in dry land” choice is also flawed because those creatures would most definitely not look that way on dry land.
NASA and JPL do similar things with Mars surface images to bring out details and color differences. The orange dusty sky normally washes everything in an orange tint and softens shading.
Article that is not really an article but video about images. Why? :(
Is anyone aware of something similar that works when the picture is taken outside the water? Something that could remove reflection and glare.
> It does not use neural networks and was not trained on any dataset.
Thats pretty cool. We could do with a GoPro version of this lol.
I was watching a blooper real for an older movie the other day and realized that a very similar color rectangle appeared in some of the shots.
Made me wonder how unique this technique is or if they’re just using bog standard movie editing tricks.
Made me wonder how unique this technique is or if they’re just using bog standard movie editing tricks.
Using color-correction charts is a very standard photographic technology for correcting colors in any situation, where there is challenging lighting. So just using one for your underwater photography would give you vastly improved colors in your picture. As I understand their work, it uses a correction model which is distance dependant, so corrects the color shifting with distance. In air, changing the distance doesn't change the color appeareance, but under water, the absoption and scattering effects vary with distance.
Color correction is standard in publishing and happened long before movie editing.
nurettin(1)
Not trying to be a jerk... but at 2:43 in the video the show video of fish swimming in the reef and those fish are beautiful, their colors aren't affected by the problem she is describing... so my question is, was that footage of the fish run through this algorithm? And if not... then what is the actual point of this ground breaking technology when it can already be done without it?
paper [pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Akkayn...