Image color replacement with numerical optimization(andersource.dev)

78 pointsby andersource5 tahun yang lalu17 comments

Image color replacement with numerical optimization

https://andersource.dev/2021/06/12/image-color-replacement.html

17 comments

maurits5 tahun yang lalu

Pretty clever. Somewhat related, if you want to match the colours of an entire image to an other, you can use optimal transport [1]

Unrelated, the author has a very nice take on generating the organic grid from Oskar Stålberg’s Townscaper game [2].

[1]: https://codimd.math.cnrs.fr/s/2eRBqV9zl

[2]: https://andersource.dev/2020/11/06/organic-grid.html

vanderZwan5 tahun yang lalu

Here is a JavaScript implementation by Fil:

https://observablehq.com/@fil/sliced-optimal-transport

Doxin5 tahun yang lalu

re: link #2

While that is a much simpler solution and quite a nice writeup, it doesn't quite solve the problem that townscaper poses. For that it needs to be infinitely expandable, which is quite a different feat.

lostinthefield5 tahun yang lalu

As someone who's not trained in math, what does "optimal transport" mean?

dandelany5 tahun yang lalu

Let’s say we have 12 iron mines with known amounts of iron ore supply, and 10 steel foundries with known amounts of demand. For each (mine, foundry) pair, there is a known cost of shipping the ore from point A to B. Which mines should send how much ore to which foundries?

AIUI, that is the general “optimal transport problem”. Now replace ore with colors and it applies to the problem above :)

andersource5 tahun yang lalu

Thanks! The optimal transport is really cool!

tyingq5 tahun yang lalu

A more pedestrian imagemagick script that uses a "fuzz value" that's a number of degrees on a hue wheel instead:

http://www.fmwconcepts.com/imagemagick/replacecolor/index.ph...

kirillcool5 tahun yang lalu

"Flowers replaced to red" looks like it also tinted the whole sky red. In general, all the image captions imply that the algorithm is able to identify objects (flowers, ice cream scoop, shirt) and only change that object's color.

andersource5 tahun yang lalu

> "Flowers replaced to red" looks like it also tinted the whole sky red.

That's true, to me at least it looks like in the original image the sky has a yellow undertone so I think it makes sense

bluetwo5 tahun yang lalu

Yes, what he is really doing is what's known in photography and video as "color grading". where you narrow or short the hues used in an image.

adampk5 tahun yang lalu

Sorry for the noob question but reading through this I can't make sense of why this is the desired transformation:

"Desired transformation: σ(σ(IT)N)"

What is I, T, and N?

andersource5 tahun yang lalu

Hey!

I is the flattened image (i.e., a matrix where each row represents the RGB values of a single pixel)

T and N are some matrices, basically of appropriate dimensions such that the transformation you quoted results in an image of the same shape as the original and also such that we can optimize certain objectives with respect to the variables that define the transformation.

The why this is the desired transformation is a tricky question, it's definitely not the only and quite probably not the best way to formulate this problem - just the first way that I tried and worked reasonably well. I tried to explain my intuition for this in the article after the definition.

adampk5 tahun yang lalu

Really appreciate the response thank you, great article!

So is "T" and "N" conventionally used in numerical optimization as the transform "variables" (like X and Y are in calculus)?

andersource5 tahun yang lalu

Gladly!

> So is "T" and "N" conventionally used in numerical optimization as the transform "variables" (like X and Y are in calculus)?

Not really, numerical optimization offers a general toolbox for various problems, which are usually defined based on domain-specific notation and concepts. So the concept of a transformation like the one described in the article is not general to numerical optimization, but specific to the optimization being done here. That being said, when talking about techniques, theorems and algorithms in numeric optimization there are conventions. So you usually move between the more theory-oriented notation to a domain-specific notation (in the article I mostly used the domain-specific, lazy notation I chose).

Bear in mind that my perspective is not very academic so I might feel less strongly about (and be less aware of) rigorous mathematical conventions.

pajko5 tahun yang lalu

Also there is https://www.cs.huji.ac.il/~yweiss/Colorization/

Had implemented a Khoros filter box based on this for my Computer Graphics class back then.

danuker5 tahun yang lalu

Very interesting! They look a bit surreal, as if inverted maybe.

andersource5 tahun yang lalu

Thanks! Yeah, I think one thing the simplistic optimization is missing is preserving the "texture" of the image. There are various numeric ways to represent textures so this is something that potentially could be added to the objective, it was out of scope for the project though. Might give it a try some day!