A smooth and sharp image interpolation you probably haven't heard of(wordsandbuttons.online)
wordsandbuttons.online
A smooth and sharp image interpolation you probably haven't heard of
https://wordsandbuttons.online/a_smooth_and_sharp_image_interpolation.html
53 comments
The abuse of notation is a useful one, as it also captures the sense that as the exponent approaches infinity, the interpolation approaches nearest neighbor interpolation. This seems similar to the “max norm”, described in the wiki link below, which also uses infinity to describe the relationship with other distance formulas.
https://en.wikipedia.org/wiki/Uniform_norm
https://en.wikipedia.org/wiki/Uniform_norm
Could be nice if there were a log-scale slider to interactively vary the exponent from 1 to ∞-ish and see how that affects the frame-cropped preview.
Bilinear isn't 1/x, it's 1-x.
Right, but the more complicated expression ends up simplifying to the bilinear expression.
Why aren't the results of different interpolation methods shown in the website?
Good point. Why not? I'll add them.
Don't forget Smoothstep:
https://en.wikipedia.org/wiki/Smoothstep
https://en.wikipedia.org/wiki/Smoothstep
Thank you very much.
probably because the visual differences are so slight that there is no benefit in showing them to someone who isn't deeply entrenched in the topic
I think that would be me. Or anyone who would be interested in viewing this website. Would anyone not interested in this topic not want to ever see the differences?
Agreed
I found the original "pixelated" image to look higher resolution / crisper / more detailed.
Really at any size I tried, I found the "upscale" to look blurry / low quality.
Am I an outlier here?
Really at any size I tried, I found the "upscale" to look blurry / low quality.
Am I an outlier here?
There's no universal best option, here. Even two different pictures of the same thing at slightly different angles might look best with different upscaling algorithms. There are also variants on the smoother one to favor certain kinds of transitions. Really, though, all of those options suck. AI-powered options in Photoshop (much more controllable and reliable than SD or the like) actually invent details, so if provenance or accuracy is not important, (as it would be in a historic artifact, legal document, piece of visual art, or unique artisinal product,) that's always a possibility, but most projects have too many images to make tweaking each one individually viable.
Except in extremely rare situations, I flat-out refuse to work with low-quality source images in my design jobs. They can either give me a proper source image, pay me to rebuild it in a vector program if it's a logo or similar, or go down to the local sign shop and get someone to "design" something for them using a template. Most people end up having a higher quality copy around and just didn't know the difference between that and the screenshot they took of it, or the thumbnail they downloaded. At 300DPI with 4 colors, those flaws are glaringly obvious compared to screen resolution and sRGB color representation.
Except in extremely rare situations, I flat-out refuse to work with low-quality source images in my design jobs. They can either give me a proper source image, pay me to rebuild it in a vector program if it's a logo or similar, or go down to the local sign shop and get someone to "design" something for them using a template. Most people end up having a higher quality copy around and just didn't know the difference between that and the screenshot they took of it, or the thumbnail they downloaded. At 300DPI with 4 colors, those flaws are glaringly obvious compared to screen resolution and sRGB color representation.
I think at this point diffusion model super-resolution out-beats everything else.
You are right. It looks more pixelated, but better quality at the same time somehow?
I am a bit confused how the formula is different from a simple lerp.
k(x) = 1/x
y_i * k(x - x_i) + y_i+1 * k(x_i+1 - x)
F = --------------------------------------- =
k(x - x_i) + k(x_i+1 - x)
y_i / (x - x_i) + y_i+1 / (x_i+1 - x)
= --------------------------------------- =
1 / (x - x_i) + 1 / (x_i+1 - x)
y_i * (x_i+1 - x) / A + y_i+1 * (x - x_i) / A
= ---------------------------------------------
(x_i+1 - x) / A + (x - x_i) / A
where A = (x - x_i) * (x_i+1 - x)
scaling to x_i+1 - x_i = 1, and using a = (x - x_i)
F = (1 - a) * y_i + a * y_i+1
Oh, it seems that a square fell off. It should have been 1/x^2. 1/x works fine for bitmaps, but in 1-dimensional case 1/x indeed turns the whole thing into linear interpolation, you're right.
Yep, there is a square in the source:
// interpolation/extrapolation with no basis functions
var weight2d1 = 2;
const wf2d1 = function(x) {return 1./Math.pow(x, weight2d1);}My go-to for any kind of very fast image upscaling is the SuperXBR algorithm. It's fast, a pixel shader implementation runs at over 60FPS on Skylake IGP.
SuperXBR is in the family of Edge-Directed interpolation algorithms, so diagonal or curved lines will properly maintain their shape when upscaled, and not produce "staircase" artifacts that you see with other classical interpolation. It does not improve the sharpness though, images will still look blurry after being upscaled, just like when you perform bilinear upscaling.
But if you are doing any kind of non-realtime image upscaling today, you'd want to use the modern AI algorithms, such as Waifu2x or ESRGAN.
SuperXBR is in the family of Edge-Directed interpolation algorithms, so diagonal or curved lines will properly maintain their shape when upscaled, and not produce "staircase" artifacts that you see with other classical interpolation. It does not improve the sharpness though, images will still look blurry after being upscaled, just like when you perform bilinear upscaling.
But if you are doing any kind of non-realtime image upscaling today, you'd want to use the modern AI algorithms, such as Waifu2x or ESRGAN.
I find the transformer based SwinIR model to be both relatively computationally efficient (when compared to ESRGAN) and provides excellent quality upscaling for a very general set of input images.
Ok. How can I apt-get install it?
PS: is it me or has it become more difficult to find cool stuff in your typical Linux distro?
PS: is it me or has it become more difficult to find cool stuff in your typical Linux distro?
It's not a standalone program, it's usually a set of pixel shaders that can be integrated into another program.
I once made an image viewer that would apply the effect, but it's not publicly released.
There is an extremely roundabout way to see the effect in action on a Linux system using a package you can apt-get. It involves installing RetroArch, starting the "Image Viewer" core, loading an image, loading the shader preset named "super-xbr-2p.glslp", changing the number of shaders from 3 to 2 to exclude the "custom Jinc resharper" step and selecting Apply, then scaling the window to be exactly 2x the size of the original image. You probably don't want to do this.
But if you really want a standalone program, I could build a Windows .NET program that will apply the shader to images you paste in from the clipboard.
I once made an image viewer that would apply the effect, but it's not publicly released.
There is an extremely roundabout way to see the effect in action on a Linux system using a package you can apt-get. It involves installing RetroArch, starting the "Image Viewer" core, loading an image, loading the shader preset named "super-xbr-2p.glslp", changing the number of shaders from 3 to 2 to exclude the "custom Jinc resharper" step and selecting Apply, then scaling the window to be exactly 2x the size of the original image. You probably don't want to do this.
But if you really want a standalone program, I could build a Windows .NET program that will apply the shader to images you paste in from the clipboard.
That is very kind of you. But my statement was meant to be more general. I just have no time to dive into all the inevitable configure and compile issues anymore and am hoping that my Linux distro will catch up some day.
[deleted]
Sounds like this: https://en.m.wikipedia.org/wiki/Inverse_distance_weighting
I note that the "example 1 on one dimension" diagram in the link you provided produces a very strange curve, typically not the one you'd draw intuitively by hand.
When I think of "smooth and sharp" my mind goes to, uh, I guess "sharp bilinear" is the most common term?
It's like nearest neighbor interpolation but with less artifacting when the zoom is not an integer. The middle of each source pixel becomes a solid color block, and then bilinear filtering is applied to the single-pixel-wide seams between each block.
It's equivalent to doing a nearest neighbor upscale to floor(scale), then finishing with bilinear.
It's like nearest neighbor interpolation but with less artifacting when the zoom is not an integer. The middle of each source pixel becomes a solid color block, and then bilinear filtering is applied to the single-pixel-wide seams between each block.
It's equivalent to doing a nearest neighbor upscale to floor(scale), then finishing with bilinear.
There was a shareware site popular with the photo enthusiasts twenty years ago that implemented numerous resizing algorithms and allowed you to compare them visually. Unfortunately I can not recall its name.
AI based interpolation will soon make these mathematical interpolation methods obsolete right?
In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow. You can't use them for stuff like scaling an <img> in a browser.
Incidentally, the way I always thought it would work is the huge and slow AI would be used to discover a tiny algorithm that just so happens to be an amazing image rescaler. Instead we just ship the whole AI.
Anybody know of any work on the former strategy?
Incidentally, the way I always thought it would work is the huge and slow AI would be used to discover a tiny algorithm that just so happens to be an amazing image rescaler. Instead we just ship the whole AI.
Anybody know of any work on the former strategy?
> In quality, yes, but they're only feasible for stuff like batch processing because they're so huge and slow.
Aren't games using these for real time upscaling many dozen times a second already?
Aren't games using these for real time upscaling many dozen times a second already?
AI-based image upscaling happened many years ago. Waifu2x first released back in 2015, and ESRGAN first released in 2018.
SwinIR hails from 2021: https://arxiv.org/abs/2108.10257
Yes, with Upscayl, we’re already doing that (https://upscayl.org).
It’s not as fast yet but it’s extremely good. Good enough that you could print a low resolution image after upscaling.
It’s not as fast yet but it’s extremely good. Good enough that you could print a low resolution image after upscaling.
Thank you for everything you do!!
For practical reasons mostly yes.
For scientifically correct scaling (think forensics) you will of course need to interpolate the source data instead of generating synthetic data which is done by AI upscaling.
For scientifically correct scaling (think forensics) you will of course need to interpolate the source data instead of generating synthetic data which is done by AI upscaling.
Not entirely sure the AI scaling is more or less "correct" than the "interpolate the source data" method.
Both use priors, the former a large and complex one, the latter a simplistic one, but both "invent data based on assumptions".
Both use priors, the former a large and complex one, the latter a simplistic one, but both "invent data based on assumptions".
You can completely understand and intuit what interpolation does, and it doesn't drag in extra detailing from anywhere.
That's correct but probably also practically irrelevant.
If you prefer hands with six fingers ...
[deleted]
No, why would they? What would you expect an "AI" to do besides somehow implement an interpolation algorithm? The math is unavoidable.
AI algorithms encode domain knowledge in their weights that isn't present in simple interpolation functions. This can lead to better perceptual quality.
With AI based interpolation, I should be able to zoom in forever without seeing pixelation at any level of zoom.
Beyond a certain point that ceases to be interpolation and just becomes simulation. An image only has so much information in it, and AI isn't magic.
For lots of uses cases, if the end output meets user expectations, does it really matter if it was interpolated from simulated data?
For many use cases (editing old family photos etc), I'm increasingly not caring. I don't need per-pixel accuracy, I need a nice artifact that reminds me of a memory. If it looks 100 percent accurate to my eye, I'm not sure I care how much "fake" data it took to generate it. The important thing is it records the scene as I remember it.
This isn't hypothetical either; Photoshop already shipped their first iteration of the idea, for sure won't be the last:
> https://www.digitaltrends.com/computing/photoshop-generative...
> https://www.reddit.com/r/photoshop/comments/13s6tp9/using_th...
For many use cases (editing old family photos etc), I'm increasingly not caring. I don't need per-pixel accuracy, I need a nice artifact that reminds me of a memory. If it looks 100 percent accurate to my eye, I'm not sure I care how much "fake" data it took to generate it. The important thing is it records the scene as I remember it.
This isn't hypothetical either; Photoshop already shipped their first iteration of the idea, for sure won't be the last:
> https://www.digitaltrends.com/computing/photoshop-generative...
> https://www.reddit.com/r/photoshop/comments/13s6tp9/using_th...
Obviously if you don't care, it doesn't matter to you. But you not caring about objective reality versus your own vibes doesn't mean no one else has a valid reason to care. I personally would prefer my family photos not to contain machine generated hallucinations, because that doesn't actually represent reality.
But, just as obviously, you're correct that AI is going to force itself into every possible application and it doesn't matter.
But, just as obviously, you're correct that AI is going to force itself into every possible application and it doesn't matter.
No, but information isn't contained entirely in the image, but also in the assumptions about it. Simulation is exactly how you can retrieve more information than explicitly encoded in pixels - the information comes from how assumptions about the universe constrain possible states that would downscale to your original image.
I can take an photo of a locked door and no amount of assumptions you can make will allow you to retrieve generate an accurate photograph of what's behind it. You can assume anything. An empty room. A pile of money. Two unicorns copulating. Assumptions about an image are not information about the image, by definition.
You'd be surprised. Depending on the resolution of the photo, I might be able to get a low-res image of what's behind the door through the light leaks in its cracks. The more I know about the camera you used, the specific setting you took that photo in, the better chance I have at making a good guess of what's behind the door. Or behind the camera. These are the "assumptions" I'm talking about. External knowledge applied to resolve ambiguities.
For simpler examples, compare:
- GPS signal power at the receiver is under the thermal noise floor, yet they can still be recovered if you know exactly where to look.
- Nonlinear image transformations like blurs can still be inverted if you know or can guess the transform and exact coefficients used. Which is why, to redact something, you really want to paint over it with uniform color, instead of blurring it.
Or just a lot of what photogrammetry is about.
For simpler examples, compare:
- GPS signal power at the receiver is under the thermal noise floor, yet they can still be recovered if you know exactly where to look.
- Nonlinear image transformations like blurs can still be inverted if you know or can guess the transform and exact coefficients used. Which is why, to redact something, you really want to paint over it with uniform color, instead of blurring it.
Or just a lot of what photogrammetry is about.
After a certain level of zoom, simulation should be expected. Interpolation will only take you so far. If someone wants to zoom in 1000x and see a microscopic image, not just a smear of pixels, then so be it.
If you only want an image to be 100% realistic you must stay at the original zoom level.
If you only want an image to be 100% realistic you must stay at the original zoom level.
[deleted]
Neat aside, k = 1/x^∞ seems to be equivalent to nearest neighbor. (please excuse the abuse of notation)