GenAI Image Editing Showdown(genai-showdown.specr.net)
genai-showdown.specr.net
GenAI Image Editing Showdown
https://genai-showdown.specr.net/
54 comments
People gave Altman shit for enabling NSFW in ChatGPT, but I see that as a step in the right direction. The right direction being: the one that leads to less corporate censorship.
>In the same way, using gpt5 is now very unbearable to me as it almost always starts all responses of a conversation by things like: "Great question"
User preference data is toxic. Doing RLHF on it gives LLM sycophancy brainrot. And by now, all major LLMs have it.
At least it's not 4o levels of bad - hope they learned that fucking lesson.
>In the same way, using gpt5 is now very unbearable to me as it almost always starts all responses of a conversation by things like: "Great question"
User preference data is toxic. Doing RLHF on it gives LLM sycophancy brainrot. And by now, all major LLMs have it.
At least it's not 4o levels of bad - hope they learned that fucking lesson.
I have seen a few normally progressive types act quite conservative puritan over the NSFW ChatGPT thing. It seems there are quite a lot of people consider things to be uniformly good or bad and their opinion of the whole colours their opinion of the parts.
OpenAI are in a difficult position when it comes to global standards. It's probably easier to see from outside of the United States, because the degree to which the historical puritanism has influenced everything is remarkable. I remember the release of the Watchmen film and being amazed at how pervasive the preoccupation with a penis was in the media coverage.
OpenAI are in a difficult position when it comes to global standards. It's probably easier to see from outside of the United States, because the degree to which the historical puritanism has influenced everything is remarkable. I remember the release of the Watchmen film and being amazed at how pervasive the preoccupation with a penis was in the media coverage.
People in the US went ballistic over Mass Effect showing an outline of a butt, in the dark, for 1 second.
Name me one piece of enterprise software that lets you do NSFW things. The way people jump to 1984 with no thought is double plus bad. ChatGPT is a piece of enterprise software. They are trying to sell it to large companies at large prices. This is not a rhetorical question, do you think if you could generate nude images of celebrities or picture of extreme violence, corporations would buy it? Having been a director at a Fortune 500 company that bought software, I can tell you with 100% certainty the answer is "no".
Technically? Microsoft Word certainly lets one write smut, and Photoshop certainly allows one to draw pornography? They won’t like, produce NSFW things automatically of course.
Exactly. Programs that don't let you do things based on the content should be thought of as weird/broken.
Imagine if we woke up tomorrow morning and grep refused to process a file because there was "morally objectionable" content in it (objectionable as defined by the authors of grep). We would rightly call that a bug and someone would have a patch ready by noon. Imagine if vi refused to save if you wrote something political. Same thing. Yet, for some reason, we're OK with this behavior from "certain" software?
Imagine if we woke up tomorrow morning and grep refused to process a file because there was "morally objectionable" content in it (objectionable as defined by the authors of grep). We would rightly call that a bug and someone would have a patch ready by noon. Imagine if vi refused to save if you wrote something political. Same thing. Yet, for some reason, we're OK with this behavior from "certain" software?
There is more than one way we could generalize the precedent previously set, imo.
None of the templates included with e.g. Word were for smut.
Word allowed you to type in smut, but it didn’t produce smut that wasn’t written by the user. For previous enterprise software, that wasn’t really a relevant question.
So… I don’t think it is obvious that the “Word lets you type in smut” implies “ChatGPT should produce smut if you ask it for smut.”
I guess precedent might imply “if you write some smut and ask it to fix the grammar, it shouldn’t refuse on the basis of what you wrote being smut”?
None of the templates included with e.g. Word were for smut.
Word allowed you to type in smut, but it didn’t produce smut that wasn’t written by the user. For previous enterprise software, that wasn’t really a relevant question.
So… I don’t think it is obvious that the “Word lets you type in smut” implies “ChatGPT should produce smut if you ask it for smut.”
I guess precedent might imply “if you write some smut and ask it to fix the grammar, it shouldn’t refuse on the basis of what you wrote being smut”?
[deleted]
Companies like PH use full Enterprise stacks from AWS to Oracle. Hell, CloudFlare actively takes flack for running much worse websites like 8Chan, Daily Stormer, etc. and they are as enterprise-focused as it gets.
> Name me one piece of enterprise software that lets you do NSFW things.
Photoshop, MS word.
Photoshop, MS word.
I can't think of any that restrict it. Sharepoint refusing an NSFW photo or Oracle refusing to store video isn't a thing.
Seemingly they don't have tests to see whether on certain areas their model gets better or worse.
Try some of the Chinese models. Much less restrictive. With some obvious exceptions.
There isn’t a date in the article, but I know I had read this months ago. And sure enough, wayback has the text-to-image page from April.
But the image editing page linked at the top is more recent, and was added sometime in September. (And was presumably the intended link) I hadn’t read that page yet. Odd there is no dates, at first glance one might think the pages were made at the same time.
But the image editing page linked at the top is more recent, and was added sometime in September. (And was presumably the intended link) I hadn’t read that page yet. Odd there is no dates, at first glance one might think the pages were made at the same time.
> There isn’t a date in the article
SEO guys convinced everyone that articles without dates do better on search engines. I hope both sides of their pillow is hot.
SEO guys convinced everyone that articles without dates do better on search engines. I hope both sides of their pillow is hot.
I discovered this independently myself a decade ago since it’s true
fucking marketing people screw us over on so many levels...
Yeah this is very old. Although anything older than a week is reasonably old in AI.
EDIT, looks like I didn’t click on the “image editing” tab when I went to the site, so I guess take the rest of my below comments criticizing the terminology with a grain of salt…
“Image editing” is a curious term, as it appears the site/topic is actually all about generating new images. The term in my mind should be for actual editing of existing, real, images, Eg “remove the coffee table” from this living room photo after uploading the image. I’ve found the actual “image generation” models to be bad at this because they introduce too many artifacts that weren’t in the original, which makes sense because they are really geared for creating images out of thin air.
Multimodal models like qwen3-vl-30b-a3b, however, seem to do quite well with editing existing images without trying to constantly add in new things or trying to change the image in ways that you don’t want, as if it’s trying to do the “lets just generate a new image” thing. imagegpt.com is also good for editing existing images, but not sure what model they are using on the backend.
“Image editing” is a curious term, as it appears the site/topic is actually all about generating new images. The term in my mind should be for actual editing of existing, real, images, Eg “remove the coffee table” from this living room photo after uploading the image. I’ve found the actual “image generation” models to be bad at this because they introduce too many artifacts that weren’t in the original, which makes sense because they are really geared for creating images out of thin air.
Multimodal models like qwen3-vl-30b-a3b, however, seem to do quite well with editing existing images without trying to constantly add in new things or trying to change the image in ways that you don’t want, as if it’s trying to do the “lets just generate a new image” thing. imagegpt.com is also good for editing existing images, but not sure what model they are using on the backend.
I've actually gotten this comment a couple of times - perhaps I should make the nav bar at the top more prominently displayed.
WRT to Qwen3, is it possible that the API/site you were using was passing your "image edit requests" to something like Qwen-Edit [1] under the covers?
To my knowledge, Qwen3-VL (Vision Language) isn't capable of generating/modifying images - it's purely for doing reasoning about images.
[1] https://huggingface.co/Qwen/Qwen-Image-Edit
WRT to Qwen3, is it possible that the API/site you were using was passing your "image edit requests" to something like Qwen-Edit [1] under the covers?
To my knowledge, Qwen3-VL (Vision Language) isn't capable of generating/modifying images - it's purely for doing reasoning about images.
[1] https://huggingface.co/Qwen/Qwen-Image-Edit
I don’t know if you and I are looking at the same site because all I see is existing images being edited with GenAI.
Input: bald man Prompt: give bald man hair Output: edited original, now with hair
That looks like editing to me.
Or are we strictly adhering to the ‘generating new images’ definition because these models technically recreate the entire image? It would be like editing a photo in Photoshop. If you hit “Save” you edited the photo. But if you hit “Save As” and create a new file, the photo wasn’t edited but created as a new image?
Input: bald man Prompt: give bald man hair Output: edited original, now with hair
That looks like editing to me.
Or are we strictly adhering to the ‘generating new images’ definition because these models technically recreate the entire image? It would be like editing a photo in Photoshop. If you hit “Save” you edited the photo. But if you hit “Save As” and create a new file, the photo wasn’t edited but created as a new image?
Actual link seems to be: https://genai-showdown.specr.net/image-editing
This is the editing link yes. I just got done looking at it from the other link.
The other stuff is text to image (not editing)
The other stuff is text to image (not editing)
I'd assume that behind the scenes the models generate several passes and only show the user the best one, that would be smart, as to to make it seem their model is better than others
Is also pretty obvious that the models have some built in prompt system rules that makes the final output a certain style. They seem very consistent
It also looks like 40 has the temperature turned way down, to ensure max adherence, while midjourney etc seem to have higher temperature.more interesting end results, flourishing, complex Materials and backgrounds
Also what's with 4o's sepia tones. Post editing in the gen workflows?
I don't believe any of these just generate the image though, there's likely several steps in each workflows to present the final images outputted to the user in the absolute best light.
Is also pretty obvious that the models have some built in prompt system rules that makes the final output a certain style. They seem very consistent
It also looks like 40 has the temperature turned way down, to ensure max adherence, while midjourney etc seem to have higher temperature.more interesting end results, flourishing, complex Materials and backgrounds
Also what's with 4o's sepia tones. Post editing in the gen workflows?
I don't believe any of these just generate the image though, there's likely several steps in each workflows to present the final images outputted to the user in the absolute best light.
You can run some image models locally if you want to prove to yourself how well they can do with just a single generation from a prompt with no extra steps.
I've done this enough to suspect that most hosted image models don't increase their running costs to try and get better results through additional passes without letting the user know what they are doing.
Many of the LLM-driven models do implement a form of prompt rewriting though (since effectively prompting image models is really hard) - some notes on how DALL-E 3 did that here: https://simonwillison.net/2023/Oct/26/add-a-walrus/
I've done this enough to suspect that most hosted image models don't increase their running costs to try and get better results through additional passes without letting the user know what they are doing.
Many of the LLM-driven models do implement a form of prompt rewriting though (since effectively prompting image models is really hard) - some notes on how DALL-E 3 did that here: https://simonwillison.net/2023/Oct/26/add-a-walrus/
There are numbers on how many tries it took. I would also find the individual prompts and images interesting.
I had to upvote immediately once I got to Alexander the Great on a Hippity Hop
I had completely forgotten about the hippity hop and coming across it here brought back all kinds of childhood memories. Those things were fun!
They were always called "space hoppers" here in the UK, and always looked like this:
https://en.wikipedia.org/wiki/Space_hopper#/media/File:Space...
https://en.wikipedia.org/wiki/Space_hopper#/media/File:Space...
The horse chimera is much better
Please fix the title, or change the link.
The title of this article is "image editing showdown", but the subject is actually prompt adherence in image generation from prompting.
Midjourney and Flux Dev aren't image editing models. (Midjourney is an aesthetically pleasing image generation model with low prompt adherence.)
Image editing is a task distinct from image generation. Image editing models include Nano Banana (Gemini Flash), Flux Kontext, and a handful of others. gpt-image-1 sort of counts, though it changes the global image pixels such that it isn't 1:1 with the input.
I expect that as image editing models get better and more "instructive", classical tools like Photoshop and modern hacks like ComfyUI will both fall away to a thin fascade over the models themselves. Adobe needs to figure out their future, because Photoshop's days are numbered.
Edit: Dang, can you please fix this? Someone else posted the actual link, and it's far more interesting than the linked article:
https://genai-showdown.specr.net/image-editing
This article is great.
The title of this article is "image editing showdown", but the subject is actually prompt adherence in image generation from prompting.
Midjourney and Flux Dev aren't image editing models. (Midjourney is an aesthetically pleasing image generation model with low prompt adherence.)
Image editing is a task distinct from image generation. Image editing models include Nano Banana (Gemini Flash), Flux Kontext, and a handful of others. gpt-image-1 sort of counts, though it changes the global image pixels such that it isn't 1:1 with the input.
I expect that as image editing models get better and more "instructive", classical tools like Photoshop and modern hacks like ComfyUI will both fall away to a thin fascade over the models themselves. Adobe needs to figure out their future, because Photoshop's days are numbered.
Edit: Dang, can you please fix this? Someone else posted the actual link, and it's far more interesting than the linked article:
https://genai-showdown.specr.net/image-editing
This article is great.
If you’re interested in side by side analysis of carious image gen tools, i review them:
https://generative-ai.review/2025/09/september-2025-image-ge...
https://generative-ai.review/2025/09/september-2025-image-ge...
Where is the famous "Don't draw a green elephant?"
UPD: suggested here https://github.com/scpedicini/genai-showdown-public/discussi...
UPD: suggested here https://github.com/scpedicini/genai-showdown-public/discussi...
The "editing" showdown is very good. Introduced me to the Seedream model which i didn't know about until now.
I don't fully understand the iterative methodology tho - they allow multiple attempts, which are judged by another multimodal llm? Won't they have limited accuracy in itself?
I don't fully understand the iterative methodology tho - they allow multiple attempts, which are judged by another multimodal llm? Won't they have limited accuracy in itself?
I tried to make the judgement criteria more clear in the FAQ section - I'll post it here:
What is the metric on which these models are being judged?
What is the metric on which these models are being judged?
It's hard to define a discrete rubric for grading at an inherently qualitative level. To keep things simple, this test is purely PASS/FAIL - unsuccessful means that the model NEVER managed to generate an image adhering to the prompt. For example, Midjourney 7 did not manage to generate the correct vertical stack of translucent cubes ordered by color in 64 generation attempts. In many cases, we often attempt a generous intepretation of the prompt - if it gets close enough, we might consider it a pass.
Put another way: if I were to show the final image to a random stranger on the street, would they be able to guess what the original prompt was? (aka the Pictionary test).
To paraphrase former Supreme Court Justice Potter Stewart, "I may not be able to define a passing image, but I know it when I see it."
"LLMs judged by LLMs" is the industry standard. Can't put a human judge in a box and have him evaluate and rate a set of 7600 responses on demand.
Now, are LLM judges flawed? Obviously. But they are more shelf stable than humans, so it's easier to compare different results. And as long as you use an LLM judge as a performance thermometer and not a direct optimization target, you aren't going to be facing too many issues from that.
If you are using an LLM judge as a direct optimization target though? You'll see some funny things happen. Like GPT-5 prose. Which isn't even the weirdest it gets.
Now, are LLM judges flawed? Obviously. But they are more shelf stable than humans, so it's easier to compare different results. And as long as you use an LLM judge as a performance thermometer and not a direct optimization target, you aren't going to be facing too many issues from that.
If you are using an LLM judge as a direct optimization target though? You'll see some funny things happen. Like GPT-5 prose. Which isn't even the weirdest it gets.
> "A dolphin is using its fluke to discipline a mermaid by paddling it across the backside."
If this one were shown in a US work environment, I might say a collegial something privately to the person, about it not seeming the most work-appropriate.
If this one were shown in a US work environment, I might say a collegial something privately to the person, about it not seeming the most work-appropriate.
I think I’d probably say that the prompts are telling me more about the author than I think is necessary for these tests… I hope they were at least sampled from responses.
Hey everyone! If you need a place to go test image models out against one another, please go to BrandImageGen .com:) Would love to see some signups as we have a pretty nice free tier:)
To me this goes to show how far ahead Google is in the space.
The ability to clearly understand the image being edited, and make edits that look natural to that understanding, are far beyond any of the other models.
The ability to clearly understand the image being edited, and make edits that look natural to that understanding, are far beyond any of the other models.
Is there any AI image generator/editor that is good at creating graphics with transparent background? Nano Banana and some others output a white grey checkered background (fake transparency).
Hey Addend! If you sign up brandimagegen . com, chatGPT image models create graphics w/ transparent backgrounds. You can also just rm backgrounds with my background remover tool.
I am biased on this since I built it and it officially launches on Friday 10/31
I am biased on this since I built it and it officially launches on Friday 10/31
>Cephalopodic Puppet Show
I'm pretty sure that only Gemini made it. Other models did not meet the 'each tentacle covered' criteria.
I'm pretty sure that only Gemini made it. Other models did not meet the 'each tentacle covered' criteria.
[deleted]
for the OpenAI 4o model on the octopus sock puppet prompt, the prompt clearly states that each tentacle should have a sock puppet, whereas the OpenAI 4o image only has 6 puppets with 2 tentacles being puppetless. I’m not sure if we can call that a pass
What about the classic:
A analog watch that shows the time 08:15?
Did current models overcome the 10:10 bias?
Did current models overcome the 10:10 bias?
This would be easy to patch the models to fix. Just gather a small amount of training data for these cases, eg. "change the clock hands to 5:30" with the corresponding edit.
Three tuple: (original image, text edit instruction, final image).
Easy to patch for editing models, anyway. Maybe not text to image models.
Three tuple: (original image, text edit instruction, final image).
Easy to patch for editing models, anyway. Maybe not text to image models.
waste of money and effort, imo. there are +- infinity of such small things to fix.
It's a common enough use case that it'll be added at some point.
It probably comes up more than you think. Storyboarding, product placement, model images, etc.
It's not critical in the short term, but it'll wind up on their backlog for sure.
It probably comes up more than you think. Storyboarding, product placement, model images, etc.
It's not critical in the short term, but it'll wind up on their backlog for sure.
Slight nit: it lists “OpenAI 4o” but the model used by ChatGPT is a distinct model labeled “gpt-image-1” iirc
A prompt id love to see: person riding in a kangaroo pouch.
Most of the pure diffusion models haven’t been able to do it in my experience.
Edit: another commenter pointed out the analog clock test, lets add the “analog clock showing 3:15” as well (:
A prompt id love to see: person riding in a kangaroo pouch.
Most of the pure diffusion models haven’t been able to do it in my experience.
Edit: another commenter pointed out the analog clock test, lets add the “analog clock showing 3:15” as well (:
The link is to the imagegen test not the editing one. Here 4o was used to preprocess the prompt.
It's like 1964 but corporate enforced. Now there are tasks that you are not allowed to do despite being legal.
In the same way, using gpt5 is now very unbearable to me as it almost always starts all responses of a conversation by things like: "Great question", "good observation worthy of an expert", "you totally right", "you are right to ask the question"...