Reverse engineering Gemini's SynthID detection(github.com)
github.com
Reverse engineering Gemini's SynthID detection
https://github.com/aloshdenny/reverse-SynthID
65 comments
The dual-watermark theory makes alot of sense for defensive engineering. You always assume your outer layer will be broken and so keep a second layer that isn't publicly testable. Same as defence in depth anywhere else. I'm curious - as new models are being built constantly and they're naturally non-deterministic, do you think it's possible for end users to prove that?
> I'm curious - as new models are being built constantly and they're naturally non-deterministic, do you think it's possible for end users to prove that?
How is the model relevant? The models are proprietary and you never see any of its outputs that haven't been watermarked.
How is the model relevant? The models are proprietary and you never see any of its outputs that haven't been watermarked.
Seems like a very low-quality AI-assisted research repo, and it doesn't even properly test against Google's own SynthID detector. It's not hard at all (with some LLM assistance, for example) to reverse-engineer network requests to be able to do SynthID detection without a browser instance or Gemini access, and then you'd have a ground truth.
I read a lot of comments on HN that say something is not hard, yet don't provide a POC of their own or link to research they have knowledge of.
I also read a lot of comments on HN that start by attacking the source of the information, such as saying it was AI assisted, instead of the actual merits of the work.
The HN community is becoming curmudgeonly and using AI tooling as the justification.
I also read a lot of comments on HN that start by attacking the source of the information, such as saying it was AI assisted, instead of the actual merits of the work.
The HN community is becoming curmudgeonly and using AI tooling as the justification.
That's how life generally works. If your friend tells you, "I went to that new movie yesterday. It was very boring, I fell asleep midway." - then you either listen to his advice or don't. You don't ask your friend if they ever made a movie of their own.. And you don't ask for a 3rd party research of that movie's either.
As for AI specifically.. life is too short to read all the interesting pages already, and AI just makes is so much worse.
- AI is verbose in general, so you are spending a lot of time reading and not getting much new facts out of that.
- Heavy AI use often means that author has little idea about the topic themselves, and thus cannot engage in comments. Since discussion with authors are often most interesting part of HN, that makes submission less interesting.
And yes, it is possible to use AI assistance to create nice and concise report on the topics you can happily talk about, but then this would not be labeled as "AI".
As for AI specifically.. life is too short to read all the interesting pages already, and AI just makes is so much worse.
- AI is verbose in general, so you are spending a lot of time reading and not getting much new facts out of that.
- Heavy AI use often means that author has little idea about the topic themselves, and thus cannot engage in comments. Since discussion with authors are often most interesting part of HN, that makes submission less interesting.
And yes, it is possible to use AI assistance to create nice and concise report on the topics you can happily talk about, but then this would not be labeled as "AI".
It says not to use these tools to misrepresent AI-generated content as human-created. But the project is a watermark removal tool with a pip-installable CLI and strength settings named "aggressive" and "maximum." Calling this research while shipping turnkey watermark stripping is trying to have it both ways in a way that's uncomfortable to read.
The README itself reads like unedited AI output with several layers of history baked in.
- V1 and V2 appear in tables and diagrams but are never explained. V3 gets a pipeline diagram that hand-waves its fallback path.
- The same information is restated three times across Overview, Architecture, and Technical Deep Dive. ~1600 words padded to feel like a paper without the rigor.
- Five badges, 4 made up, for a project with 88 test images, no CI, and no test suite. "Detection Rate: 90%" has no methodology behind it. "License: Research" links nowhere and isn't a license.
- No before/after images, anywhere, for a project whose core claim is imperceptible modification.
- Code examples use two different import styles. One will throw an ImportError.
- No versioning. If Google changes SynthID tomorrow, nothing tells you the codebook is stale.
The underlying observations about resolution-dependent carriers and cross-image phase consistency are interesting. The packaging undermines them.
The README itself reads like unedited AI output with several layers of history baked in.
- V1 and V2 appear in tables and diagrams but are never explained. V3 gets a pipeline diagram that hand-waves its fallback path.
- The same information is restated three times across Overview, Architecture, and Technical Deep Dive. ~1600 words padded to feel like a paper without the rigor.
- Five badges, 4 made up, for a project with 88 test images, no CI, and no test suite. "Detection Rate: 90%" has no methodology behind it. "License: Research" links nowhere and isn't a license.
- No before/after images, anywhere, for a project whose core claim is imperceptible modification.
- Code examples use two different import styles. One will throw an ImportError.
- No versioning. If Google changes SynthID tomorrow, nothing tells you the codebook is stale.
The underlying observations about resolution-dependent carriers and cross-image phase consistency are interesting. The packaging undermines them.
Agreed. This isn't punk this just helps the bad guys. Society needs to know what content is AI generated and what is not.
This was never going to be a reliable way to do it. It's basically the evil bit . It only works for as long as everyone is making a good-faith effort to follow the convention. But the bad guys do not do that.
It really doesn't need such capability. Nor does it need the capability to know what human generated it either.
If that's the case, society will inevitably be disappointed.
There are already ten million AI image generators, the overwhelming majority of which do not watermark their outputs. Google auto-inserting them is nice, but ultimately this kind of tool to remove them will inevitably be widespread.
There are already ten million AI image generators, the overwhelming majority of which do not watermark their outputs. Google auto-inserting them is nice, but ultimately this kind of tool to remove them will inevitably be widespread.
[deleted]
Okay... this tests its own ability to remove the watermark against its own detector. It doesn't test against Gemini's SynthID app. So it does nothing...
SynthID is visible in some generations (areas with a lot of edges, or text), I wonder if this would make them look better.
And it gets more and more visible in every edit if you use Nano Banana for the edits.
Im confident i saw the watermark in use today, in nano banana, i copied the image from chrome into slack. the resulting upload was a black square with a red dot. and not the image i had generated.
I remember experiencing something similar. But then iirc I noticed you can draw on a screenshot and I think i was copying the random dots I made by accidental clicks... You sure it wasnt this?
Ok i get that eventually someone was gonna do this but why would we want to purposely remove one of the only ways of detecting if an image is ai generated or not...?
Because an attacker will do that the same thing and without sharing that knowledge good actors are in the dark. It's the same reason we share known security problems, since there will be bad actors that discover the same bugs and use them for much worse.
It was always going to be available to some people, but not everyone would know or believe that. Now they will.
Much like every other thing in the tech world. He'll, it's why AI will kill us off eventually.
If a system depends on every person on the planet not doing one particular thing or the system breaks, expect the system to break quickly.
This is an especially common trope in software. If someone can make software that does something you consider bad, it will happen. Also it's software. There is no difference between it being available to one person or a million. The moment the software exists and can be copied an unbound number of times.
If a system depends on every person on the planet not doing one particular thing or the system breaks, expect the system to break quickly.
This is an especially common trope in software. If someone can make software that does something you consider bad, it will happen. Also it's software. There is no difference between it being available to one person or a million. The moment the software exists and can be copied an unbound number of times.
More likely than not it would be used to deanonymise the author.
So it's a "no" by default.
So it's a "no" by default.
Fundamentally it's a fuzzy signal and people shouldn't rely on it. The general public does not understand Boolean logic (oh, so the SynthID is not there, therefore this image is real). The sooner AI watermarking faces its deserved farcical demise the better.
Also something about how AI is not special and we haven't added or needed invisible watermarks for other ways media can be manipulated deceptively since time immemorial, but that's less of a practical argument and more of a philosophical one.
Also something about how AI is not special and we haven't added or needed invisible watermarks for other ways media can be manipulated deceptively since time immemorial, but that's less of a practical argument and more of a philosophical one.
I’m not very well read on the topic and you seen to take a strong “con” stance. Curious to hear why you think it deserves such a demise
People think that just because they have a way to prove that an image is AI, their worries of misinformation are solved. Better to acknowledge that wherever you look people will be trying to deceive you even if their content won't have as obvious an indicator as SynthID.
[deleted]
Not GP, but I’m pretty “con” too.
Because it’s meaningless for what it’s being marketed for. It’s conceptually inverted. It’s a detector that will detect 100% of the stuff that doesn’t mind being detected, and only the dumbest fraction of stuff that doesn’t want to be detected.
No fault of the extremely smart and capable people who built it. It’s the underlying notion that an imperceptible watermark could survive contact with mass distribution… it gives the futile cat-and-mouse vibes of the DRM era.
Good guys register their guns or whatever, bad guys file off the serial numbers or make their own. Sometimes poorly, but still.
All of which would be fine as one imperfect layer of trust among many (good on Google for doing what they can today). The frustrating/dangerous part is that it seems to be holding itself out as reliable to laypeople (including regulators). Which is how we end up responding to real problems with stupid policy.
People really want to trust “detectors,” even when they know they’re flawed. Already credulous journalists report stuff like “according to LLMDetector.biz, 80% of the student essays were AI-generated.” Jerry Springer built an empire on lie detector tests. British defense contractor ATSC sold literal dowsing rods as “bomb detectors,” and got away with it for a while [2].
It’s backward to “assume it’s not AI-origin unless the detector detects a serial number, since we made the serial number hard to remove.” Instead, if we’re going to “detector” anything, normalize detecting provenance/attestation [e.g. 0]: “maybe it’s an original @alwa work, but she always signs her work, and I don’t see her signature on this one.”
Something without a provable source should be taken with a grain of salt. Make it easy for anyone to sign their work, and get audiences used to looking for that signature as their signal. Then they can decide how much they trust the author.
Do it through an open standards process that preserves room for anyone to play, and you don’t depend on Big Goog’s secret sauce as the arbiter of authenticity.
I hear that sort of thinking is pretty far along, with buy-in from pretty major names in media/photography/etc. The C2PA and CAI are places to look if you’re interested [1].
…and that is why I am “con.”
[0] https://contentcredentials.org/
[1] https://c2pa.org/ , https://contentauthenticity.org/
[2] https://en.wikipedia.org/wiki/ADE_651
Because it’s meaningless for what it’s being marketed for. It’s conceptually inverted. It’s a detector that will detect 100% of the stuff that doesn’t mind being detected, and only the dumbest fraction of stuff that doesn’t want to be detected.
No fault of the extremely smart and capable people who built it. It’s the underlying notion that an imperceptible watermark could survive contact with mass distribution… it gives the futile cat-and-mouse vibes of the DRM era.
Good guys register their guns or whatever, bad guys file off the serial numbers or make their own. Sometimes poorly, but still.
All of which would be fine as one imperfect layer of trust among many (good on Google for doing what they can today). The frustrating/dangerous part is that it seems to be holding itself out as reliable to laypeople (including regulators). Which is how we end up responding to real problems with stupid policy.
People really want to trust “detectors,” even when they know they’re flawed. Already credulous journalists report stuff like “according to LLMDetector.biz, 80% of the student essays were AI-generated.” Jerry Springer built an empire on lie detector tests. British defense contractor ATSC sold literal dowsing rods as “bomb detectors,” and got away with it for a while [2].
It’s backward to “assume it’s not AI-origin unless the detector detects a serial number, since we made the serial number hard to remove.” Instead, if we’re going to “detector” anything, normalize detecting provenance/attestation [e.g. 0]: “maybe it’s an original @alwa work, but she always signs her work, and I don’t see her signature on this one.”
Something without a provable source should be taken with a grain of salt. Make it easy for anyone to sign their work, and get audiences used to looking for that signature as their signal. Then they can decide how much they trust the author.
Do it through an open standards process that preserves room for anyone to play, and you don’t depend on Big Goog’s secret sauce as the arbiter of authenticity.
I hear that sort of thinking is pretty far along, with buy-in from pretty major names in media/photography/etc. The C2PA and CAI are places to look if you’re interested [1].
…and that is why I am “con.”
[0] https://contentcredentials.org/
[1] https://c2pa.org/ , https://contentauthenticity.org/
[2] https://en.wikipedia.org/wiki/ADE_651
To pass fake/modified image as genuine?
Uh... you can do this pretty easily since day 1. Just use Stable Diffusion with a low denoising strength. This repo presents an even less destructive way[0], but it has always been very easy to hide that an image is generated by Nano Banana.
[0]: if it does what it claims to do. I didn't verify. Given how much AI writing in the README my hunch is that this doesn't work better than simple denoising.
[0]: if it does what it claims to do. I didn't verify. Given how much AI writing in the README my hunch is that this doesn't work better than simple denoising.
[deleted]
kinda ironic you can clearly see signs of Claude, as it shows misaligning table walls in the readme doc
Parenthesized, comma-separated lists with no “and” is an even stronger tell. Claude loves those.
I also use those extensively, they just flow better, especially if you have an "and" in the surrounding sentence.
> kinda ironic you can clearly see signs of Claude, as it shows misaligning table walls in the readme doc
This one is such a gigantic clusterfuck... They're mimicking ASCII tables using Unicode chars of varying length and, at times, there's also an off-by-one error. But the model (not Claude, but the model underneath it) is capable of generating ASCII tables.
P.S: I saw the future... The year is 2037 and we've got Unicode tables still not properly aligned.
This one is such a gigantic clusterfuck... They're mimicking ASCII tables using Unicode chars of varying length and, at times, there's also an off-by-one error. But the model (not Claude, but the model underneath it) is capable of generating ASCII tables.
P.S: I saw the future... The year is 2037 and we've got Unicode tables still not properly aligned.
I mean, just reading the readme content it is pretty obvious it is Claude
It's not really hard as it looks https://deepwalker.xyz/blog/bypassing-synthid-in-gemini-phot...
if you downscale then upscale it removes the watermark
I don't understand all the handwringing. If it's this easy to remove SynthID from an AI-generated image then it wasn't a good solution in the first place.
There is no solution. I don't know why people discuss this subject as if there is a technical solution. As if there are fairies or souls hidden in the pixels that help us tell what is AI generated and what is not.
If you want to make an AI generated image but don't want other people to know that it's AI, the most obvious solution is to not use Gemini. Synth ID is watermarking. It's only ever going to be useful to good actors, who want an AI generated image and aren't trying to hide the fact that it's AI generated.
Never underestimate that people are lazy.
Sure, and things like this help drive home that SynthID wasn't a solution at all.
Sure there is a solution, you are just looking at it the wrong way. Make non-AI images provably unaltered with signed keys from the device (e.g. the camera) that took it.
That's pretty much impossible though.
One workflow that some artists use is that they draw with ink on paper, scan, and then digitally color. Nothing prevents someone from generating line art using generative AI, printing it, scanning it, and coloring it.
And what if someone just copy pastes something into Photoshop or imports layers? That's what you'd do for composites that mix multiple images together. Can one copy paste screenshots into a multi layer composition or is that verboten and taints the final image?
And what about multi program workflows? Let's say I import a photo, denoise it in DxO, retouch in affinity photo, resize programmatically using image magick, and use pngcrush to optimize it, what metadata is left at the end?
One workflow that some artists use is that they draw with ink on paper, scan, and then digitally color. Nothing prevents someone from generating line art using generative AI, printing it, scanning it, and coloring it.
And what if someone just copy pastes something into Photoshop or imports layers? That's what you'd do for composites that mix multiple images together. Can one copy paste screenshots into a multi layer composition or is that verboten and taints the final image?
And what about multi program workflows? Let's say I import a photo, denoise it in DxO, retouch in affinity photo, resize programmatically using image magick, and use pngcrush to optimize it, what metadata is left at the end?
Next comes registration your camera with the government to ensure you're not doing "bad" things with it.
How many minutes do you think it would take before someone figured out how to crack that?
On Pixels and iPhones it would be impossible since they have actually secure hardware that could both hold the keys and sign/verify the image.
The camera module sits outside the secure area, meaning it would need to send data in to be signed. How does the phone know that it's getting legitimate data from the camera module, or data someone else is just piping in? Also, you could probably get a fairly high quality image by just taking a photo of something AI generated in the right lighting conditions.
If the premise is that everyone would just agree on the same protocol, I have an even more unbreakable solution: every image has to be upload to a blockchain the moment it is (claimed to be) created. Otherwise it's AI.
If only everyone just agrees with me.
If only everyone just agrees with me.
Which works for about 5 minutes until someone leaks a manufacturer's private key or extracts it from a device...
Yes. This kind of project needs aggressive red teaming, it leads to better products and we need excellent products in this space.
This project proves what red teaming was in place wasn't good enough.
This project proves what red teaming was in place wasn't good enough.
> We're actively collecting pure black and pure white images generated by Nano Banana Pro to improve multi-resolution watermark extraction.
Oh hey, neat. I mentioned this specific method of extracting SynthID a while back.[1]
Glad to see someone take it up.
[1]: https://news.ycombinator.com/item?id=47169146#47169767
Oh hey, neat. I mentioned this specific method of extracting SynthID a while back.[1]
Glad to see someone take it up.
[1]: https://news.ycombinator.com/item?id=47169146#47169767
FWIW, I had Nano Banana create pure white/black images in February, and there was no recognizable watermark in them (all pixels really were #ffffff / #000000 IIRC).
Meta: your comment was marked [dead], like a few other constructive comments I saw in recent days. Not sure why.
Meta: your comment was marked [dead], like a few other constructive comments I saw in recent days. Not sure why.
I tried it with Nano Banana 2 through the API just now, and it was content filtering me on both white and black images.
I suspect they strip the SynthID for these specific cases to prevent exfiltration of the steganography.
I appreciate you pointing it out, but this account is banned. Thank you for vouching though!
I appreciate you pointing it out, but this account is banned. Thank you for vouching though!
[deleted]
[deleted]
If you assume competence from Google, they probably have two different watermarks. A sloppy one they offer an online oracle for and one they keep in reserve for themselves (and law enforcement requests).
Also given that it's Google we are dealing with here, they probably save every single image generated (or at least its neural hash) and tie it to your account in their database.