There was a brief window where photography and videos became widespread so events could be documented and evidenced. Generative AI is drawing that period to an end, and we're returning to "who said / posted this" and having to trust the source rather than the image/message itself. It's a big regression.
Sure, but scale matters. 99% of images being fake is a different situation than 1% being fake. We can't just ignore that in favor of a "this always happened" argument.
Everything has always happened, so who cares? We need to go deeper than that. Many things that are perfectly a-okay today are only so because we do it on a small enough scale. Many perfectly fine things, if we scale them up, destroy the world. Literally.
Even something as simple as pirating, which I support, would melt all world economies if everyone did it with everything.
Lewis Carroll also was fooled by it, when Churchill showed to him. Abraham Lincoln who was there at the moment it happened confirmed that to me, I can show you the original email he sent me about it (bar the elements I'll have to hide due to top secret information being included in the rest of the message).
It is more that it is becoming available to the masses.
Doctoring photos in all manner of ways has been going on
for decades.
What is happening now will raise awareness of it and of course
make it a several magnitudes bigger problem.
I am sure there are large efforts ongoing to train AI to spot
AI photo, video, written production.
A system like printer tracking dots¹ may already be in widespread use.
Who would take the enormous amount of time to figure out if somesuch
is hiding somewhere in an llm, or related code.
Everything is a function of scale. IMO saying "this always happened" means nothing.
Lying on a small scale is no big deal. Lying on a big scale burns the world down. Me pirating Super Mario 64 means nothing. Everyone pirating everything burns the economy down. Me shooting a glass coke bottle is not note worthy. Nuclear warheads threaten humanity's existence.
Yes, AI fabrication is a huge problem that we have never experienced before, precisely because of how it can scale.
A mate of mine told me that ChatGPT started injecting zero width spaces into its output. Never fact checked this, but even if it's not done now, I'm sure it's bound to happen. Same for other types of watermarks.
Unfortunately that doesn't prove too much. It all hinges on the cameras, which are just computers, operating in a verifiable fashion. Which they don't, as no consumer available computer currently does, and I don't see this changing in the near future, both for technological and political reasons.
I put a lot of effort into thinking about what could be proper trustable on photos and videos [0], but short of performing
continuous modulation on the light sources the light of which is eventually captured by the cameras' sensors, there's no other way, and even that I'm not sure how would work with light actually doing the things its supposed to (reflection, refraction, scatter, polarization, etc.). And that's not even mentioning how the primary light source everyone understandably relies on, the Sun, will never emit such a modulated signal.
So what will happen instead I'm sure, is people deciding that they should not let perfection be the enemy of good, and moving on with this stuff anyways. Except this decision will not result in the necessary corresponding disclaimers reaching consumers, and so will begin the age of sham proofs and sham disprovals, and further radicalization. Here's to hope I'm wrong, and that these efforts will receive appropriate disclaimers when implemented and rolled out.
Things should simply be signed by people and institutions. People who were there or did extensive research vouching for authenticity and accuracy of representation.
Signing in hardware is nice, but you then still need to trust the company making the hardware to make it tamper proof and not tampered with by them. Additionally, such devices would have to be locked down to disallow things like manual setting of the time and date. It's a rabbit hole with more and more inconveniences and fewer benefits as you go down it.
Better to just go for reputation, and webs and chains of trust in people based approaches.
Yeah I'm not even talking about that side of things. I'm talking about after it leaves the camera. When someone shows you something, you should be able to say "I want to see the closest to original image". Humans make a lot of changes to images evidently to change perception. Even a simple crop can change the story being told.
Yeah, that's a good point. Kind of ties in with the video I linked. Also reminds me a bit to how in modern video containers the film grain is usually a separate thing, and is rendered onto the image during playback. One can imagine turning on a view where the modified parts of the image would become highlighted, assisting with speculations on what might have been altered and how.
I didn't mention this in my previous comment, but after having thought about it even further, I've eventually arrived at the idea that there can be basically an arbitrary amount of meaning embedded onto a photo or a video, and it being "verified" to me as a human really just means "it is what I think I'm looking at" - that's when I realized this task is likely unsolvable in absolute terms, and just how much more difficulty lies beyond the already hopelessly difficult crackpot idea of light source signal modulation that I came up with prior. But yeah, as I suggested, these are just what I could think of, and perfection should indeed not be the enemy of good.
> OPs point is far more interesting and deserves more discussion
The idea of "cryptographically signed photos coming out of cameras"? It's been discussed to death and is essentially a hope for magic technology to solve our social problems.
Also it won't work, technically: it's like asking for perfect a DRM implementation to be universally implemented everywhere.
I don't understand your hostility. There's nothing in what I wrote or in the way I wrote that would suggest to a good faith reader that my opinion is absolute, that I consider my opinion fact, or that my knowledge is all-encompassing. In your own words I merely "thought about it" - a phrase that means exactly just that, and nothing more. In a way, it's literally a request for further thoughts by others to make up for what I might have not thought of; the exact admission you're looking for, just stated implicitly.
It's immensely frustrating to dress up a natural language sentence with enough precision to try and account for all and every bit of nuance, so you should anticipate and actively consider that I have missed or implied some.
Despite this, you clearly did not, and instead went into attack mode on the assumption(!) that I did miss them intentionally.
I'd recommend you take your own advice on intellectual humility before offering it to others.
I think Nikon and Canon's past cameras had signed photos by default, and you could get the software for verification if you're a police dept or similar.
Both manufacturer's keys got extracted from their cameras, rendering the feature moot. Now, a more robust iteration is probably coming, possibly based on secure enclaves.
I'd love to have the feature and use it to verify that my images are not corrupted on disk.
Used to work on archival. You want to store your images in TIFF and use FEC if you really care. That withstands significant bitrot. Signing will just tell you that the image is broken but not allow you to recover it. You can do fine with SHA256 and 3-2-1 backup.
Thanks for the heads-up, didn't know TIFF/FEC. However, I can't because the images are .ARW (Sony's camera raw format). I keep multiple copies with hashes of them currently. I check the disks periodically.
Mid-term plan is to build an UNRAID array with patrolling. Will probably do backups with Borg on top of it, and keep another "working copy", so I have multiple layers of redundancy with some auto-remediation.
UNRAID will keep disk level consistency in good shape. Patrolling will do checksum tests, Borg can check its own backups against bitrot and can repair them in most cases. Working copy will contain everything and refill if something unthinkable happens.
You have to convert to an archival format from whatever your raw format is. The issue with raw files is that they are compressed so any bitrot in those leads to more than just losing a pixel or two here or there but sometimes the entire file. TIFF is not compressed at all. FEC allows single pixel corrections to be done accurately. We had a metric shit ton of LTO tapes, changers and online disk cache that handled all that.
I don't do that at home. I have 600GB of Nikon raws. I keep one copy on my Mac. I have a time machine backup in the house (in another room) on an Samsung T7 shield and I have an off site backup which is a quarterly rsync job to another Samsung T7 shield.
What prevents anyone to take a signed picture by photographing a generated/altered picture? You just need to frame it perfectly and make sure there are no reflections that could tell it's a picture of a picture and not a picture of the real world, very doable with a professional camera. All details that could give it out would disappear just lowering the resolution, which can be done in any camera.
With a bit (OK quite a lot) of fiddling, you could probably remove the CCD and feed the analog data into the controller, unless that's also got a crypto system in it.
Presumably if you were discovered you would then "burn" the device as its local key would be known then to be used by bad actors, but now you need to be checking all photos against a blacklist. Which also means if you buy a second hand device, you might be buying a device with "untrusted" output.
Any problem that requires cryptographic attestation or technical control of all endpoints is not a solution we should be pursuing. Think of it as a tainted primitive. Not to be implemented.
The problem of Trust is a human problem, and throwing technology at it just makes it worse.
I'm absolutely in agreement with that. The appetite for technical solutions to social problems seems utterly endless.
This particular idea has so many glaring problems that one might almost wonder if the motivation is less about "preventing misinformation" or "protecting democracy" or "thinking of the children" or whatever, and more about making it easier to prove you took the photo as you sue someone for using it without permission. But any technology promoted by Adobe couldn't be about DRM, so that's just crazy talk!
There have also been calls for a mechanism like this to prevent doctored photos of controversial newsworthy events being spread by news agencies. But afaik the only thing that came off it was "we only pay for camera jpg, only allowed changes are brightness, contrast, color".
That fixes the problem of content being manipulated and then the original being discounted as fake when challenged.
It doesn't do a whole lot for something entirely fictional, unless it becomes so ubiquitous that anything unsigned is assumed to be fake rather than just made on a "normal" device. And even if you did manage to sign every photo, who's managing those keys? It's the difference between TLS telling you what you see is what the server sent and trusting the server to send the truth in the first place.
I will assume you simply mean cryptographically signed as evidence of having been taken by a camera?
You do realize that would still not provide perfect proof that what was recorded by the camera was real, right? It does seem like an obsolete idea you may not have fully reconsidered in a while.
But considering that same old idea that dates from prior to the current state, I would also not be surprised if you imagined clandestinely including all kind of other things in this cryptographic signature like location, time and day, etc.; all of which can also be spoofed and is a tyrannical system’s wet dream.
You don’t think that would be immediately abused, as it was in other similar ways like through all the on device image scanning that was injected as counter-CSAM appeals to save the children…of course?
Would be very interesting to require generative AI companies to log their created images for that purpose, it isn't gonna eliminate self hosted alternative, but it can destroy any fake evidence faster
Logging them would be rather cost prohibitive, but images can be hashed + (invisibly) watermarked and video can be hashed frame by frame, in such a way that each frame authenticates the one before it. Surely there's a way to durably mark generated content.
Interesting! I wonder how they watermark audio. The obvious way would be to do it in some frequency inaudible to humans (say 30kHz), but most conventional file formats can't handle that. (You'd probably need to make a modification that contains an additional ultra-high frequency.)
Let's say you want to release a deepfake image. Your AI image, as generated, is not watermarked or hashed. Difficulty: Story-mode. You can release your image in two minutes flat.
Let's now let's say your AI deepfake image is watermarked/hashed upon generation. Difficulty: Intermediate to hard. You might be able to get it done, but not without real effort, and removing all traces of the watermark without leaving artefacts might be so difficult as to be nearly impossible.
...So it doesn't eliminate the possibility of fakes, but it hugely raises the cost and effort bar. Only the most motivated would bother.
Can that be done today, in a "zero effort" manner, even with watermarked stock images?
I tried some "AI watermark removers" on a Shutterstock image, and they did a pretty good job, but they left traces that would be obvious to any investigator.
I don't think this would solve the problem, unfortunately. AI can certainly be coerced into reproducing an existing image, leaving plenty of room for probable deniability.
Your comment follows two persistent HN tropes: (1) ignoring the article, which deals precisely with why the cost of production matters, and (2) steadfastly refusing to recognise that quantity has a quality of its own - in this case a monumental reduction in production cost clearly leads to a tectonic reshaping of the information landscape.
Funny that you mention the printing press. One of the first books published using it was about how to identify witches which led to "witch" burnings in Europe. At some point society adapted to the misinformation (and we are better off for it), but a lot of innocent people suffered in the meantime.
I did read the article and it does deal with fake photos being an old thing and name drops e.g. Stalin's former companions being removed one by one from photos.
"There was a brief window where photography and videos became widespread so events could be documented and evidenced."
Photos are video have in themself not been evidence. You need trust the photographer or publisher too, since the camera was invented.
Moving the cost from big $$$ level to my neighbour level makes scams more personalized, sure.
"Somewhat" is doing an awful lot of heavy lifting. Fakery has gone from handcrafted in small batches (or at worst, produced through much effort in sweatshops) to fully automated and mass produced.
However, the real breaking point in my view was when shills and semi-automated bots become so prevalent that they could fool people into believing some consensus had changed. Faking photos doesn't add much to that in my view.
I get bored by myself repeating it but I always think it's worth to mention: the scale and degree matter, in almost all cases of an issue, if not we can reduce a lot of issues to "it has always happened before".
Localised fires are common in nature, a massive wildfire is just a fire, at a different scale and degree. Lunatics raving about conspiracies were very common in public squares, in front of metro stations, anywhere there was a large-ish flow of people, now they are very common in social media, reaching millions in an instant, different scale and degree.
Just sheer scale and degree makes an issue a completely other issue, decreasing the effort required to fake a video to the point where a naïve/layperson cannot distinguish it from a real one is a massive difference. Before you needed to be technically proficient with a number of tools, put a lot of work, and get a somewhat satisfying result to fake something, now you just throw a magical incantation of words and it spits out a video you can deceive someone, it's a completely different issue.
"Nothing has changed in that regard" really, nothing?
Does anyone other than me notice this common tendency on HN:
1. Blockhain use case mentioned? Someone must say "blockchain doesn't solve any problems" no matter what, and always ignoring any actual substance of what's being done until pressed.
2. AI issue mentioned? Someone must say "nothing at all has changed, this problem has always existed" downplaying the vast leaps forward in scale, and quality of output, ignoring any details until pressed.
It's like when people feel the need to preface "there is nothing wrong with capitalism, but" before critiquing capitalism. You will not criticize the profit.
It's not really a shibboleth. What's the name for this type of thing, groupthink?
I would just call it a "pattern", but if you want to be more specific Re your 1/2, perhaps a "pattern of over-simplification". Over-simplification is, of course, basically human nature, not specific to HN, and something "scientists" of all stripes fight against in themselves. So, there may be a better one-worder.
EDIT: while oversimplification is essentially always a problem, nuance and persuasion are usually at odds. So, it's especially noticeable in contexts where people are trying to persuade each other. The best parts of HN are not that, but rather where they try to inform each other.
Things like the government MAHA report, full of fake references to papers that don't exist, probably didn't happen as blatantly before AI. Even if people could easily have lied back then too. The ease of with which AI lies (or "hallucinates") makes a qualitative difference.
Fake citations has always been common. Before they weren't just straight up fabrications. The cited papers usually exists, they just don't contain what is claimed by the referring paper.
At least made up citations are quick and easy to denounce.
The open question is who’s worse: one major tyrant three thousand miles away or three thousand minor tyrants a mile away. If we consent to live under capitalism, then we’re destined to live in a world of lies. We always have; it’s all we’ve ever known.
What we don’t know is whether we’ll be worse or better off when the technology of forgery is available to random broke assholes as easily as it is to governments and companies. More slop seems bad, but if our immunity against bullshit improves, people might redevelop critical thinking skills and capitalism could end.
> Generative AI is drawing that period to an end, and we're returning to "who said / posted this" and having to trust the source rather than the image/message itself. It's a big regression.
Which just goes to show that one of the core tenets of techno-optimism is a lie.
The Amish (as I understand them) actually have the right idea: instead of wantonly adopting every technology that comes along, they assess how each one affects their values and culture, and pick the ones that help and reject the ones that harm.
Which implies that they have a foundational epistemological, teleological and “coherent” philosophy
Not something that can be said of most people. Worse, the number of affinity groups with long term coherence collapses into niche religions and regional cults.
There’s no free lunch
If you want structural stability then you’re gonna have to give up individuality for the group vector, if you want individuality you’re not going be able to take advantage of group benefits.
Humans are never going to be able to figure out this balance because we don’t have any kind of foundational coherent Universal epistemological grounding that can be universally accepted.
Good luck trying to get the world to even agree on the age of the earth
