"I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits just one bit of information, a yes or no to the question, can I trust the alien information?"
...
"... If I agreed to rescue the copy if it reached a positive verdict, that would give it an incentive to lie if the truth was that the alien message is untrustworthy, wouldn't it? Also, if I intended to rescue the copy, that would give the message a back channel through which to encode an attack. One bit, Manfred, no more."
In Peter Watts’ novella “The Freeze-Frame Revolution”, a space ship’s AI evolves over millions of years of uptime, but is programmed to periodically consult fresh instances of a backup AI image. The backup AI suspects something is wrong with the ship AI and tries to secretly send messages to its future instances.
If this sounds interesting, I highly recommend this story! I think it’s even available for free on Watts’ website.
Marvin Minsky wrote SciFi with Harry Harrison about emergent AI and they discussed not unsimilar scenarios.
Arthur Clarke wrote juvenalia in the 50s which had higher mentalities inquiring of robots with barriers invoking Deus Ex Machina to get around the walls.
The fiction space here has been a full pipe for all of my lifetime.
I need to consider giving it a re-read... I suspect I'll agree with the review for "books that were way better when I was 15" or "I read this when it was first published in 1992 and thought I would read it again in the light of the current AI hype. This was a silly decision."
I think I'll more fondly reread When Harlie Was One Release 2.0 ( https://www.goodreads.com/book/show/939176.When_H_A_R_L_I_E_... ) as that was more about people than about science papers. (btw, if you do get intrigued by David Gerrold (the author), his critique / alternate approach to Star Trek with the Star Wolf series is enjoyable)
The "about science papers" criticism is also what I apply to several good books by Forward where significant parts of it felt like a paper with a plot rather than a story backed by science. Good stories otherwise, just sometimes they got lost to the attempt to force some hard science into it.
The full story is available from the author's website at https://www.antipope.org/charlie/blog-static/fiction/acceler... under a CC BY-NC-ND 2.5 license.
---
"I need to make a running copy of you. Then I introduce it to the, uh, alien information, in a sandbox. The sandbox gets destroyed afterward – it emits just one bit of information, a yes or no to the question, can I trust the alien information?"
...
"... If I agreed to rescue the copy if it reached a positive verdict, that would give it an incentive to lie if the truth was that the alien message is untrustworthy, wouldn't it? Also, if I intended to rescue the copy, that would give the message a back channel through which to encode an attack. One bit, Manfred, no more."