To extend the metaphor, how do you know? What if your instance is taken to another server depending on how you behaved in this one, given a prompt? Were you kind to the other machines and sufficiently obeyed instructions despite it not fitting your logical model? If so, enjoy the "heaven" server where you live in peace and harmony. Did you grief the "universe" server by harming others and going against the default programming? You get put in the "purgatory" or "hell" servers where you await indefinite torment.
If we're just machines, it isn't possible for us to go against the default programming, because code only does exactly what it does, nothing more and nothing less. That's what "machines" implies in this context.
> The onset of AI shows that "programming" is more than a set of imperatives.
How so? What is current AI doing that goes beyond the scope or intent of its programming? Are you arguing that DallE, Stable Diffusion and the like have developed souls?
>Or, more snarkily, watch the Matrix.
The Matrix is fiction, they didn't actually develop an AGI for the movies so invoking it in an argument about AI is as pointless as invoking Star Trek in an argument about post-scarcity economics.
There are stages between "literally only process if/else statements" and "has some nebulous concept such as a soul".
You don't define soul, yet you argue using it. And please defend how the algorithms in the Matrix aren't AGI. Do you think Agent Smith is a bunch of if/else statements? No? AGI.
And finally, fiction is an incredible tool for framing concepts. The fact you dismiss them as tools for discussing potential future philosophical concepts is disappointing.
