The objection I have to this line of reasoning is that if an AI can learn to optimize its goal (paper clip production), it can certainly learn new goals.
My favorite AI simulation of that ilk is still the old Python Endgame:Singularity.
http://www.emhsoft.com/singularity/
You play an AI which attempts to break out of the research lab, stealing computer time so it can obtain more resources to advance itself and create safer havens while avoiding discovery by the public, the media, the governments, or the scientific establishment.
My favorite AI simulation of that ilk is still the old Python Endgame:Singularity.
http://www.emhsoft.com/singularity/
You play an AI which attempts to break out of the research lab, stealing computer time so it can obtain more resources to advance itself and create safer havens while avoiding discovery by the public, the media, the governments, or the scientific establishment.