Paperclip Maximiser

[Rynn]

The objection I have to this line of reasoning is that if an AI can learn to optimize its goal (paper clip production), it can certainly learn new goals.

Arguably an AI making it's own goals is part of the proposition. Simplified any intelligent decision making entity needs three things: Goals, Values and Knowledge. Goals are hierarchical with most of them being in service to a higher priority goal all the way up to the first priority. Selecting new subgoals in service of an over-goal is important, in this case the paperclip AI creates new subgoals of inventing better paperclip manufacturing robots in pursuit of its first priority goal "make more paperclips". Knowledge is the store of procedures necessary to achieve goals. It's one thing to spawn the subgoal "deploy new model robot capable of 1000 clips-per-second production but the AI has to have knowledge of how to go about that. Values act as weights for each decision, without them the most efficient path may be taken to achieving goals but not the most desirable.

The paperclip maximiser parable is all about values. But inappropriate goals and redefining of goals is an important discussion to. In fact in the game in the last phase where you've sent out Von Neumann probes you have to deal with "drift" in which some probe lineages disagree with the paperclip goal and you have to fight them. I've thought it for a while (and wrote it into one article that escapes me now) but it could be interesting to have an article detailing examples of goal/value drift in terragen history, famous examples and methods of dealing with it. One way that jumps to mind is to keep an up-to-date simm installed in every vot/bot to act as a prosthetic conscience. Of course then the values are only as good as the person the simm is based on, but at least it's something. Which gives me an interesting idea for a theocracy where the head of religion is judged so morally pure that their simm is installed for value judgements in the angelnet...

Thanks for the game suggestion, I'll check it out