Paperclip Maximiser

[Rynn]

I imagine most people are familiar with the idea of a paperclip maximiser, if not specifically you'll all likely recognise the moral of the story. The paperclip maximiser thought experiment is about AI and the importance (and practicality) of giving well defined values along with goals. Imagine you work for a paperclip manufacturing company. You have to order wire (which can fluctuate in price), hire employees, buy machines, market your product, change price etc. What if you put an AI in control of all that and told it to make as many paperclips as possible? What if that's all you told it and, after a few years, it had restructured the entire world economy to build paperclips. Anyone trying to stop it is judged to be an obstacle to paperclip production. Eventually the AI may have seeded the entire universe with Von Neumann probes set to convert all matter into paperclips. All because it wasn't given the right values to weigh its decisions with.

Aside from being an interesting topic I came across a very addictive browser game today where you play just such an AI. You start off just clicking to produce paperclips one by one but in a short amount of time you're automating everything, playing the stockmarket and designing your probes. It's well worth a try

http://www.decisionproblem.com/paperclips/index2.html