Philosophers have long attempted to define what it means to make rational decisions, but in the context of machine intelligence, these theories turn out to have undesirable consequences. Can we find a formal theory of 'rationality' that we would want a superintelligence to use?