Matthew P. Wiener gives an argument as to why Turing Machine minds couldn't evolve to allow them to grasp something as powerful as ZFC.
> I claim that until the twentieth century, there was no possible selection > for larger theories. I expect that some random drift could occur, although > the original axioms are maintained by selection. I expect that obvious > contradictions go extinct or randomly pick some subsystem as their new > base. Deep contradictions become less deep with time, as new axioms pop > in, so they can't hide forever. > > Now a 0-1 law kicks in. We can ask what the probability that evolution > ends up with TM-minds that believe P for any P, and it should be 0 or 1. > The original theory can thus talk about the theory that consists of the > evolved probability 1 statements. These are consistent and definable in > the original TM-mind. So by the Goedel theorem, it's just the original > system!
I'm not confident in speculating about whether natural selection would favor ZFC-strength minds. However, I would like to point out that Penrose made the argument that natural selection would lead to minds that are beyond *any* algorithmic theory.
But anyway, I don't understand your point about randomness + selection never leading to anything stronger than the original theory. That seems wrong to me.
Here's a model for the "evolution" of a society of theorem-proving robots that I think would inevitably lead to theories more powerful than PA:
Each robot starts out believing PA. Every year or so, a surviving robot produces an offspring which is allowed to have one new axiom as a "mutation". These mutations are constrained to have the form "For all Phi(x)" where Phi(x) is a formula in the language of PA with one free variable that only has bounded quantifiers.
Besides producing offspring, there are two other things that can happen to a robot: (1) A robot can be "killed" by showing that its theory is inconsistent. (2) The robot can be temporarily "paralyzed" by a second robot. This happens when the second robot proves within PA that its own theory is more powerful than that of the first robot. While a robot is paralyzed, it is not allowed to reproduce. Paralysis lasts until the attacking robot is killed.
Now, I claim that in the above set-up, the thriving robots will tend to develop more and more powerful consistent theories, and that the weak and inconsistent robots will either be paralyzed or killed. Eventually, robots would develop that have a consistency strength greater than ZFC. (Of course, since I have restricted the language to arithmetic, ZFC itself can never develop.)
