Of course, we know that Fermat did NOT "have a simple proof". We know this because after writing that note, in the margin of a book, so not intended for public knowledge, he published separate proofs of the n= 3 and n= 4 cases. If he HAD a general proof he would not have done that.
What happened to Fermat is exactly what happens to mathematicians all the time. He looked at the problem, thought that he saw a simple proof and made a brief note to remind himself. Later he tried to work out that proof, in depth, but found it did not work.
People keep looking for easy proofs because the equation looks so simple and they have an inflated ego.