This question has fascinated me all my life. The best explanation I have come across is the theory of ‘Bisociation’ described in his book, The Act of Creation, by Arthur Koestler. Bisociation, he suggests, is when two quite different ideas become forced together in one mind by one circumstance or another. Usually, Koestler maintains, one of the ideas is a problem that someone has kept in his mind week after week, month after month or perhaps year after year. And then, one day, the long resident problem in one person’s mind becomes accidentally associated with a random perception of event.
The most famous problem that’s usually mentioned by everybody who writes books of creativity is that of how the benzene molecule of six carbon atoms could hold themselves in a ring. Hundreds of chemists in countries all over Europe puzzled over this problem in the 19th century. It was an important problem because benzene was the source of many of the dyes which manufacturers use when designing new clothes and fabrics. One day, August Kekule fell asleep in front of his fireplace. When he woke up he could see flames leaping about and seemingly joining together head to toe — almost like being in a ring. And suddenly, Kekule’s problem was solved. The molecule of benzene is a ring of six carbon atoms.
The secret of significant new ideas is not so much the intelligence of their creators, but of individuals hanging onto problems fro as long as it takes another random idea, perception or events to come along and join forces.