The wisdom of crowds hinges on the independence and diversity of their members’ information and approach. Here I explore how the wisdom of scientific, technological and business crowds for sustained discovery and invention operate through a process of collective abduction—the collision of deduction and induction—wherein unexpected findings stimulate innovators to forge new insights to make the surprising unsurprising. Drawing on tens of millions of research papers and patents across the life sciences, physical sciences and patented inventions, here I show that surprising designs and discoveries are the best predictor of outsized success and that surprising advances systematically emerge across, rather than within researchers or teams; most commonly when innovators from one field surprisingly publish problem-solving results to an audience in a distant and diverse other. This scales insights from my prior work that shows how across innovators, teams and fields, connection and conformity is associated with reduced replication and impeded innovation. Using these principles, I simulate processes of scientific and technological search to demonstrate the relationship between crowded fields and constrained collective inferences, and I illustrate how inverting the traditional artificial intelligence approach to avoid rather than mimic human search enables the design of diversity that systematically violates established field boundaries and is associated with marked success of innovation predictions. I conclude with a discussion of prospects and challenges in a connected age for sustainable innovation through the design and preservation of difference.