Can we engineer Collective Intelligence?

I have been following the developments in web 2.0 for some time now with keen interest and observing the unfolding of Collective Intelligence (CI). This phenomenon of CI, as you know, is the characteristics of a complex system to organize itself and appears to be more intelligent than its constituent parts. This has been in evidence in nature from the beginning of evolution.

The most interesting part of CI is its ability to come up with a totally unexpected and surprising outcome. Imagine, five years back, somebody predicting that we will shortly have an encyclopedia on the web freely accessible and richer in content than Britannica; where anybody in the world can contribute with no editorial control. I am sure it would have been considered as a laughable idea. Yet, today, we have Wikipedia.

Similarly, the success of Open Source is nothing but astonishing. Whoever would have believed, 10 years back, that the biggest threat for Microsoft Windows will come not from IBM, Apple, Sun or not from any new startup but from a large community of people who write code for an Operating System for no monetary gain. In fact, before the success of Linux, it would not have been thought possible for a very large community of people to write an operating system collaboratively without any central control.

However, if we look at nature, we should not be very surprised. Have you ever noticed the highly coordinated movements of flocks of birds or schools of fish. It is one of the most fascinating phenomena to be found in nature. The group seems to turn and maneuver as a single unit, changing direction almost instantaneously. There is no leader, no overall control; instead the flock's movements are determined by the moment-by-moment decisions of individual birds, following simple rules in response to interactions with their neighbors in the flock. There has been a fair amount of research into this area and many simulation models have been produced.

In the film Batman Returns a horde of large black bats swarmed through flooded tunnels into downtown Gotham. The bats were computer generated. Each bat was instructed to move about on its own on the screen following only a few simple rules: don't bump into another bat, keep up with your neighbors, and don't stray too far away. When the algorithmic bats were run, they flocked like real bats.

Ant colonies display similar intelligent behavior. Though an individual ant is a relatively simple organism, the colony has a highly structured social organization. As a result of this organization, ant colonies can accomplish complex tasks that in some cases far exceed the individual capabilities of a single ant. Their ability to locate food is nothing short of astonishing.

These phenomena have led to a new and novel researcher discipline called Swarm intelligence. Algorithms like the ant colony optimization algorithm are inspired by the behavior of ants in finding paths from the colony to food. Such algorithms are used to solve complex optimization problem like traveling salesman problem.

One common characteristic of the examples I have mentioned above is the surprise element each one of them posses. If you breakup each of the system into its constituent parts and analyze them, you will find it difficult to believe that combining them can produce such startlingly intelligent behavior.

Currently, we do not have a theoretical understanding of Collective Intelligence or Emerging Behavior of a complex system. The normal method we follow in understanding any large system is to break the system down to smaller, more manageable units and understand each of them separately. In case of CI, this approach will not work because in most cases, intelligent behavior will not emerge till a critical mass is reached. So, the individual elements will not tell us much about what the behavior of will be.

Of course there is always a possibility of a scientific breakthrough happening in our understanding of complex system. But till then we will not be able to either manage or engineer systems based on CI with any certainty. Any success will be pure chance. However, that should not stop us from taking advantage of any known or emerging CI behavior. That will be a subject of my future discussion.