In response to my comment last week about the potential usefulness of stereotyping, David posted an insightful observation. He says that not only are stereotypes useful, but that they are necessary. His comment goes to the heart of some very serious issues that are the reason I write these columns.

The success of human beings in building civilizations and a formidable knowledge base is largely dependent on our amazing ability to see patterns in data. At the best, these patterns give us useful insight into the operation of the universe. At the worst, they lead us astray into the regions of prejudice and unwarranted bias. All ancient cultures looked at the nearly random pattern of stars in the heavens and organized them into related constellations even though the stars that appear to be related are widely separated and the organization is only an artifact of the location of our observing station. That is an example of a relatively harmless and potentially useful extraction of patterns where no meaningful pattern really exists. Naming the constellations is useful because it is a convenient way to compress and organize observations that can aid navigation, calendar monitoring, and timekeeping. I do not know of any instance in which the naming of constellations was harmful.

Harm has been done by the offspring of these early attempts to organize astronomy because the pattern extraction was extended to form what we now know as astrology — the false imputing of patterns of behavior based on the patterns of celestial events. This substitution of false attribution for thinking no longer seems to cause serious problems like deciding when to fight a battle, but it still results in much waste (which can be partially written off as entertainment). More people know their birth signs than can point out Jupiter in the night sky.

When attempting to survive in a primitive and hostile environment, one is well served by being able to sense the pattern of an approaching predator before the predator can strike. Evolution favors beings who can extract meaningful patterns from environmental sensory data. The cost of false alarms is less than the cost of missed events. The reward for correctly predicting an attack is continued life. But most of us no longer fear being eaten by cave bears or other animals significantly meaner than us. So we are free to look for patterns in non-threatening places. We can go back outside and look at the night sky and see the patterns of planetary motion. From this patterns extraction, we can study the effects of gravity and eventually work out galactic dynamics.

I suspect the original use of term "stereotyping" included this type of pattern extraction without the negative connotations the word now carries. Remember my quote from the dictionary. The first definition indicates a stereotype is a faithful reproduction. The second definition included the negative overtones. These days we label actions as "stereotyping" when we want to discredit some conclusion based on what we feel is spurious interpretation of data.

Pattern extraction is good. Stereotyping is bad. But what about stereotyping’s educated brother, "profiling?" Is profiling bad? If you are hassled by an officer simply because you happen to be of a certain minority, you certainly think it is bad. But when you look at your fellow passengers lining up to board an airplane, do you really think that no one has a right to assume swarthy men furtively speaking in foreign accents are no more likely to be terrorists then an eighty-year-old woman who looks slightly disoriented with her boarding pass in hand? Either could be at terrorist, but which is more likely? We profess not to be prejudiced against swarthy men, but…

Life is difficult, and I do not know the answers. That is why I study decision theory and statistics. How do we best make decisions based on sensory data? Is there a better way than relying exclusively on informal stereotyping or more formal profiling? The best answer is "sometimes." Both profiling and stereotyping are useful tools. Like all tools, they can be misused. That misuse is the fault of the user, not the tool.

In response to the interest my original tutorial generated, I have completely rewritten and expanded it. Check out the tutorial availability through Lockergnome. The new version is over 100 pages long with chapters that alternate between discussion of the theoretical aspects and puzzles just for the fun of it. Puzzle lovers will be glad to know that I included an answers section that includes discussions as to why the answer is correct and how it was obtained. Most of the material has appeared in these columns, but some is new. Most of the discussions are expanded compared to what they were in the original column format.