The Problem of Expertise

I awoke in a daze. Instead of resting peacefully in my bed I found myself sitting, not unpleasantly,  in a circular room. Small lights blinked all around me. In front of me, a table with small artifacts scattered around, sitting on top of an unusual drawing or diagram. Confused, I stood and walked over to the table in an attempt to make sense of the underlying image.

As I stood, a door of sorts opened with a pneumatic hiss. Looking up, my heart jumped as I saw a tall grey humanoid step through the door. Suddenly, I felt as if I knew, this is a UFO that is an alien… I’ve been abducted.

The alien, I assume after noticing the look of shock and fear on my face, spoke perfectly intelligibly and calmly, “Hello Earthing, I am Zorf. There is no need to be afraid. We’ve simply brought you here to help us choose a new Horfing champion. As soon as we are done, you’ll be allowed to return home. If you like we can even help you forget this entire experience.”

“Umm… ok. First, what is Horfing?” I said, hoping to get on with the experiment and get back to bed (or wake from this dream?) as soon as possible.

Zorf made an expression that I could only understand as confusion. “Were you not just examining the state of play of the Horfing board on the table in front of you? It is our understanding that you are earth’s most prominent game theorist and decision-making expert. I have to wonder have we made a poor choice? Never the less, we must proceed now as the rules of the Horfing Consortium dictate.”

At this point two more aliens walked through the shhhushing door.
“May I introduce your two advisors? Zomog and Zefier.” said Zorg. “They are both are experts on Horfing. They have years of experience”

Advisors? Now I was thoroughly  confused, “Ok… I am going to go out on a limb, Horfing is a game? This image and the artifacts on top of it are the game board and pieces?”

“Yes, of course” said Zorf. Zomog and Zefier gave each other looks of concern.

“Ok… so I am here to choose between Zomog and Zefier as the new Horfing champion?”

Zorf snorted and looked extremely offended. “You are here to help us choose between myself and Zimmerger. As I stated Zomog and Zefier are experts and advisors.”

I felt good and lost now. “Ok. How can I help?”

Zorf made a pleasant smile “I need to decide our next moves in the game. I believe we are very close to winning. If I win you will of course be greatly rewarded.”

“Why don’t you ask the Zomog and Zefier?” I pondered aloud. “They are the experts”

Zorf gave me one of his “you aren’t as smart as I had hoped” looks. “I have consulted with Zomog and Zefier extensively, that is the problem. They don’t agree on the hurfing zeta six anomaly and its practical implications in this current context. Of course we could just wait, and the board will evolve itself as usual… but I feel certain that there is something we can do, I hate being passive.”

“OK you want me to help you decide how the hurfing zeta six anomaly applies to the current state of play, and Zomog and Zefier have opinions on that?”

“Yes that is roughly accurate. And I must run now… I am very busy setting up a Slingvort game and trying to get a group to focus on the outcomes and learnings of our Lingfrington match results after that.” Zorf turned quickly and marched out of the room.

I turned to the two remaining aliens, “Zomog and Zefier… how can I help?”

Zomog spoke first, “You must understand, Zorf appeared to be calm and reasonable, but, your choice between us will mean one of us will be sent to the Volgor Sinx Mines for 100s of years of hard labor.”

My only reply as I lay my head in my hands was “Oh, wonderful”

What followed was a mind boggling display of information (and I presume logic). Points were made passionately… using equations and maths with which I was completely unfamiliar. Artifacts on the board were pointed at vehemently, and areas of the board circumscribed with laser pointers. I hadn’t a clue what was happening.

In the end… I flipped a coin and told Zorf to follow Zomog’s advice.

—- The Problem of Expertise —-

In philosophy one of the problems illustrated in the story above is often thought of as the Problem of Expertise. This problem involves epistemology and our ability to transfer practical knowledge to others.

Frequently the problem of expertise is illustrated in a court room setting. The naive jury is presented with two experts on, say, blood splatter patterns. One expert is there to inform the jury based on the intentions of the defendant and the other expert represents the concerns of the prosecutor. As naive blood splatter analysts, the jury is asked to make a judgement on which of the expert stories to believe.

Of course when the experts agree, the jury needn’t make a decision it is only when they disagree that things become problematic.

To begin , let us accept that what cannot happen in general is to make the naive jury INTO blood spatter experts. This would take years of training to understand the subtleties of fluid dynamics and viscosity.  Blood splatter experts have SEEN many blood splatters in context. They have worked with blood-like substances to create blood splatters. While it would be fun to imagine the jury flinging blood-like substances around the court room, it is highly unlikely. Without practical experience with a theory and its outcomes, it is incredibly difficult for humans to make judgements on the theory’s validity.

The jury will be fooled if they are led to believe that what they have experienced and therefore can understand ( “I know what blood is”, “I have seen water splash on a wall” and “Sometimes when I cut a raw steak there are blood splatters”) will allow them to engage in the esoteric knowledge that is being described by the experts. The logic that follows from these experiences, through the expert’s experience, knowledge, and carefully considered argumentation, is not accessible to the novice. Jurors may feel as though they can understand the beginning and the end, but they cannot justifiably understand the relationships required to arrive at the conclusion of the expert.

Given that the jury cannot then engage experts in their pro-offered expertise, what ways do they have to make a judgement?

First, and likely least effective, they can examine the logic of the arguments of each expert. It is likely in a adversarial argument that the experts will attempt, when it is to their benefit, to point out holes in each other’s logic. This for the novice observer is the most likely to require “accepting” into argument, concepts and experiences they in fact do not understand, and therefore puts them at risk of misjudgment.

The next way one might judge an expert as a novice is via credential. Doctoral degrees in blood splatterology are,  in theory,  indicators of expertise or at minimum a judgements made by another expert as to the knowledge of the Doctor. There are problems with this as well. First, the certifying body could, in theory, represent or prefer one side of an argument. Second, any lawyer worth his weight will find an “equally credentialed” advocate for their side.

The jury might ask to have further experts come in to testify. They could make a judgement from here based on the agreement of the experts, and the number of experts who agree. This has the problem of a arms race, wherein we simply produce more and more experts to repeat our position as clearly and coherently as possible. This is the classic, “stay on message” silliness we get from politicians and seems to assume that, upon repetition, truth will materialize.

The jury may also want to understand the expert’s actual history in judging blood splatters. In general, is this expert’s opinion correct? While past performance is not a guarantee of future outcomes, we are running out of ways to make a decision pretty quickly here…

One final way an naive jury might try to suss out who to believe is to ask, which expert would have the most to gain by deceiving us? Is there a bias involved? Is the expert arguing specifically for this case, or will he benefit from “being right” in the future (take a moment to reflect on the “past performance” judgement above).

In practice, what many experts realize is, the best way to win this argument is to be good at arguing. This means, appearing confident, polished and quick to answer questions with firm statements. These qualities make the expert appear as though they have thought through the argument thoroughly… of course the naive jury has no way to judge the validity of any of the responses. Another way to describe someone who appears polished, smooth and confident is… a con-man. Experts can learn it is more important to establish blind trust, than it is to establish valid arguments.

In the story above, lacking any way to judge the arguments, I flip a coin. In decision theory, it would be my guess, in this abstract case a coin flip would produce the most equitable result and anything more would simply be meddling in the fate of Zomog and Zefier.

Why would any of this matter?

Have you ever had to make a decision between two DBA’s esoteric database optimizations (assuming you aren’t a DBA) or maybe you’ve needed to help two architects “come to an agreement,” maybe it is as simple as knowing which PO’s vision to invest in next?

In my opinion, this problem of expertise is one that plagues modern Cognitive Cultural Economies. We can all come to agreements rather quickly on tacit problems in front of us. “Is this screw within tolerance” seems to be a simple question to answer, assuming we know the tolerances. However, due to the increasing amount of work that is performed intangibly,  thinking, argumentation, collaboration, and deep expertise (the kind we achieve by engaging directly in the work) are becoming more and more critical. How should we design this widget and even more critically, what kind of widget should we design, these questions are being asked far more frequently.

As managers we value experts for their ability to use this expertise to make reasonably justifiable recommendations of goals and actions to achieve goals. Experts do this by combining past experience (some might say 10,000 hours) with current evidence to produce recommendations. Past experience tells the experts at least 2 things, what is important to observe (what is valid evidence) and what actions are likely to change the balance of what is important to observe. In this way, there is a double trust on the part of the manager (read novice) that the expert has chosen the right goal, and that the actions the expert has recommended are likely to help achieve that goal.

The expert problem, isn’t just “how” it is often “what”…

All of this places modern managers smack in the middle of the Problem of Expertise. If the experts we are working with come to loggerheads, can our decisions be any better than a coin toss? If they aren’t, or… if they are only marginally better than 50/50… how might we go about minimizing the risk that we have made the wrong choice? Finally… if we are constantly trying to devolve decision making, to solve this problem… aren’t there likely to be more decisions to be made, increasing the likelihood of the expert problem?…

I’d love your thoughts… I’ll share more of mine soon.