Learning from Silly Assessments

Sometimes, you can learn about a serious matter by contemplating something silly. I have gained insights into serious career-related assessments by looking at frivolous assessments.

Almost every day, one or another of my Facebook friends is likely to post the results of a quiz that reveals such important insights as “What character in The Wizard of Oz are you?” or “Which state should you have been born in?” One quiz that appeared a few months ago even tried to reveal “What career should you be pursuing?” Call me a killjoy, but I am almost never tempted to take these silly assessments. I guess I worked at Educational Testing Service too long to be able to regard any assessment as harmless fun.

But I did make an exception for an assessment on Clickhole.com that lampoons these silly assessments: “Are You An Introvert, An Extrovert, Or A Sea Monster?” It offers items such as “What’s your signature look? (a) Bright and flashy. (b) Muted and professional. (c) A rippling shadow below the surface.”

In addition to poking fun at the proliferation of personality quizzes, this one points out a common failing of assessments: that the scale each item loads on is painfully obvious. In this example, it’s unmistakable that (a) loads on extroversion, (b) loads on introversion, and (c) loads on…well, you can guess.

The result of this transparency is that it’s easy for all but the most naïve test-takers to rig the answers. The first time I took a career assessment was in eighth grade; I was given one of the Kuder interest inventories. When it asked questions such as “Do you like adding columns of figures?” even a tweenie like me could tell that it was getting at my interest (or, in my case, lack of interest) in math. I knew I didn’t like math, so it was easy for me to make sure that the assessment results voiced this dislike loud and clear.

If you take the Clickhole.com assessment, you’ll notice something else that causes some assessments to be overly transparent: All of the items load the responses to (a), (b), and (c) according to the exact the same pattern as in the “signature look” example I cited above. Paper-and-pencil assessments almost universally follow this fixed method of arranging the options—in columns—to enable the test-taker to compute the scores simply by going down each column, counting the number of selections.

When an assessment is this transparent to the user, it wastes that person’s time. In eighth grade, I could have saved a lot of time and effort by simply rating 10 or so academic fields on a scale of one to ten—or by rank-ordering them.

To be sure, some very naïve clients may feel they benefit from this kind of assessment. These would have to be people with such sparse insights into their preferences that the assessment items are opaque to them; the scales the items load on are not obvious to them.

But there’s another major way in which career assessments fail. It’s the opposite situation: when the item-to-scale relationship is not at all obvious. That is, the test-taker fails to understand how his or her responses produced the end results. This kind of assessment is a black box, a fortune cookie. A sophisticated test-taker, rightly or wrongly, may be skeptical of the results from such an assessment.
To be sure, naïve test-takers (again) may feel they are benefiting from the assessment, in this case because of their blind faith in the accuracy of the results. But I do wonder how secure that faith is. I wonder especially how these test-takers will process any new self-insights that contradict the assessment results. Should they believe what the test told them or what they are now discovering about themselves?

That’s why I believe that the most useful way of assessing work-related preferences is by using informal methods based on introspection, such as making lists of enjoyable past experiences and drawing inferences from them. These methods definitely take more time and are vulnerable to the many distractions that our always-wired world throws at us. They also may be more difficult for clients who are naïve about their likes and dislikes, or those with limited inferential ability.

Finally, I realize that many people seeking career advice are simply lazy. “Just tell me what I should do,” they plead, and therefore they prefer a quickie assessment to the hard work of introspection and inference.
Nevertheless, because introspective and inferential methods require clients to do the work of prioritizing and inference themselves—and do not get their answers from a black box—the decision-makers buy into the results. And they can reinforce or refine their insights by factoring in additional information, such as their reactions to new experiences (say, job shadowing) or input from friends and family.