My first company, The Learning Consultants, is an educational consulting company. While we help students and families with all educational issues, the marketplace dictated that SAT prep and other high stakes tests become one of our main areas.
I am an idealist by nature but a pragmatist in the way that I go about ensuring the ideals of my clients become reality. For that reason, my response to criticism of the SATs or anything else that doesn’t suit idealists has always been “it is what it is” and then I explain that “if you want to reach certain objectives, you simply have to do what has been presented to you.”
I have the same response here for those who will lament the use of testing for careers and jobs.
Fortunately, given our work with both career counseling and aptitude testing, we can help if you are faced with taking a test for a job application or job advancement.
This article is from The Atlantic
The private-equity firm Vista Equity Partners owns a sprawling portfolio of technology companies, from Automated Insights to the Xactly Corporation. Together, they provide software to just about every major industry. Since its founding, in 2000, Vista has engineered a nearly unbroken string of profitable acquisitions, a run of success that has made Robert Smith—a co-founder, and its chairman and CEO—a very wealthy man. At $4.4 billion, his estimated net worth makes him the richest African American, just ahead of Oprah Winfrey.
Smith is not the household name that Oprah is, and he seems to prefer it that way. He rarely gives interviews—the recent cover story of Forbes’s “World’s Richest People” issue was an exception—and he guards Vista’s secrets closely. On one subject, however, Smith has been relatively open, even chatty: the firm’s knack for hiring and promoting the right people. Vista’s marketing materials tout its “talent and performance management,” a key element of which is a kind of assessment many people find harrowing. Applicants to Vista companies, from the entry to the senior-executive levels, are subjected to a timed standardized test.
Testing, Smith says, helps his companies find talented people—people the competition has overlooked because their résumé lacked certain credentials or because of the inherent biases of managers. Smith describes Vista as a pure meritocracy, where high performers succeed regardless of their background, race, or gender. He likes to tell rags-to-riches stories: senior employees who began as a mail-room worker, a roofer, a shelf stocker.
Vista is an especially enthusiastic proponent of testing in the workplace, but it is not alone. Testing is sufficiently widespread today that a cottage industry has developed to help people prepare; perusing the site of one such firm, JobTestPrep, I saw sample exams for jobs at Amazon, Ford, UPS, and others. Like Vista, many companies now subject not only entry-level applicants but also managers to testing. Criteria Corp., an employment-testing company, says its Criteria Cognitive Aptitude Test has been administered to more than 2 million people, many of them current and prospective managers. The company describes its tests as “especially useful for mid- and higher-level jobs.” Mettl, another testing firm, has more than 1,500 corporate clients globally, including giants such as Accenture and Microsoft. It boasts that it can help identify “top talent” through a battery of exams, such as the Mettl Cognitive Abilities Assessment. I recently texted a friend from college, a 50-year-old former chief marketing officer who had switched jobs recently, to ask whether he had taken this kind of test. “For my last three permanent employers,” he replied.
Research from the “people analytics” movement holds that firms can use objective assessment tools—not just old-fashioned intuition—to hire and develop employees. Cognitive testing of executives may be at the bleeding edge of this focus on analytics, but it also harks back to the middle of the last century, when employees up and down corporate organizational charts were routinely given IQ tests. In 1956, the business journalist William Whyte reported in his classic study The Organization Man that about a quarter of U.S. corporations were using tests to evaluate managers and junior executives.
Indeed, when Robert Smith describes his management philosophy, it sounds less like the foosball-and-free-snacks stereotype of software start-ups than something out of the dog-eat-dog corporate culture of old. Smith advises Vista managers to be constantly on the lookout for their worst performers. “Here’s how this works: Our ship goes down. We’re all in a lifeboat, and we’ve got 11 people on it.” He then pauses dramatically. “Who’s the first one you throw overboard? Who’s the second? Who’s the third?”
If the Vista approach continues to gain favor—and when someone makes as much money as Smith is making, the competition tends to notice—you may soon be asked to take a timed standardized test, whether you’re fresh out of school or an executive trying to make a mid-career move. Your spot on the lifeboat could be at stake.
Vista treats its test as proprietary, and declined to discuss it with me or let me take it. Forbes reported that the hour-long test “assesses technical and social skills, and attempts to gauge analytical and leadership potential.” Reports from test takers suggest that the cognitive component of Vista’s test differs considerably from college-admission tests. Unlike the SAT and ACT, which measure specific knowledge (word usage, algebra) and skills (reading comprehension), many new workplace tests are designed to measure raw cognitive ability.
The Criteria Cognitive Aptitude Test, for instance, measures problem-solving, critical thinking, learning ability, and attention to detail. You have only 15 minutes to answer 50 questions—or about 18 seconds per question. The average person gets fewer than half correct.
I signed up with JobTestPrep and took a sample of the Ccat online. (Criteria cautioned me that although the style of sample questions is likely similar to actual Ccat questions, the company does not endorse the sample exam.) Before I clicked on the red button for “Timed Test,” I had a flashback to the cramped classroom at the University of Kansas where I’d taken the Law School Admissions Test. I’d always done well on standardized tests, but it had been decades since the day I’d taken the Lsat, and I was feeling rusty.
The first question—“Select the next number in the pattern 2, 6, 10, 30, 34”—was a little tricky, but I was able to deduce the correct response from the four multiple-choice options. (The answer is 102: If you multiply 2 by 3 to get 6, and 10 by 3 to get 30, then you must multiply 34 by 3.) The second question was harder for me: What goes in the empty box of this three-by-three grid? (See image.)
Which of these choices is best? I felt the seconds ticking away. They all have smiley faces. Do the black stars follow a pattern? What about the horizontal lines? My mind wandered. Eventually, I picked the correct answer: C. (The key: realizing the black stars rotate clockwise as you move from left to right within each row.)
It might seem far-fetched to believe that an employer could glean much information from such questions, especially when they’re asked under extreme time pressure. Could my ability to write software code, or manage a team of coders, really be predicted by a test like this?
Shane Frederick, a professor at the Yale School of Management who studies cognitive testing, has found that the answers to a very small number of well-framed questions can, in fact, be highly informative. Frederick told me he can predict performance on a wide array of important tasks by asking just one question. His research has established that certain questions—“If a bat and ball cost $1.10 total and the bat costs $1.00 more than the ball, how much does the ball cost?”—are reliable measures of cognitive reflection, or the ability to overcome an incorrect gut response. Cognitive reflection, in turn, is often correlated with key employee traits, such as attentiveness. Frederick believes that cognitive ability is rarely the only requirement for a white-collar job, but he is unequivocal about the value of testing for it, especially as work seems to grow more complex in every industry. “If a job requires intelligence, you should be using these kinds of questions,” he told me.
Hiring, of course, isn’t just a matter of finding the right people, but weeding out the wrong ones. According to Forbes, last year Vista companies administered 850,000 tests to hire just 6,000 people. Retired Navy seal Rear Admiral Alex Krongard, now an investment banker, told me that timed exams can be especially informative when an employer wants to eliminate candidates who won’t thrive in trying conditions. The software business is not the Navy, but a programmer or sales rep may be called on to perform under pressure. Administering a difficult timed test is roughly analogous to observing a seal candidate swim while bound hand and foot in water—you see whether someone can remain calm and function under stress. Krongard points out that the Navy has never really been able to predict who will complete the arduous process of becoming a seal, but “they have been able to use testing to determine who will most likely not make it through training.”
Testing’s power to weed people out is one reason that it lost favor, however. In the middle of the last century, courts found that employers were using discriminatory tests to keep racial minorities out of the workplace. The cognitive tests that firms are using today purport to avoid such biases; Mettl, for instance, says it rejects questions that rely on topics or language that would be “unequally familiar to different groups.”
Many companies also attempt to mitigate bias by assessing other characteristics, including technical and social skills, not intelligence alone. The new tests could nevertheless prompt legal challenges from another protected class: older people who are no longer at their cognitive apex. (Research suggests that some aspects of cognitive performance peak early in life.) Technology firms might prefer to have younger, sharper workers, but federal law bars discrimination based on age.
Another reason the mid-century vogue for testing came to an end: The tests just weren’t effective. William Whyte once persuaded a group of corporate presidents to take some of the assessments popular at the time. None of the executives scored high enough to be hired by their own company. Are today’s tests any better? Some evidence indicates that the new regimens might have improved. In a recent working paper, the business-school professors Steven Kaplan and Morten Sorensen looked at more than 2,600 executive assessments by ghSMART, a leadership-consulting company. The assessments are more subjective than the Ccat, but they do include numerical rankings of the executives’ skills, as gleaned from a wide-ranging four-hour interview. Kaplan and Sorensen found that “general ability,” which includes brainpower and analytic skills, is a significant factor in predicting who will become a chief executive officer.
Some executives I spoke with, however, felt that the most-important skills in the C-suite aren’t ones that register on tests of cognitive ability. Stephen Tomlin, a managing partner at the venture-capital firm Avalon Ventures, told me that no 15- or 30-minute test can capture the qualities of successful business leaders, particularly at technology companies. Tomlin described two kinds of “intelligence” that he wants in a leader. The first is heavily domain-specific, meaning that a candidate knows “a ton about their industry, the underlying technologies and their possibilities, and their present and potential competitors.” The second is emotional: “a healthy understanding of manners and ethical behavior (so as not to be a complete jerk in the boardroom or to his or her teammates, or to get everyone sued). Raw cognitive processing power seems like a distant seventh or eighth in the kinds of things I would be screening for in a CEO.”
For the moment, Tomlin’s view remains more common than that of Vista’s Robert Smith. If you’re considering a mid-career move, knowledge of your sector and impressive emotional intelligence (which peaks much later in life) are more important than solving a bunch of brainteasers. Can’t hurt to ace the test, though. The bat costs $1.05. The ball costs a nickel.