Spotlight for Recruiting Professionals

In her op-ed in the November issue of the NACE Journal, Mary Scott, founder of Scott Resource Group, writes that given the ubiquity of online survey and focus group tools, there are many organizations that offer commissioned and fee-based research, not to mention the widespread employer use of a DIY survey approach.

Scott offers several considerations for assessing the reliability and applicability of such data:

• What are the qualifications of those building the survey instruments?—A lack of expertise in writing questions or probing focus group responses can easily result in skewed findings. Effective research design and execution depend on the credentials and experience of those fielding the research. The tools simply provide a vehicle to gather data and do not replace subject matter expertise.
• Question the methodology—How, specifically, is the research fielded? Scratch the surface. Kick the tires. Are the participants consistent with your targeted students and sourcing channels? Remember the old IT adage: GIGO = Garbage In, Garbage Out.
• How are research participants sourced?—In addition to the “echo chamber” issue associated with a closed system, it is critical to understand how an organization is identifying research participants; having a randomized population that represents the greater population is, of course, the goal of valid research, but “randomized” does not mean “haphazard.”
• How is data integrity managed?—Is the survey instrument encrypted, or is it available on a public URL where anyone can access it? How are the data scrubbed? It is hardly surprising that some students submit bogus (and multiple) entries, especially when an incentive to participate is offered. Beware of “dirty data.”
• What are the credentials of those analyzing the data?—In their article titled “The role of expertise and judgment in a data-driven world” published by McKinsey Analytics, Rik Kirkland and Dan Wagner discuss the critical roles of both “nerds” and “experts” in formulating strategy based on research. Their point, simply stated, is that one needs to apply a healthy dose of skepticism that is rooted in expertise when analyzing metrics. (To quote: “Data is not a religion.”) One absent the other can very easily lead to flawed conclusions and ill-advised strategic direction-setting.