What the Data Is Evidence Of

Around 2014, a team at Amazon began building a tool that would score job candidates from one to five stars, the way shoppers rate products. To teach it, they gave it ten years of resumes submitted to the company, along with the hiring outcomes that went with them.

Most of those resumes came from men, a reflection of the industry. The model learned what the data actually contained: a pattern of who had been hired, not who would have been good at the job. According to Reuters, it learned to mark down resumes that included the word "women's," as in "women's chess club captain," and to downgrade graduates of two all-women's colleges.

Engineers edited the tool to treat those terms neutrally. That was no guarantee the model would not find other routes to the same pattern. By early 2017, Reuters reported, executives had lost hope for the project and the team was disbanded. Amazon has said the tool was only used in a trial and that recruiters never relied on its rankings alone.

The sign was "past hiring decisions." The thing was "ability to do the job." A model can only learn the sign. Closing the gap is the leader's work, and it happens before training starts, in deciding what the data is evidence of.

What is your organization's historical data actually evidence of?