The interview with the machine

A growing number of employers use software to sift applications before any person reads them. The attraction is obvious: a popular job can draw two thousand applicants, and a human cannot give each one the ten minutes fairness would require.

The difficulty is what such systems learn from. A model trained to recognise good candidates is usually trained on the company's past hiring decisions, which means it learns to reproduce them, including the parts nobody would defend out loud. One well-documented system penalised applications containing the word 'women's', as in 'women's chess club captain', because the historical data it learned from contained very few successful applicants with such phrases. Nobody wrote that rule. The system derived it, which is worse, because a written rule can be found and deleted.

There is a second problem that is less discussed. Systems of this kind are usually validated by asking whether they predict the same outcomes as the previous process. A system that agrees with past decisions scores well. But if the previous process was the thing you wanted to improve, agreement is not evidence of quality — it is evidence that nothing has changed except the speed.

Useful responses exist and none of them is purely technical. Train on outcomes rather than on past selections, if you can measure who turned out to do the job well. Test the system separately on groups it might disadvantage, before deployment rather than after. Keep a route by which a rejected person can reach a human, because every automated system needs an exception path and a system without one will eventually produce an injustice nobody can undo.