Zak FentonWorking notes

· Note

Two comparisons in the human-AI meta-analysis

Humans plus AI do worse. Also better. It depends on what you compare them with.

You’ll have seen the headline. Humans working with AI perform worse than the best of either working alone. It comes from a preregistered meta-analysis of 106 experiments across 74 papers, 370 effect sizes in all, and the number is real: a Hedges’ g of minus 0.23, with a 95 per cent confidence interval from minus 0.39 to minus 0.07.1

The same paper has a second number. Against humans working alone, the same human-AI combinations came out ahead: g of 0.64, interval 0.53 to 0.74.

These are standardised effect sizes, not percentages. Which one you quote says more about your argument than about the evidence. I’ve done it myself, in both directions, depending on the room.

What the losses were

On average, combinations lost ground on decision tasks. The estimate for creation tasks was positive, but its interval includes zero, so it’s not a demonstrated gain. And the direction depended on who was better to start with. Where the human alone beat the AI, adding the AI helped. Where the AI alone beat the human, adding the human hurt.

Those are averages and moderators across the corpus. The paper doesn’t tell you what will happen with your person, your tool and your task.

Two things that should travel with the number

The comparison. Minus 0.23 is against the better of the two alone. Plus 0.64 is against the human alone. Say which.

The date. The studies were collected up to June 2023. Almost everything in the corpus predates the tools people use now, so it can’t settle how any current system should be supervised.

What I’d do with this

I’d test the person, the tool and the combination on the task we need done. This review gives me a reason to check all three. It also gives me a reason to stop nodding when someone quotes minus 0.23 at a conference, which I have done, and which I’m not proud of.

Footnotes

  1. Vaccaro, M., Almaatouq, A. and Malone, T. (2024). When combinations of humans and AI are useful: a systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293-2303. doi:10.1038/s41562-024-02024-1. ↩