TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher than a group using Google and reference sites on three data science tasks. After ChatGPT was removed, researchers found no evidence that the consultants had gained technical knowledge they could apply on their own.
A randomized trial at Boston Consulting Group (BCG) found that consultants given ChatGPT scored higher than colleagues using Google and reference sites on three data science tasks, but showed no measurable advantage on a follow-up test without the chatbot. The study, published in Nature Human Behaviour on October 26, 2026, adds evidence to a question facing employers: whether AI assistance builds lasting job skills or mainly improves performance while the tool is available.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The trial involved 986 consultants allocated to treatment and control groups. According to the authors, 573 began the survey and 487 submitted work on both assigned tasks; that group formed the main analysis sample. Participants had at least a bachelor’s degree, and most reported no coding experience. The trial was preregistered on March 13, 2024, and data collection took place in late March and early April that year.
The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. The control group received guidance on using Google, Stack Overflow and Khan Academy. Each consultant completed two 90-minute tasks drawn from coding, statistics and prediction. The tasks were developed by BCG data scientists with OpenAI’s Economic Impacts Research team. The authors said ChatGPT could help participants, but could not correctly answer the questions independently.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. Scores were also compared with 44 BCG data scientists who completed the tasks without AI. Consultants using ChatGPT closed much of the gap on some tasks, but their average results remained below the data scientists’ average. Afterward, participants answered five technical questions without access to ChatGPT. The researchers found no statistically significant group differences on those questions; the joint test had a p-value of 0.531.
Task Gains Did Not Persist
The results separate doing a task with AI from being able to do similar work independently. For employers, that distinction affects how they evaluate training, assign work and check outputs. A worker who produces stronger results with a chatbot may be more productive in that setting, while still needing technical review when the tool is unavailable or its answer is wrong.
The study does not show that AI assistance has no workplace value. It reports substantial performance differences while participants had access to ChatGPT. Its narrower finding is that, under this trial’s conditions, the short intervention did not produce a detectable improvement on the later test without the tool. That matters for companies deciding whether AI use alone counts as employee development or evidence of independent expertise.
The authors also report that the ChatGPT group became more confident that the system could solve problems it could not actually solve. They describe this as an exploratory result. It points to a possible quality-control concern: confidence in an AI-assisted answer may exceed the user’s ability to recognize where the answer fails. The study does not establish how often that mismatch causes errors in routine business work.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Clicky Switches
Samsung Galaxy Tab A11 8.7-Inch Wi-Fi Tablet, 64GB, 4GB RAM, 90Hz, Gray, with 20W Charger
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
A Trial Built Around AI Limits
The trial was conducted at BCG in 2024, more than two years before the paper’s 2026 publication. Researchers compared ChatGPT with an active alternative: participants in the control group could use search and technical reference sites, rather than being asked to work without any support. This design tests the chatbot against tools workers might plausibly use to find help.
Participants completed two of three tasks: cleaning, merging and filtering datasets in Python; evaluating a statistical model and interpreting residual plots; or building a model related to the predictability of international football matches. The expert benchmark came from 44 BCG data scientists who did the same tasks without AI. Their average served as the reference point for task scores.
The paper’s authors characterize the effect as “temporary upskilling” and compare AI assistance to an exoskeleton: it can extend what a person can do while in use, without necessarily changing what that person can do alone. That framing reflects the study’s combination of task results and the follow-up test. It should be read within the limits of this single firm, the selected tasks and the brief training period.
““find no evidence””
— Wiles, Krayer and co-authors, in the study
Limits of the Follow-Up Evidence
The follow-up measured responses to five technical questions, not performance on a second full workplace project. The study therefore cannot establish whether longer training, repeated use, different teaching methods or other kinds of tasks would lead to durable learning. Nor does a result from BCG consultants necessarily describe workers in other industries or people with different levels of technical experience.
The source reports no statistically significant differences between groups on the no-ChatGPT questions, with estimated differences ranging from 7.7 percentage points below to 4.8 points above the control group and a joint test of p = 0.531. That finding is not proof that the groups were identical or that no participant acquired knowledge. It means the study did not detect an advantage for the ChatGPT group on that measure. The exploratory confidence result also needs cautious interpretation; the supplied study summary does not establish its broader effects on workplace decisions.
What Employers Should Track
The paper reports results from a completed trial; the supplied material does not identify a scheduled follow-up study or a next research milestone. The immediate question for employers is how to distinguish AI-assisted output from independent capability. Organizations using chatbots for technical work can track both the quality of work produced with the tool and whether employees can explain, check or reproduce key steps without it.
Further evidence would be needed to determine whether longer or more structured instruction changes the outcome, and whether the same pattern appears in other occupations and settings. Until then, the BCG findings support a limited conclusion: ChatGPT improved performance on these tasks while available, while the trial’s later test found no measurable knowledge advantage without it.
Key Questions
How much did ChatGPT improve task scores?
Compared with the group using Google and reference sites, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction.
Did consultants retain the skills without ChatGPT?
The researchers found no statistically significant group difference on a five-question technical test taken without ChatGPT. The result does not prove that nobody learned anything; it means the study did not detect an advantage for the ChatGPT group on that test.
Did ChatGPT users match BCG data scientists?
No. The ChatGPT group closed much of the score gap on some tasks, but its average performance remained below that of 44 BCG data scientists who completed the tasks without AI.
When did the trial take place?
The BCG trial’s data collection took place in late March and early April 2024. The paper was published in Nature Human Behaviour on October 26, 2026.
Source: Thorsten Meyer AI
