Recent research from MIT scientists compared the brain activity of students who used ChatGPT when writing to those who did not. The team’s findings suggest that using ChatGPT resulted in less brain activity and inferior writing for students.
The results were published with the provocative title ”Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task” in preprint form, meaning these have not been peer-reviewed or officially published by a scientific journal. Even so, the study has garnered a great deal of attention in the education world and beyond with write-ups about it appearing in Time Magazine, Psychology Today, and elsewhere.
All this coverage has led to some misconceptions and exaggerations about its findings, says Nataliya Kosmyna, a research scientist at MIT Media Lab’s Fluid Interfaces group and lead author of the study.
“We didn’t find any brain rot,” Kosmyna says. “Some people thought it was IQ measurement or something like that, but we didn’t measure IQ in our study.”
Kosmyna explains exactly how the study was conducted, what it found, what its implications are for education, as well as its limitations and some of the pressing questions related to it, that require more research.
What Did The Your Brain On ChatGPT Study Find?
Researchers split 54 study participants, who ranged in age from 18 to 39, into three groups and had them write a short paper. Participants in one group were placed in a large language model (LLM) group and asked to use ChatGPT for help. The second group could use a search engine, while the third group had to write their essays without the use of external tools.
Participants’ brain activity was studied with EEG devices and the results did not paint a flattering picture of AI use. The researchers measured brain connectivity, essentially communication happening between different regions of the brain, Kosmyna says, adding, “The ChatGPT group had the least brain connectivity.”
This is not necessarily surprising, given that we intuitively notice that actually writing a paper ourselves takes more mental effort than copying and pasting info from an AI chatbot. But the cognitive impact seemed to extend beyond the initial experience.
For the conclusion of the experiment, those who were initially in the LLM group were asked to write the essay without help—these participants still exhibited less brain connectivity than participants who had never interacted with the LLM in the study.
If this result holds up in further research, it could have major impacts for when and how AI tools are introduced in the classroom.
“The timing might be absolutely critical when you introduce the tools,” Kosmyna says. She adds that it could make sense for educators to have students learn a skill such as essay writing without the use of AI, and then introduce AI as a tool later once students have achieved a certain level of mastery.
What Does This All Mean In Real-World Settings?
These results should be interpreted with some caution, as judging the implications of brain activity is notoriously difficult, and some education studies in the past have been criticized for making overly broad claims based upon brain scans.
The MIT team also assessed student output by having two human writing teachers score all the papers written for the study.
“They did not know that those participants were from an LLM group, but they called essays from the LLM participants ‘soulless,’ that’s a direct quote,” says Kosmyna.
The papers submitted by LLM participants also exhibited less stylistic and word choice diversity, and were considered average by the human graders, who also wondered if they had been written by the same student.
Because of the way AI is being integrated at all levels of society with what Kosmyna says is “unprecedented speed,” she believes educators should push for research into the impact of AI use in their students, both from researchers at institutions such as MIT, but also in their own districts and settings.
“Nothing prevents a school from providing a class with an LLM and then having [the same] class without an LLM, and just comparing the results,” she says.
Where Is More Research Needed?
This research only looked at a limited population from a similar demographic. It also only looked at the impact of ChatGPT on a specific skill, writing.
Kosmyna has completed follow-up research that looks at how ChatGPT impacts brain activity and quality when students engage in a coding and programming activity. Though the study has not been published yet, she says it trends in the same direction but with variation. “Of course, they are different because the nature of the task is very different here,” she adds.
Future research into AI use needs to look at its influence on extended use and among different age groups, particularly teenagers and children.
“I would say it is almost critical to do it on younger teenagers and on children, on minors,” Kosmyna says. “The brain that just starts to learn is potentially in bigger danger.”
