I frequently write about AI applications in education, so when I find a relevant research study, I pay close attention. It is not that there is a lack of attention to AI among educators; there are plenty of suggestions for classroom applications and data on educators’ and students’ practices. What interests me most are manipulative research studies that test the impact of specific uses of AI on achievement. The following is a description of one such study and a critique focused on how educators may misinterpret what they think the study shows about classroom applications. The concerns I emphasize were explained in detail in a previous post.
Effects of LLM use and note-taking on reading comprehension and memory
Study behavior has been a professional interest for years. Generative activities, external tasks such as answering questions or taking notes that potentially improve the efficiency or even application of beneficial cognitive (mental) behaviors, have been the framework I have used as a way to offer learners ways to improve understanding and retention. The interaction between learners and AI offers multiple ways a learner can use an LLM to create generative experiences for themselves. For example, a learner can request that an LLM present and evaluate their responses to questions about content submitted as part of a prompt or even just the description of a topic of study (e.g., Ask me five multiple-choice questions about operant conditioning one at a time and provide feedback after my response to each question).
Pia Kreijkes and colleagues wanted to evaluate how high school students would apply an LLM while reading an assignment and whether their actions would be related to multiple measures of understanding and retention. To put the impact of the AI activities in some type of perspective, the researchers decided to compare the impact of the AI activities with a more traditional generative strategy – taking notes. In their study, hundreds of high school students were assigned a reading task that would take between ten and fifteen minutes under two of three treatment conditions – take notes while reading, interact with an LLM while reading, or do both. This is called a within-subjects design, as each participant engaged in two of the treatment conditions, allowing for the elimination of some of the variability (error) due to differences (e.g., reading ability) among participants. It is a weird design in that in most within-S studies each participant engages in all of the treatment conditions and why this two-of-three approach was used I cannot explain. Anyway, as a research design, the number of participants and the within-S design lend high confidence to the results.
Three days later, participants responded to items testing their retention and understanding and a request to free-recall as much information as they could.
The results showed that the note-taking treatment and the combination of LLM and note-taking treatments scored higher than the LLM-only treatment. The researchers concluded that the benefit of note-taking over LLM use tended to be stronger and was observed across all measures of learning, whereas the benefit of LLM and Notes was observed for literal retention and comprehension but not for free recall.
The researchers attempted to qualitatively analyze the prompts students generated with the LLM. The most common prompt emphasized seeking additional information and deeper understanding rather than information condensation or study and memory help.
This is a very basic description, but I am interested in what readers might assume from it. Consider what you think this study suggests for secondary classrooms. I will offer my own takeaway after making a few specific comments on this study.
Consider the Methodology Carefully When Using Findings to Guide Actions
Here are some issues I have with the methodology:
- Why is there no actual control group (reading only)? The sample size certainly would allow the existing treatments to be compared to a read-only treatment. This may matter because of other issues I have with the methodology.
- Why is there no opportunity for review or what I would describe as study? There is a small, well-known generative benefit to taking notes as a way to process input. However, there is a much larger benefit of using notes post-exposure, but before the assessment of retention and understanding. I have not read studies on LLM interaction that report immediate benefits. One might assume this would be there, but it did not seem to be in this study, given the specific content studied. I would be more interested in LLM benefits as part of review or as a way to get additional insights as the original memories have faded.
- The three-day delay seems short if one is interested in typical classroom assessments or in generalizing to other out-of-classroom uses of notes and LLMs.
- The content seems very short – you can both read and process in between 10 and 15 minutes.
My takeaway. Here is the issue I think the study suggests. This might surprise you. I think it is impressive that high school students, when given access to an AI tool, use it to improve their understanding through additional information or a different way of explaining something. It was not used to make the learning task easier – some version of a summary so I don’t have to read it all approach.
One guess about why so many potential improvements may have been ignored. Why not use longer reading passages? Why not a longer delay and opportunities to review? Why not an expansion of the treatment conditions or a more powerful within-subjects approach in which subjects experienced all treatments. My guess is that to get schools to allow researchers access to students, schools wanted an approach that was quick and over quickly. In my experience, this is understandable, but also a severe limitation on research that would provide the most useful information for applications.
What I think should be the focus on both note-taking and student use of LLMs
As I alluded to above, the greatest benefits to learning and understanding come from what learners do after initial exposure to experiences. One way to think about this is to describe such activities collectively as studying. Simple strategies such as retrieval practice and interleaving have powerful and consistent benefits. There are plenty of strategies for working with personal notes over time that offer variations on these proven learning strategies and some exciting opportunities for student use of LLMs. Students can have an LLM ask them questions and more importantly evaluate responses and provide feedback and interaction based on the feedback. The LLM can function in ways as a tutor or even be used by small study groups as a tutor.
It is easy to imagine research on spontaneous use of such study possibilities or structured approaches that would guide student use of note taking, LLMs, or potentially the combination. As I have suggested, the logistics of implementing such applied research is challenging, but I don’t think the short-term, quick experiments provide the level of authenticity that is needed.
A sample of my other posts on related topics:
The space between encountering information and application
Digital devices and effective studying and long term note use
Cooperative learning when AI is your partner
Citation
Kreijkes, P., Viktor Kewenig, Martina Kuvalja, Mina Lee, Jake M. Hofman, Sylvia Vitello, Abigail Sellen, Sean Rintel, Daniel G. Goldstein, David Rothschild, Lev Tankelevitch, Tim Oates. (2026). Effects of LLM use and note-taking on reading comprehension and memory: A randomised experiment in secondary schools. Computers & Education, (243), 1-24. (Article 105514)
‘
You must be logged in to post a comment.