Using AI to Analyze the Limitations of a Research Methodology

Posted on October 11, 2026 by admin

Because I commit to reading original research papers, specifically the Methodology sections, and taking notes on them before writing for practitioners, I have always felt I have an advantage over those who read summaries of research or use AI prompts to make statements about what research suggests for educational practice. As an example I am going to use here, my critical reactions to those who promote taking lecture notes by hand rather than using a digital device stems from this focus on interpreting research conclusions after reviewing what the methodology requires and allows.

I wondered whether large language models would analyze research studies in the same way. To understand questions about the analytical, rather than summarization, skills of large language models, I decided to see what I could accomplish by including a specific prompt asking the AI tool to consider the research methodology when summarizing suggestions for practice and identifying limitations in the research design. One AI subscription I use is Recall.AI. It integrates PDFs, summarizes them at controllable levels of detail, and lets you query the sources. You can enter your own notes, but I prefer Obsidian for that purpose. This set of circumstances should allow a test of whether an LLM could use an “understanding” of the methodology to set limits on what it could propose based on the collected data. 

One of the most cited studies proposing that students would be better off taking notes by hand is Mueller and Oppenheimer’s “The pen is mightier than the keyboard.” I first applied the following prompt to this PDF, then to all PDFs in my collection, comparing handwritten and keyboarded notes.

Prompt: It is essential to consider the methodology used to collect data when suggesting what the data suggest for practice. Does the methodology of Mueller & Oppenheimer comparing the impact of notes taken by hand and using a keyboard provide clear guidelines for how college students should take notes in lecture classes to prepare for examinations?

Here is what I think my small experiment demonstrates, followed by an Appendix that includes the response to the prompt for the single study and a collection of studies for those who want to review the outputs themselves. 

My issue with the Mueller and Oppenheimer study and most of the studies of this type is that the studies are more about the benefits of handwriting as a method of encoding, ignoring the reality that student note-taking college students take lecture notes to have content they can eventually use to study for an exam that may be a month or more in the future. Researchers distinguish encoding from external storage, acknowledging that external storage (studying) accounts for more variability in test performance.

The methodology in much of the research is not really an evaluation of the value of external storage that is acknowledged in what the AI analysis points out in the limitations of Mueller and Oppenheimer’s research. The study simply is not relevant to the situation students face in taking notes to prepare for a college exam.

The AI analysis of a broader collection of studies comparing handwriting and keyboarding as input methods further clarifies the methodological limitations. Variations in methods and outcomes clarify what the AI can offer in criticizing unwarranted extrapolation of study results to test-preparation circumstances. 

Summary

I intend this as an examination of whether AI can identify what an experiment’s methodology justifies for practitioners. The example used involves one study from a larger collection of studies I have analyzed without the use of AI with a careful focus on the methods of the studies and how these methods related to the circumstances learners face. My reaction to the results AI generated is that limitations were identified challenging what others who have reviewed this study were willing to conclude.  The comments generated in response to the request to evaluate the methodology are provided in an appendix for your consideration.

Citation

Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological Science, 25(6), 1159-1168.

Appendix

The methodology of the Mueller and Oppenheimer (2014) study provides significant insights into the cognitive processes of note-taking, but its application as a “clear guideline” for college students requires careful consideration of its experimental constraints.

Methodological Strengths and Findings [1][2]

The study established a clear link between the medium of note-taking and the depth of cognitive processing:

  • Generative vs. Nongenerative Processing: The researchers found that longhand note-taking is inherently generative (summarizing, paraphrasing, and concept mapping), whereas laptop note-taking tends to be nongenerative (verbatim copying) [3].
  • Performance Outcomes: Participants using longhand performed significantly better on both factual and conceptual questions compared to laptop users [2]. This remained true even when researchers explicitly instructed laptop users to avoid verbatim transcription [4].
  • Verbatim Overlap: The study identified that the high rate of verbatim overlap in laptop notes was a primary driver of poorer performance, as it indicates shallower processing during the encoding phase [3].

Limitations for Practical Application [1][3]

While the findings are robust, the methodology has specific characteristics that may limit its direct translation into a universal guideline for all college scenarios:

  • Short-Term vs. Long-Term Utility: The study highlights the encoding hypothesis—the benefit gained during the act of taking notes. However, it also acknowledges the external-storage effect, which is the benefit of reviewing notes later [3]. In a real-world college setting, the ability to review comprehensive (even verbatim) notes might sometimes outweigh the immediate encoding benefits of longhand.
  • Artificial Environment: The study used filmed lectures lasting approximately 7 minutes on specific topics like bats, vaccines, and respiration [1]. This may not fully replicate the cognitive load or endurance required for a 60- to 90-minute complex university lecture.
  • Participant Pool: The data was collected from a specific subject pool at the University of California, Los Angeles [1], which may not account for the diverse learning needs or varying typing/writing speeds of the broader student population.

*********

Response without targeting just the Mueller and Oppenheimer paper

The methodology of Mueller and Oppenheimer (2014) does not provide clear, definitive guidelines for college students, primarily because subsequent replications and meta-analyses have failed to consistently support their original findings. While the original study suggested a “longhand advantage,” the methodology itself has significant limitations when applied to real-world academic practice.

Original Findings and Proposed Mechanism [10][12]

Mueller and Oppenheimer argued that the medium of note-taking affects the encoding process:

  • Laptop Note-taking: Leads to “shallow processing” because students can type fast enough to record lectures verbatim, acting as stenographers rather than deep thinkers [15][29].
  • Longhand Note-taking: Forces students to be selective and paraphrase because they cannot write as fast as a lecturer speaks. This “desirable difficulty” was thought to promote better conceptual understanding [10][26].
  • Initial Results: They found that while factual recall was similar between groups, longhand note-takers performed significantly better on conceptual-application questions [1][10].

Methodological Limitations for Practice [3][28]

Several aspects of the study’s methodology limit its direct application to college students:

  • Artificial Content: The study used non-course materials, such as TED Talks or unrelated prose passages, rather than authentic, semester-long course content [16][28].
  • Short Duration: The lectures were approximately 7 minutes long, which does not reflect the cognitive load or volume of information in a standard 50- to 90-minute college lecture [16].
  • Isolation: The study assumed students take notes in isolation, whereas modern classrooms may involve collaborative or group-based digital note-taking that could mitigate the “stenographer” effect [29].

Contradictory Evidence from Replications [1][7]

The most significant challenge to using this study as a guideline is the lack of robust replication:

  • Urry et al. (2021): A direct pre-registered replication found no significant difference between laptop and longhand conditions for either factual recall or conceptual application [1].
  • Morehead et al. (2019): Found no differences in note-taking mode for conceptual understanding after a 30-minute delay, and no differences in either factual or conceptual understanding after a two-day delay [18][22].
  • Voyer et al. (2022): A multilevel meta-analysis concluded that the note-taking method does not promote deeper processing or significantly affect academic performance, suggesting that factors like individual differences or distractions are more impactful [7].

Practical Implications for Students [9][15]

Rather than a strict “pen vs. keyboard” rule, the data suggests more nuanced guidelines:

  • Quantity vs. Quality: Higher word count generally predicts better performance because it provides a better “external storage” for later review [9][15]. Laptops excel at this.
  • The Verbatim Trap: The real detriment is verbatim transcription, not the laptop itself. Students using laptops should be coached to paraphrase and synthesize rather than transcribe [15][27].
  • Reviewing is Key: Some research suggests that the advantages of verbatim laptop notes may actually emerge when students have the opportunity to study their notes before an exam, as they have a more complete record to review [27].

Loading

Posted in Uncategorized | Tagged | Comments Off on Using AI to Analyze the Limitations of a Research Methodology

When are practices evidence-based when it comes to taking notes?

Posted on October 2, 2026 by admin

I want to explore evidence-based practice. Just what does this mean and why is it important? At one time, I probably would have assumed this topic would interest only research geeks like me, but all the hoopla over teaching elementary school kids to read with an evidence-based approach changed my mind. Governors and other politicians were talking about mandating that schools take an evidence-based approach. Forty states require early elementary kids to be taught reading in a certain way if the school they are enrolled in expected state money. Most of us became competent readers without such funding threats, but that is beside the point. Throwing the phrase “evidence based” around is not limited to researchers. 

I use this example, even though I don’t intend to focus on reading, to show that the phrase has meaning and consequences. To reach my broader goal, which is to question a policy that would require students to take notes on paper, I want to consider the different types of research conducted and which are essential before we can assume a given approach has real value. 

One more quick diversion based on an example I assume most would recognize. How do researchers get from a hunch that a given substance might have medicinal value to the meds we take when ill or to maintain our health? In medical research, scientists screen thousands of chemical compounds or biological molecules to find “hits”—molecules that interact with the target in a desirable way. Perhaps some bacteria in a petri dish die when the compound is introduced. Promising candidates undergo extensive laboratory testing in vitro (cell and tissue cultures) and in vivo (animal models). Finally, tests are conducted with humans to evaluate tolerance and side effects, and then the impact in achieving the desired goal. Even then, data continue to be gathered after release for general use to identify overlooked issues. 

Switching back to reflections on how best to learn, an important practical question might be – at what point has the requirement of “evidence-based” been achieved? It does matter. Educational practice is frequently described as prone to drastic swings. With the issue of learning to read, at one point in time reading is taught with a strong focus on the use of context and word recognition and then the pendulum swings in the opposite direction and readers are expected to sound out words and even nonsense words to develop phonetic skills. Computers are exciting and valuable at one point and then distracting and inconsistent with learning basic academic skills at another. How does an evidence-based approach fit with such inconsistency? Perhaps more relevant, what goes wrong when a supposed evidence-based approach is abandoned for another supposed evidence-based approach? It turns out it is quite difficult to rigorously investigate applied educational practices.

Studying is difficult to study

I happen to be on the wrong side of the popular recommendation that notes should be taken by hand. I wouldn’t really care, except educators may require class notes be taken by hand, which I think should require a higher standard of certainty. All I can do is try to surface reasons to reexamine assumptions one might make. 

Like the medical research model I identified, there is a fairly standard sequence. It is a different type of sequence based on the advantages and disadvantages of different types of research. There are two types of sequence. First, there is the ease and advantages of doing “lab” studies versus more applied studies. The second sequence involves what might be described as differences in internal versus external validity. Lab studies are best at establishing internal validity so one can be more certain of what caused what. Field studies are better at establishing external validity. Will a given treatment work as predicted when applied? With note-taking, college students are exposed to a short video to simulate a classroom lecture and are required to take notes using a keyboard or by hand, there is typically what is called a “filled delay” or distraction task to better focus on what is retained in memory, and then some type of performance task. Other research methods attempt to model actual classroom practice, by adding elements such as a longer delay (e.g, the performance task follows a week delay) and components of study such as the opportunity to review notes before the performance task. Eventually, researchers might study what students do in actual classroom settings. By considering these different methods, you can see the trade-offs between internal and external validity. 

One difference between the sequence I identified in medical treatment and educational practice is that in medical applications all levels must exist before applications are allowed. When I describe my interest in taking notes on a computer, professionals I know may even cite well-known studies (e.g., Mueller and Oppenheimer – The Pen is mightier than the keyboard) to inform me that handwriting is superior and they are taking an evidence-based position. Yes, I have read that study. It does have a catchy title. I then ask if they know that in that study students listened to a 15 minute TED talk, the test was given after a short delay, and students did not study their notes. What do you think that study has to say about college classes in which students listen to maybe 4 weeks of 50 minute lectures and study their notes repeatedly before taking an examination? It is not that the study is without value, it is that it poorly matches the factors that would provide external validity. Digital notes have advantages – searchability, easy modification to fill in gaps and personalize – much more difficult with handwritten notes. A plausible case can be made for taking notes with a laptop and until a greater variety of research exists, I trust individuals to take notes in whatever way they find to be productive. Jumping on the bandwagon based on one component of the full scope of necessary research is likely attractive, but not justified. This is my point, we don’t really know that taking notes by hand is superior. If someone challenges your opposition, you don’t have to accuse them of being lazy, reading only the Abstracts of research studies, or worse yet relying on popular online posts opposing technology in classrooms. Just ask if they have read the Methods section of the note-taking studies they think are informative. 

Loading

Posted in Uncategorized | Tagged , | Comments Off on When are practices evidence-based when it comes to taking notes?

You don’t have to convince me humans can do a better job than AI providing important educational services, you have to convince me you are willing to pay these humans

Posted on September 16, 2026 by admin

It has been a challenging week for those of us promoting the potential benefits of AI. Despite the reality that resigning from Anthropic may have cost him a fortune when his company goes public, an employee did quit, claiming he could not, in good conscience, continue to work at an AI company when there is a chance more powerful models may end up killing everyone (10% chance was his estimate). Other employees agreed; congressional leaders panicked, but Trump said to forge ahead because we must compete with AI development in other countries. All the tech pundits immediately focused on the topic, and most news outlets frantically tried to find someone willing to come on their programs who sounded like they knew something about this concern. In this context, I am the guy trying to write something about the value of AI in K12 and secondary classrooms.

First, while not the focus of this post, let me say that the AI models I support have been around for some time and aren’t the newest models that have created the current concern. I have no idea if we are locked in a “Manhattan-like” race to achieve the level of AI intelligence that would make our adversaries cower in fear or at least assume we have achieved a level of mutual assured destruction. I have opinions, but I try to stick to things I can argue with at least some degree of insight and credibility. I want to make a pro-AI argument based on a couple of topics I have addressed before — AI tutors and AI Writing Feedback.

You don’t have to convince me humans can do a better job.

Tutoring — I must have a dozen or so posts focused on tutoring. I became interested in tutoring perhaps 40 years ago because I was interested in mastery learning, and I read Benjamin Bloom’s two-sigma challenge argument. Bloom proposed that researchers should develop approaches to school learning that matched the level of achievement learners could reach with consistent access to a tutor. He proposed mastery learning because it was a strategy that provided some of the benefits of tutoring — e.g., meeting the student where they are, addressing deficiencies rather than moving ahead whether or not a student was ready, and multiple opportunities to demonstrate their competence and understanding. While I was interested decades before AI tools were available, it seemed to me that technology could offer a practical way to make some of these component strategies feasible.

AI rekindled my interest. AI has some obvious advantages to the benefits I saw earlier. Most simply, it could be responsive at the level of the individual. AI does not have to rely on preestablished smaller units of instruction and the benefits of evaluating and shaping an optimal rate of progress through these units. AI can respond to a learner’s requests for help and evaluate performance tasks to provide feedback and make independent decisions about what should happen next.

Responding to student writing — So much of the commentary surrounding AI and learning to write or writing to learn has focused on cheating. How can educators give a grade when they cannot be certain that a student has actually written the work that has been turned in? How can students learn to write when experiences designed to exercise writing subtasks can easily be completed by AI rather than the student? Certainly, those of us who assign and evaluate writing tasks must grapple with these possibilities.

My interest has been more in making use of AI to provide feedback on writing tasks. We have very useful ideas about developing writing skills that focus on writing a lot, receiving immediate feedback on initial attempts and rewriting the initial drafts based on this feedback, and even writing conferences in which a student meets face to face with a teacher to discuss what they have written and their thoughts on their strategies and challenges. Research shows the impact of these tactics.

The Reality

You do not have to convince me that a skilled human being does a better job at vital human tasks. You have to convince me you are actually willing to pay for one. We live in societies that demand universal, high-touch services while systematically starving public education budgets. Whenever we are faced with the staggering bill of hiring, training, and retaining competent professionals at scale, we blink. Both tutoring and writing instruction make great examples of drastic understaffing.

It would be great if every student, but especially students who struggle, had access to a well-trained tutor. Wealthy parents can afford such assistance. Schools receive some government assistance. However, consider the summary of a recent study from USC — Two-percent of U.S. children receive high quality tutoring. The definition of “high quality” was based on the following factors:

  • Is delivered at school
  • Is delivered during school hours
  • Groups are composed of three or fewer students to one tutor (four or fewer students can be effective with older students)
  • Is provided to a student three or more times per week
  • Lasts at least 30 minutes per session.
  • Is delivered by teachers or well-trained professional tutors rather than volunteers, peers, or parents.

These are high expectations, but even if you relax them, you get the point. It is simply impractical to provide enough human tutors to meet classroom needs simply because of the cost and availability of suitable professionals. School budgets are already constantly being challenged at existing levels of taxation with larger and larger class sizes and other cost containment strategies becoming the standard.

Developing writing skills faces a similar reality. I know some folks who teach the freshman-level English Composition classes and always felt they had a really difficult job. Most are hired as adjuncts and work 4 classes if they can get what is considered a full load. Class sizes are typically 20 students, making a full load the responsibility for 80 students. My English Comp classes were 50+ years ago, and that may have biased my opinion. I thought the classes were pretty much worthless, with a focus more on literature than writing. The one writing class I had that was beneficial was called Business Writing (I think this was the title) and required more writing and more thorough evaluations. I probably wrote the most as an undergraduate in the Psychology Research Methods course which had a sizeable component focused on APA style and writing research reports. I encountered the “no-no” passive voice for the first time and struggled to express myself in a form that just did not sound right to me.

One of the books I have read on the impact of college — Academically Adrift — is quite critical of the college experience, blaming both institutions and the commitment of learners. The book draws heavily on a longitudinal study of the college experience, questioning what students learn in early-year college courses. Written communication is one area that receives attention.

I tried without much luck to access studies documenting how much writing a typical Freshman writing course requires. You would think such data would exist. AI requests returned data that were based on multiple syllabi the AI appeared to access to respond to my request. The conclusion from this approach was that students turn in 15–20 pages of what was described as formal writing and maybe 25–30 pages of informal writing. I was unfamiliar with this distinction so I requested an explanation. The description of informal writing follows.

Informal writing often means lower-stakes composing that supports learning and revision, such as:

* Freewrites and brainstorming.

* Reading annotations and response journals.

* Discussion posts.

* Peer-review comments.

* Outlines and research notes.

* Draft reflections or process memos.

* Exploratory writing before a formal essay.

The 20 pages of writing I mentioned above mentioned in the Academically Adrift report as the point at which measurable improvement was typical. However, 50% of students did not reach this level of activity and showed lower levels of improvement and sometimes very little at all.

I don’t see a practical way for educators working with 80 or so students to increase the amount of writing that can be carefully reviewed and critiqued, discussed with students from time to time, and rewritten and reevaluated. I don’t see any way to provide feedback within a couple of days so students can rework the original submission on a reasonable schedule. I doubt institutions will be able to lower class sizes by higher additional experienced writing instructors.

Writing instruction in K12 was difficult for me to describe based on sources that quantify the amount students do. I was surprised the amount in dedicated higher ed writing courses was so low and I know without the equivalent kind of data that similar concerns are made about writing among younger learners. Distrust of out of class writing will likely only add too this issue.

I assume these data provide a reasonable description of what we can do with the current Human Resources commitment.

Summary

This situation reminds me of the expression “don’t let perfect be the enemy of good”. My proposal and the topic of plenty of earlier posts is that AI tutoring and writing assistance seems a reasonable way to augment the work of beleaguered educators.

Loading

Posted in Uncategorized | Tagged , | Comments Off on You don’t have to convince me humans can do a better job than AI providing important educational services, you have to convince me you are willing to pay these humans

An Analysis of AI Information to Output Processing Options

Posted on August 23, 2026 by admin

There are multiple ways to think about what using AI in writing could mean. Here is one approach. As I have studied and used AI in writing, I have noticed what I think is a continuum of categories that I can operationalize based on different AI tools and how they move from information to written product. Most of these tools are more flexible than I describe here, but I think the distinctions I identify make sense.

The products I am referring to range from a completed blog to ideas or an outline I might use to guide my own writing. Here is one example. 

Assume my goal is to write a 1000-word blog post on how taking notes by hand versus using a keyboard influences student learning and understanding. Here are various approaches I might take and the tools/services I would use. 

  1. ChatGPT or Claude. Submit the goal just stated to an AI tool as a prompt and accept the output as the written product. In this case, the input information is the knowledge base stored as the model within the system and the mechanism the system uses for translating that knowledge into a product. Beyond submitting the prompt, I need to do nothing else to generate an output. 
  2. Recall.ai. This approach uses RAG to designate information sources that an AI prompt then uses to generate a product. Given my goal, I might submit, say, 15 PDFs of journal articles relevant to the topic I have in mind. The AI tool then creates a summary (I can control brief vs extended) of each article. Once completed, I submit the prompt explaining my general goal. The one thing I control is the information input to the system. I don’t have to read the documents and the AI system creates an output based on the designated content based on the request identified in the prompt.
  3. Obsidian and AI plugin. I read, highlight, and annotate 15 PDFs of journal articles on the topic of interest. Highlights and annotations are first exported from the tool I have used to read these articles and are then stored as a note or notes within Obsidian. The AI plugin in Obsidian then uses the same prompt identified earlier to generate the written product. In this approach, I have selected the articles, designated information I think is relevant (highlights), and added my own insights as annotations. The AI uses the information I designated as relevant and my annotations to generate an output. 
  4. Obsidian to Karpathy Wiki to Product. I read, highlight, and annotate 15 PDFs of journal articles on the topic of interest. Highlights and annotations are exported from the tool I used to read these articles and stored as notes in Obsidian. AI is now used in two distinct stages. First, the AI runs on the content extracted from the original PDFs to identify key concepts, entities (e.g., researchers), and summaries of individual sources. The concepts, entities and summaries are linked by AI to establish relationships – e.g., which researchers conducted which studies, which studies reached a given conclusion or contradicted a given conclusion. The second use of AI applies the goal prompt to this more organized information to produce the final product. (Note: The goal of this process is really to be able to add new resources over time and not have to process the entire batch of inputs each time a new prompt is submitted. This approach offers an efficiency and cost advantage. However, I am focused here on initially identifying themes and concepts, then using them as distinct inputs for the final product as a different approach and possible parallel to how human writers function.)
  5. Obsidian. I read, highlight, and annotate 15 PDFs of journal articles on the topic of interest. I first export highlights and annotations from the tool I used to read these articles and store them as notes within Obsidian. Once stored, these notes are tagged, linked, and new related notes are generated by thinking about the content already stored. I write the final product to meet my proposed goal using this content. Obviously, there is no AI use at this end of the continuum I have constructed but I wanted my list to move from total AI to total human. 

I have used most of these approaches, except for simply posting something written without any input beyond my prompt. I wonder objectively which of these approaches produces the product of the highest quality, but that is likely a nuanced question and I will leave it for others to explore for now. 

Loading

Posted in Uncategorized | Tagged | 1 Comment

Reading With LLM Access: It’s Not About Studying

Posted on August 7, 2026 by admin

I frequently write about AI applications in education, so when I find a relevant research study, I pay close attention. It is not that there is a lack of attention to AI among educators; there are plenty of suggestions for classroom applications and data on educators’ and students’ practices. What interests me most are manipulative research studies that test the impact of specific uses of AI on achievement. The following is a description of one such study and a critique focused on how educators may misinterpret what they think the study shows about classroom applications. The concerns I emphasize were explained in detail in a previous post. 

Effects of LLM use and note-taking on reading comprehension and memory

Study behavior has been a professional interest for years. Generative activities, external tasks such as answering questions or taking notes that potentially improve the efficiency or even application of beneficial cognitive (mental) behaviors, have been the framework I have used as a way to offer learners ways to improve understanding and retention. The interaction between learners and AI offers multiple ways a learner can use an LLM to create generative experiences for themselves. For example, a learner can request that an LLM present and evaluate their responses to questions about content submitted as part of a prompt or even just the description of a topic of study (e.g., Ask me five multiple-choice questions about operant conditioning one at a time and provide feedback after my response to each question). 

Pia Kreijkes and colleagues wanted to evaluate how high school students would apply an LLM while reading an assignment and whether their actions would be related to multiple measures of understanding and retention. To put the impact of the AI activities in some type of perspective, the researchers decided to compare the impact of the AI activities with a more traditional generative strategy – taking notes. In their study, hundreds of high school students were assigned a reading task that would take between ten and fifteen minutes under two of three treatment conditions – take notes while reading, interact with an LLM while reading, or do both. This is called a within-subjects design, as each participant engaged in two of the treatment conditions, allowing for the elimination of some of the variability (error) due to differences (e.g., reading ability) among participants. It is a weird design in that in most within-S studies each participant engages in all of the treatment conditions and why this two-of-three approach was used I cannot explain. Anyway, as a research design, the number of participants and the within-S design lend high confidence to the results. 

Three days later, participants responded to items testing their retention and understanding and a request to free-recall as much information as they could. 

The results showed that the note-taking treatment and the combination of LLM and note-taking treatments scored higher than the LLM-only treatment. The researchers concluded that the benefit of note-taking over LLM use tended to be stronger and was observed across all measures of learning, whereas the benefit of LLM and Notes was observed for literal retention and comprehension but not for free recall.

The researchers attempted to qualitatively analyze the prompts students generated with the LLM. The most common prompt emphasized seeking additional information and deeper understanding rather than information condensation or study and memory help.

This is a very basic description, but I am interested in what readers might assume from it. Consider what you think this study suggests for secondary classrooms. I will offer my own takeaway after making a few specific comments on this study.

Consider the Methodology Carefully When Using Findings to Guide Actions

Here are some issues I have with the methodology:

  1. Why is there no actual control group (reading only)? The sample size certainly would allow the existing treatments to be compared to a read-only treatment. This may matter because of other issues I have with the methodology.
  2. Why is there no opportunity for review or what I would describe as study? There is a small, well-known generative benefit to taking notes as a way to process input. However, there is a much larger benefit of using notes post-exposure, but before the assessment of retention and understanding. I have not read studies on LLM interaction that report immediate benefits. One might assume this would be there, but it did not seem to be in this study, given the specific content studied. I would be more interested in LLM benefits as part of review or as a way to get additional insights as the original memories have faded. 
  3. The three-day delay seems short if one is interested in typical classroom assessments or in generalizing to other out-of-classroom uses of notes and LLMs.
  4. The content seems very short – you can both read and process in between 10 and 15 minutes. 

My takeaway. Here is the issue I think the study suggests. This might surprise you. I think it is impressive that high school students, when given access to an AI tool, use it to improve their understanding through additional information or a different way of explaining something. It was not used to make the learning task easier – some version of a summary so I don’t have to read it all approach. 

One guess about why so many potential improvements may have been ignored. Why not use longer reading passages? Why not a longer delay and opportunities to review? Why not an expansion of the treatment conditions or a more powerful within-subjects approach in which subjects experienced all treatments. My guess is that to get schools to allow researchers access to students, schools wanted an approach that was quick and over quickly. In my experience, this is understandable, but also a severe limitation on research that would provide the most useful information for applications. 

What I think should be the focus on both note-taking and student use of LLMs

As I alluded to above, the greatest benefits to learning and understanding come from what learners do after initial exposure to experiences. One way to think about this is to describe such activities collectively as studying. Simple strategies such as retrieval practice and interleaving have powerful and consistent benefits. There are plenty of strategies for working with personal notes over time that offer variations on these proven learning strategies and some exciting opportunities for student use of LLMs. Students can have an LLM ask them questions and more importantly evaluate responses and provide feedback and interaction based on the feedback. The LLM can function in ways as a tutor or even be used by small study groups as a tutor. 

It is easy to imagine research on spontaneous use of such study possibilities or structured approaches that would guide student use of note taking, LLMs, or potentially the combination. As I have suggested, the logistics of implementing such applied research is challenging, but I don’t think the short-term, quick experiments provide the level of authenticity that is needed. 

A sample of my other posts on related topics:

The space between encountering information and application

Digital devices and effective studying and long term note use

Cooperative learning when AI is your partner

Citation

Kreijkes, P., Viktor Kewenig, Martina Kuvalja, Mina Lee, Jake M. Hofman, Sylvia Vitello, Abigail Sellen, Sean Rintel, Daniel G. Goldstein, David Rothschild, Lev Tankelevitch, Tim Oates. (2026).  Effects of LLM use and note-taking on reading comprehension and memory: A randomised experiment in secondary schools. Computers & Education, (243), 1-24. (Article 105514)

‘

Loading

Posted in Uncategorized | Tagged , | Comments Off on Reading With LLM Access: It’s Not About Studying

AI’s Future: Bigger and Bigger, Smaller and More Personal, or Both

Posted on July 25, 2026 by admin

I have been reading Columbia University’s legal scholar Timothy Wu’s Age of Extraction for a second time. I highly recommend this book for those interested in an analysis of the big online platforms (e.g., Google, Amazon, Facebook). With the passage of time and the new life experiences that occur, a second reading can allow different insights and new topics for sharing. I first read Wu’s book soon after it was published and at a time I was also concerned about “free” social media and the damage done by manipulative algorithms. I was exploring AI, but like Wu made few connections beyond noting AI companies like social media were dominated by large platform companies. It is these insights into AI platforms and possible alternatives I want to explore here.

Wu’s book is partly about the history of our opportunities online and how the original opportunities for individuals to independently share their insights and explore commercial opportunities were eventually taken over by a few major platforms. This transition was explained in terms of first the opportunities platforms provided, first free everything, and then the transition to the manipulation of time and attention in a way that made money for platforms and for the associated boards and stock holders. So many have been captured because of the network effect (I can’t leave because my friends are all there) and while little can be done about this for the majority, Wu offers suggestions for those who are interested in alternatives. 

Is AI different or is it still too early?

Upon my second reading, I am more thoroughly steeped in the short history of AI and the opportunity to explore the possibilities both for those in the education community and for myself. AI platforms don’t seem to have followed the same pattern as social media. AI did not grow from the initial interests and activities of hobbyists and individual explorers toward the integration of everyone by the platforms making everything easier and more powerful. The ideas may have originated with the community of academics exploring various approaches to thinking machines over the decades, but the present large language models required the huge financial resources of major corporations and their opportunity to raise massive sums of money necessary to both explore and build the models we can now access. Even the massive budgets of these companies may not be able to feed the beast that is emerging as the way to develop this field. Governments may need to be involved to limit the redundancy that is at the core of the typical capitalistic approach. I explore the ideas of those who believe China has an advantage because the powers that be there can require a more integrated and less competitive approach. 

The situation with AI has changed a bit. Progress has been made to the point at which individuals and smaller organizations can create and obtain value from the application of smaller models on equipment as inexpensive as a Mac Mini. These opportunities have captured my time and treasure for the past few months. Is it possible we are witnessing a reversal of the typical platform dominant pattern Wu describes?

Here is a short description with a few more details describing Wu’s platform model.

Platforms begin by offering genuine convenience. Leading AI companies offer cheap or free assistants, APIs, coding tools, and cloud-hosted models that lower barriers for developers and users.

Dependency follows scale and network effects. A company can become hard to leave once an organization has built workflows, data pipelines, fine-tuning, agents, evaluations, and staff skills around its models and APIs.

The platform becomes a gatekeeper.  AI firms and their cloud partners can control access to compute, model capabilities, distribution channels, app ecosystems, and sometimes the terms under which others can build businesses.

Extraction comes after lock-in. This could appear as higher API prices, usage metering, restrictive contracts, preferential access for the platform’s own products, data capture, or fees are imposed on developers and businesses that rely on a given platform.

Concentrated power shapes the whole economy. If a small group controls the main models, chips, cloud infrastructure, and routes to users, it can influence which firms survive, what work is automated, and who receives AI-generated productivity gains. The power to influence government decisions when AI platforms reach a scale of influencing the economy are possible. Note the present argument that these platforms must be supported to match or exceed advances in other countries. 

The process sounds scary when described like this. For me, we are already well on our way. 

My personal interest has been in the application of AIs analysis and writing skills to the content I have abstracted from my personal reading and augmented with my thoughts as notes. To do this, I must first upload my collected content (a process typically called injestion) and then interact with this content searching for insights and ways of expressing what such insights might reveal. I am much less interested in interacting with AI based on the existing models than the application to my content. This is most frequently described as Retrieval Augmented Generation (RAG). 

As long as you can trade time for speed and power, RAG applications can now be executed on your own equipment. While not the equivalent of similar approaches using more powerful platform models, the output is certainly useful to my personal work flow and I assume will only become more so if these efforts on personal AI are allowed to continue.

The issue is whether this process will be allowed to flourish. Will the companies developing these smaller models start selling them or not release them at all? Wu doesn’t address such innovations, but the pressure to commercialize that he describes would make this prediction. While these smaller models are available from several sources, I am presently using QWEN which is made available by a Chinese company. I suppose this will eventually raise concerns (see the following comment about dangerous uses), but after some trial and error and suggestions from others models from this company just seem to work better for what I am trying to do.

One thing I already see that worries me is the concern that effective AI outside of the control of big corporations opens up the possibility that dangerous actors will use these less controlled tools to create dangerous biological weapons or engage in hacking activities making our reliance on online resources more dangerous. The approach of raising concerns of hypothetical dangers seems a common corporate strategy – typically it is about children and who can best protect them. I see this more about what we believe is the best approach to experimentation and improvement. Present approaches cost out individuals, small companies, and even university researchers. It seems about who will be able to convince Congress of what? 

Source:

Wu, T. (2025). The age of extraction: How tech platforms conquered the economy and threaten our future prosperity. Knopf

Loading

Posted in Uncategorized | Comments Off on AI’s Future: Bigger and Bigger, Smaller and More Personal, or Both