Learning Aloud

The “you are doing it wrong” excuse and Classroom AI

There is a common perspective on the practice of education, intended as a criticism, I think, that proposes if a visitor from the past were to be time-traveled to the present, he or she would be amazed, but bewildered by so many areas of civilization (travel, medicine, farming), but feel completely at home in K12 or university classrooms. As an educational researcher, I admit this claim has always troubled me. Was the process of passing on knowledge and developing important skills optimized centuries ago despite all of the folks like me who study how people learn and how the processes supporting learning might be improved? If I disagreed, what would I identify as a counterexample, or how, at the very least, would I justify the time, effort, and resources people like me have invested in changing the status quo?

My Interest in Individualization

A general topic that has long been at the core of my personal research interests has been individualization. It seems obvious that learners differ on important variables that impact learning. Some have greater aptitude than others. Some, due to an endless list of differences in life experiences, at a given point in time have significant gaps in relevant background knowledge and prerequisite skills. For economic and historical reasons, our approach to assisting student learning largely ignores these differences. Our system, despite claims, fails to actually meet students where they are to most efficiently move them forward. Where students are also ignores differences in goals, interests, and whatever else might come under the general heading of motivation.

Those who have followed my posts over the years will likely recognize that much of what I have done has focused on evaluating approaches that make use of technology to expand the flexibility educational systems can practically offer. I will identify two such topics for those who might want to explore my past posts –mastery learning andtechnology-supported tutoring. I admit that these seemingly logical opportunities have not yet yielded the benefits in application I had hoped, and this is the topic I want to examine in this post.

Research in the social sciences which would include applied research in education (e.g., classroom learning) has notorious weaknesses, but unique challenges. For example, recent criticism offered to the public notes the high rate at which published research cannot be replicated. We seem in an era in which funding for science in general has been questioned, so with cutbacks those of us working in more challenging areas have reason to be concerned. Yes, I said more challenging. I agree the “basic sciences” are of great importance and deserve support, but think of the claim I used to offer when I taught the research section of Introduction to Psychology – the chemicals in the test tube, the electrons in the circuit, or the planets in space don’t think about how they feel like reacting today. The rules that explain such behaviors may be intricate and difficult to ascertain, but at least most are reliable. The challenges social scientists face are simply different, but the general trend has been toward greater understanding.

Back to the thought experiment about the visitor from the past

If one assumes progress should happen when it either hasn’t or at least not to the degree that seems reasonable, is there reason for optimism? Are optimists delusional? What are optimists up against when it comes to criticism of present practices and seeking funding and attention for new approaches?

Changing a massive system with highly ingrained beliefs and behaviors is tremendously difficult. New ideas struggle to take hold and mature within this environment. An “intellectual pessimism” used to resist deep exploration of theoretically logical and basic research justified changes I have decided to describe as the “you are doing it wrong” plea for continued experimentation. I don’t think you can search for additional references to this phrase expecting a lot of success, but it is a phrase I have decided captures the attitude I think typifies the resistance others have identified.

The phrase implies criticism of researchers who insulate themselves from scrutiny of their “big ideas” by attributing poor outcomes to implementation failures rather than to flaws in the ideas themselves. In other words, why do “big ideas” continue to resurface repeatedly over time when attempts to apply these ideas have not previously been successful. Perhaps most simplistic put, it is about excuses.

The “you are doing it wrong” explanation works like this. When a widely adopted educational innovation – learning styles, discovery learning, whole language reading, AI tutoring, open classrooms, etc. produces disappointing or mixed results, proponents rarely concede the theory is wrong. Instead they argue: the idea is sound, but practitioners didn’t execute it faithfully or well enough. The failure belongs to the implementers, not the framework.

This rationale functions as an unfalsifiable escape hatch. Any negative evidence gets reframed as a measurement of implementation quality rather than a test of the underlying idea. The theory can never lose, because every failure is a fidelity problem.

In education, common variants are typified by the following:

“Teachers didn’t receive adequate training” – used with constructivism, project-based learning, differentiated instruction, AI

“It wasn’t implemented with fidelity” – the research or theoretical components were not followed with sufficient care.

“The conditions weren’t right” – class sizes, demographics, resources, culture

“It was a watered-down version” – the pure form was never really tried

With these excuses, it is the grain of truth that makes it plausible. Of course, there is always the possibility that the excuse is valid. This pattern is worth naming clearly in writing about learning science, because it explains a lot about why education cycles through fashions without accumulating settled knowledge the way other applied fields do.

Are classroom uses of AI the most recent examples of “you are doing it wrong”?

AI applications in classrooms represent recent examples of promising innovations, but also potentially of an impotent fad (e.g., Gerlich). Claims of cheating instead of learning abound and while theory and carefully controlled research point to logical and demonstrated benefits it would seem fair to argue educators are concerned about most student use.

Recently, I have encountered multiple accounts that propose “you are doing it wrong”. I intend to develop an extended analysis of the core ideas of these claims in a future more analytical post, but I might quickly summarize here by explaining that true success in the use of AI is most likely when certain conditions of student motivation, metacognitive proficiency, and working memory issues are met. In many cases, these variables are not functioning at desirable levels.

Several writers have backed a three-level AI use model proposing that some levels (the lowest and the highest) are likely to be successful and the middle level, which is presently most common, is less likely to produce satisfactory results.

The three levels have the following characteristics:

Zone 1: No AI Involvement

In this level, learning occurs without any AI assistance. While learning happens, it is often “capacity-constrained” because the learner must spend significant time and effort on execution and task completion, leaving less bandwidth for higher-order reflection.

Zone 2: Scattered, Half-Hearted Use

This is characterized by using AI for minor tasks like fixing sentences, checking facts, or tidying paragraphs. It often produces the worst learning outcomes. The learner still carries nearly the full cognitive load but adds the overhead of managing AI interactions without gaining significant cognitive savings. Note: this summary paraphrases the description of the authors. My version would add having the student using the AI tool to perform the task based on simplistic instructions.

Zone 3: Committed, Strategic Delegation

This level involves offloading entire categories of substantive work to AI to free up genuine cognitive capacity. This freed bandwidth is then redirected toward tasks AI cannot do, such as critiquing frameworks, questioning assumptions, and making complex judgment calls. This zone is where “transformative learning” is thought to live, provided the course design is intentional about how and why tasks are delegated.

My suggest for making sense of these differences is to take a familiar task and work through what these different zones might look like in practice. I think learning to write makes a good case.

In this example, Zone one is easy – no use of AI. Zone two might include simply asking the AI tool to complete an assignment for you or perhaps using a tool such as Grammarly to check spelling and grammar. What would Zone three look like? Perhaps you might use an AI tool to suggest a list of topics you might address based on your general goal or perhaps create a draft and then ask the AI for a critic based on concerns you have about your initial effort.

Summary

I hope it is obvious how “you are not doing it right” would apply to how educators may allow students to use AI. The challenge then to evaluate whether such uses are an example of a typical educational fad or actually are limited because the learner is not doing it right.

Source

Gerlich M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies. 15(1), https://doi.org/10.3390/soc15010006

The U.S. and China AI Competition

The very recent summit involving Presidents Trump and XI Jinping dealt with many political controversies of the day, which included AI and related issues such as intellectual property. The mention of AI brought to mind a book by Kai-Fu Lee, which I think I read in 2019. I remembered some of the comments Lee made about China, computer science, and AI at that time. Lee, who has held both U.S. and Taiwanese citizenship, wrote that China would have important advantages in the development and application of technology, which surprised me at the time but made some sense given what I knew about China. Lee was educated in the U.S. (Carnegie Mellon Ph.D.), worked for Apple, then returned to Taiwan and later worked for Google in China. I explored my notes and highlights from that book and also from The Big Nine. My interest in the role of AI in education and its application across different countries led me to another article in my personal archive (Hao, 2019). The following comments are mostly based on Lee’s ideas, with some expansion using the other two references I have mentioned. All sources are a bit dated, given the rapid pace of AI developments, but I still find the core ideas worth considering.

According to Lee, China’s advantages in AI come from scale, data, industrial capacity, talent, and state coordination.

Scale equals more data

China’s 1.4 billion people give it control of “the largest, and possibly most important, natural resource in the era of AI: human data”—and that its huge number of internet users gives it both data quantity and quality for training models. This resource is roughly the equivalent of the combined resources of the United States and Europe. Lee offered this perspective some years ago when finding content seemed more a priority for U.S. companies who encountered push back when scrapping the web and books without permission.

Industry integration

Chinese companies share. For example, Tencent’s ecosystem is noted as perhaps the single richest data ecosystem of all the giants and combines multiple services, say, in contrast to X and Amazon. Concentration of data and services in a few massive platforms offers a related quantity and quality advantage.

Quantity of Talent

There is a Thomas Friedman quote I have always remembered. “Remember in China if you are a one in a million talent, there are 1400 others just like you.” Lee offers a different assessment of the talent situation specific to AI. He claims that the U.S. has more superstars, but China has the advantage in the number of engineers and computer scientists working in on AI and related fields. Aside of the great difference in population, engineering, programming and science are simply fields of advanced study that are seen as more of an opportunity in China. My own way of thinking about this difference is that in the U.S., business and finance attract many and in China these fields are less of a draw.

State Coordination and Standards

A “big advantage for China: it doesn’t have the privacy and security restrictions that might hinder progress in the United States”. The commitment to the massive surveillance of its own population is known focus of the Chinese government and a means of control and manipulation of its population. We rightfully consider the use of technology to probe the personal lives and values a violation of basic human rights and bristle internally at the collection of information about us by companies and the government. Simply put, China doesn’t have the privacy and security restrictions that might hinder progress in the United States. Despite tolerated abuses, the commitment to collecting and analyzing this type of information is a source of funding and a focus of experimentation in China.

” Move fast and break things” was the original Google creed, but a value system that has come under increasing criticism in China. Without the pressure to curb potential negative aspects of AI, China moves faster. Related to this is the greater top down decision making of the Chinese system. In the U.S., you have multiple businesses trying to raise huge sums of money and are often isolated from each other, often duplicating similar approaches. We historically value competition and assume the motivation has advantages. While true, I wonder about the “business model” sucking up a large share of the available investment money in this sector in the US. The amount of money required has to a great degree squeezed out university researchers who either leave universities or work around the edges of AI innovation. While AI research is a high priority in China, the U.S. has cut funding for NSF funding for AI and cybersecurity.

AI in China and Education

The personal interest that has driven my own interest in AI has been potential opportunities in education. This has been a messy issue in this country with pushback due to legitimate concerns for cheating, failure to address skill development, and lack of interest in instruction presented by a computer. China has committed to exploring AI-facilitated education.

Academic competition in China is tense. Millions of students a year take the college entrance exam, the gaokao. Your score determines whether and where you can study for a degree, and it’s seen as the biggest determinant of success for the rest of your life. Parents willingly pay for tutoring or anything else that helps their children get ahead. The options tech can provide outside of classrooms offer opportunities to sell experiences to well-meaning parents. (Hao)

Two companies that are likely unfamiliar to most U.S. educators, Squirrel AI and Alo7, make good example. Since the Hao article was published both services became available in the U.S.

Squirrel AI uses an “adaptive learning” model that breaks subjects into thousands of “knowledge points”—far more granular than traditional textbooks. The system diagnoses a student’s specific gaps and provides targeted video lectures and practice problems. The teachers are intended to act like “pilots,” stepping in only for emotional support or complex issues while the algorithm handles the core instruction. Educators will likely recognize similarities to the Kahn Academy.

In contrast, Alo7 emphasizes a “quality-oriented education” focusing on creativity and the liberal arts. This “intelligent classroom” use AI to analyze student engagement, pronunciation, and even “joy” through facial and vocal recognition

The interest in AI in education seems to be a combination of the emphasis of standardized test performance for advancement and opportunity, the larger population, and the greater risk tolerance within the context of exploration for improvement.

Summary

This post is not a value judgment comparing U.S. AI policies, but rather an attempt to summarize what some experts have said about the differences. My personal issue concerns the economic pressure in the U.S. based in our trust in competition among corporations to drive innovation. While this is an approach that has worked in many areas, the huge investments that are required have to this point sucked a great deal of capital from the economy and seem largely and unnecessarily redundant. I personally also find the focus of interest in AI in education (personalized and adaptive instruction) interesting as this emphasis has appealed to me based on my interest in mastery learning.

Sources

Hao, K. (2019). China has started a grand experiment in AI education. It could reshape how the world learns. MIT Technology Review, 123(1), 1-9.

Lee, Kai Fu. 2018). AI Superpowers: China, Silicon Valley, and the New World Order. Boston, Mass: Houghton Mifflin.

Webb, A. (2019). The big nine: How the tech titans and their thinking machines could warp humanity. PublicAffairs.

Ignoring The Instruction Option Of EdTech

When I first began writing professionally about K-12 use of technology in the mid-1990s, a popular approach was to organize content around the tutor, tool, tutee model. This model proposed that technology in the hands of students could deliver instruction (tutor), facilitate the activities of being a student (tool), and program/code (tutee). While AI now blurs the lines between these roles, this simple organizational scheme still seems useful.

This post was prompted by what I sense to be dissatisfaction with the instructional component of this model and a recent paper entitled the “5% problem. This paper challenged the positive benefits of commercial instructional offerings (e.g., Kahn Academy, CK-12) as misrepresenting what the data on achievement they have collected demonstrate. Ignore my descriptor of such programs as commercial when I know you can use at least many of the features of such offerings at no cost. How these efforts are funded is a different issue. The relevance of “5%” lies in the hidden expectation that only those who use the learning system as intended are included in the analyzed data. Some studies reporting high effectiveness are based on 5% of those provided access and this important factor is not highlighted in the reporting of results.

Such assertions make me uncomfortable. Despite what to me seems a backlash against screen time, cautions related to AI allowing learners to offload the experiences intended by learning tasks, and concerns classroom circumstances associated with technology have caused educators to limit meaningful social contact with students and students with each other, now I am feeling I must question the studies I have explored on the benefits of AI tutoring and the personalization of the rate of progress through instructional materials allowed by computer supported instruction (e.g., Kulik & Fletcher).

Teacher Commitment

As I have considered this recent challenge, it has occurred to me that I have encountered a variant of it throughout my career. In 2019, I wrote a blog post titled “There is a reason teachers don’t use the software provided by their districts.” At the time, this issue caught my attention because my wife and I were serving on an advisory group for our local school district and the tech director reported on a monitoring software used to track the use of software the district had purchased to make decisions about which license access packages could be dropped so funds could be reallocated to other requests. I noticed some researchers were using what seemed like a similar system to examine the use of instructional technology and to consider why it was underutilized. These scholars reached a conclusion nearly identical to that of the more recent, in-depth examination of online instructional tools. “One of the other primary findings of this report is that usage of apps is generally lower than might be expected. Most apps are used only for a limited time, and most purchased by districts go unused. This has an impact on efficacy – an app cannot be effective if it is not used” (p.25).

At that time, it seemed the issue was explaining teacher commitment. Thomas Arnett has weighed in on the issue of school-funded software being seriously underutilized, speculating, based on his Jobs to be Done Theory, that educators simply don’t perceive that the software they have access to helps them satisfy the jobs they perceive as expected of them, relative to more traditional approaches. These jobs are described as 1) Help me lead the way in improving my school, 2) Help me find practical ways to engage and challenge more students, and 3) Help me replace a broken instructional model so I can help each student. From my perspective, many technology-based instruction systems seem purposefully designed to address individual learning speeds and existing knowledge, but perhaps this is how these resources by educators. In a more detailed version of this only online description, these authors propose that educators might respond if a greater effort were made to engage educators with data and anecdotal accounts of the success of peer educators.

What about the learners?

As I explored this history and what seems a frustrating pattern for those of us who have been influenced by the seeming promise of personalized progress systems and intelligent tutoring systems in a carefully controlled context, when turned loose in the complexity of schools and classrooms. The challenge of matching key elements of the controlled setting in which concepts are developed in applied settings is termed fidelity and is an issue in many fields (e.g., Trustschel and colleagues). I have struggled with this challenge in my own research, which has often focused on creating technology-facilitated study environments for college students enrolled in large introductory classes.

Cognitive research has accumulated a massive amount of evidence demonstrating the effectiveness of retrieval practice and the challenge that less capable learners are often much less aware of their specific knowledge gaps and a false sense of understanding (i.e., metacomprehension). In other words, less capable learners often don’t know what they don’t know and thus are very inefficient at remediating their problem areas. One way to provide retrieval practice and address poor metacomprehension is to provide practice tests. More sophisticated applications that make use of technology can also track weak areas so that these areas can be emphasized, link the student to remedial content when individual elements of information are not known or misunderstood, and even request students to predict the accuracy of their performance in an effort to increase awareness of strengths and weaknesses.

If you are interested in the details of this study, I have provided a citation below. The relevance of this study for the present post concerns the willingness of learners, college students in this case, to take advantage of a resource designed to improve their performance. The following graph is an easy way for me to make my point. Learners were divided into three groups based on course performance. For each of the three exams, the percentage of learners in each performance group who satisfy the stated goal of the study task, use but do not meet this standard, or do not use the study task is identified. There is a clear pattern: those performing the worst do not meet the study goal. Most persuasively in keeping with the other data reported in this post is the data on those who made no effort to use the system. It is possible trying but failing to reach the stated standard is related to understanding or aptitude, but failing to try, which should still be beneficial, is not.

As was the case in the 5% paper, those less in need of assistance participated more in a likely beneficial activity. In fairness, the “perceived suitability” of a learning opportunity proposed while vague offers a second possible explanation.

Summary

In this post, I consider the persistent “underutilization gap” in educational technology, where instructional tools—from commercial platforms to AI tutors—frequently fail to achieve their promised impact because they are either ignored by teachers or avoided by the students who need them most. It is true that the “5% problem highlights how efficacy data is often skewed by only including the small fraction of users who follow the system as intended, while struggling learners consistently participate the least in these personalized systems. Ultimately, I suggest that EdTech’s potential for personalized progress remains stalled by a lack of “fidelity” in real-world settings and a failure to align software with the practical “jobs” educators and students actually prioritize.

Citations:

Grabe, M., & Flannery, K. (2010). A Preliminary Exploration of Online Study Question Performance and Response Certitude as Predictors of Future Examination Performance. Journal of Educational Technology Systems, 38(4), 457-472.

Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of educational research, 86(1), 42-78.

Trutschel, D., Blatter, C., Simon, M. et al. (2023). The unrecognized role of fidelity in effectiveness-implementation hybrid trials: simulation study and guidance for implementation researchers. BMC Medical Research Methodology, 23, 116. https://doi.org/10.1186/s12874-023-01943-3

Searching the Scientific Literature

My work has always required that I locate, read, and keep track of the content of scholarly papers – mostly journal articles. This is typical of those of us whose academic interests combine research with teaching the core ideas of a science-related field of study. My personal focus was educational psychology and, even more specifically, reading skills and study behavior. Over the years with this foundation, I became interested in the role technology could play in these same topics and most recently, including how technology can effectively be employed in the reading, processing, and application of information by independent learners (i.e, learners who guide their own learning outside of formal classroom settings).

Over the course of 50+ years, the means by which those of us with such interests have experienced many changes in how we locate, read, and keep track of the content that forms of the basis and sometimes the outlet for our work. We usually purchased the journals we could afford and perused others in our local library. We once had postcard-sized forms we used to send requests to researchers to see if they had free copies of papers they would return as a professional courtesy. When you published a paper the journal at one time would provide you 50 or so individual copies you would use to participate in this exchange. Libraries have always had limited budgets and some of the less popular journals might be purchased as microfilm or microfiche that could be used to guide personal notetaking or perhaps be connected to a coin-fed “xerox” machine. Now, there are many more journals and libraries that still have limited budgets may buy access to digital collections of journals that allow patrons to download PDFs.

A challenge then and now in this process is how one goes about finding the specific articles and chapters you would read and collect. Libraries used to subscribe to services that provided intricately organized periodicals that would attempt to label research studies. If you didn’t peruse the journals on the “just arrived” section or the shelves, you would try to use these periodicals to guess what labels had been used to identify the content you might want to find in the stacks of your library or send for. Which articles you found, you would use the “reference” section to identify related work that seemed promising. We still do this, but it only works to find documents that are older than the one you happen to be reading at the time. As technology played a more and more important role in organizing content, large databases were developed that could be searched first by matching key words and now with AI capabilities that can respond to prompts that do not have to rely on exact matches to specific words or phrases.

This bring me to my goal in this post. There are now many tools available to both academically affiliated and independent learners to find what they hope will be useful resources. Some of these tools will now go further in summarizing what is found and even attempt to apply what was found in the creation of papers for different purposes. I am most interested in the location. I want to read the documents for a variety of reasons that I think are important, but I do not intend to discuss. I also have access to a research library that allows me to download PDFs of documents so I don’t need a service that will do that for me.

So, to summarize, where this leaves me personally. I am now retired, but retain online access to library resources. I do not have an easy way to work with library personnel or the most powerful tools available if I could work directly from a library. I do not want to spend a great deal of money on what I guess I would call “search tools”, but I have spent a good deal of time exploring a variety of free or inexpensive tools. I want to share insights related to my own experiences.

Here is one issue that may not be obvious to those with access to more expensive tools or those with no reason to explore as I have. Most of the literature I am interested in is behind a paywall. Many probably have been exposed to issues related to this reality. Why can’t citizens who, in a way, pay for much of this research through their taxes, read what the research looks like and what it concludes? Who makes the money from this component of academic scholarship? The researchers don’t get paid by journals for their papers. They are expected to review submitted papers for publication to identify high-quality work without compensation. Where does the huge fees libraries pay for access to scientific journals go?

These issues aside, most search engines that scour the Internet for information that users can search for cannot typically access content protected by paywalls. My personal issue is this how can I efficiently identify useful sources to read. Others have an even greater challenge. How can those without a “faculty pass” learn what recent research has to offer?

My current approach

I currently make use of the following tools/services:

SciSpace is the only one of these options I pay a subscription service to use so the rest have a free level or do not charge for any of the services provided. Again, I only need to locate citations as I have full access to a research library and I am not under the immediate pressure of working on a thesis or dissertation.

Comments

For articles behind paywalls, Google Scholar is usually my best starting point. It provides citations (sometimes incomplete in my experience). It also lists other publications that have cited the item you have targeted, which can be very useful. The citations include links to the journals in which the articles are published, which provide the full abstract and may or may not allow downloading the full article, depending on the individual journal’s policy.

Long-time Google Scholar users who have not explored the Google Labs option for Scholar should take a look. Rather than search terms, you can ask research questions much as you would with an AI tool. This approach allows a user to identify key topics and related issues. So, to stay focused on searching for journal articles on cyberbullying, I could request articles that examine school programs to combat it. After evaluating the results, the system identifies relevant papers and explains how each paper addresses your request.

Semantic Scholar provides features similar to Google Scholar (see below), but I have found it less effective in identifying sources I know exist. Given the overlap with Google Scholar, I use this service much less frequently.

I use Research Rabbit once I have identified a source I find valuable. Research Rabbit will then surface other sources from this entry point and show the citation map of how these sources are connected. This is also somewhat redundant, but the interconnection graphs are interesting.

SciSpace is useful for semantic searching and summaries of the contents of papers that are located. It is my impression that it is a hit-and-miss tool for locating documents on paywalled journals and I would not depend on it for this purpose.

The following sequence of images shows the return from the prompt “What is the average daily writing time for K12 students?”. The tool responds with a summary based on the best sources found and provides access to specific information for the sources it identified. Often, a PDF is not available for paywalled sources, but a citation is available, allowing me to try to find that paper in some cases.

Perplexity can help you find references and surface source links, but it is a general web answer engine, so it is usually not the best choice for systematically searching scholarly journal literature. I do use it to offer insights into how I might address topics for which references are less important.

When access to a journal is not available

When you have identified an article that looks good but is paywalled, there are still things you can try. Scholars may post prepublication versions of papers elsewhere. Just try a traditional search using the title of the article you want.

Some official repositories of alternatives can be identified through Google Scholar. After identifying an article of interest, check whether the response indicates there are alternative versions.

In this case, one of the alternatives (see following image) identifies a secondary source as ResearchGate, and this repository offers a full pdf of the article the journal protects. These are not illegal copies so you do not have to hesitate to make use of this option.

Summary

For my purposes, which involve paywalled content, Google Scholar is usually the best starting point because it is broad and often surfaces publisher pages, institutional copies, and free versions when they exist. It also indexes paywalled articles themselves, so you can still discover the citation even when the full text is inaccessible.

Semantic Scholar is also strong for discovery, but it focuses on open-access options where available and is less oriented toward paywalled content than Google Scholar.

Research Rabbit is very good once you already have one paper or author and want related literature through citation chaining, but it is less of a primary search engine for broad paywalled journal discovery.

SciSpace is useful for semantic searching and paper summaries, but it is better as a literature-review assistant than as the main tool for hunting down paywalled journal records.

Perplexity can help you find references and surface source links, but it is a general web answer engine, so it is usually not the best first choice for systematically searching scholarly journal literature.

Hallucinated Citations and Related Problems

Many of my posts are based on applying research to personal or classroom practice. I am retired, so I am no longer involved in experiments myself, but I now spend time reading both new and older published studies on a topic that interests me.

My change in location and social circles has led to some adjustments. I can’t walk across the street to a university library, though I still have access to online resources. Without students and colleagues, my interests are now far more self-driven and self-perpetuated. I have used Google Scholar since it was around, but the emergence of newer AI-supported tools for investigating the literature has been of great personal value.

This shift in how I locate the articles I read has exposed me to a strange phenomenon. I get excited when I find a reference relevant to a topic I have missed, particularly when it comes from an influential, productive researcher I follow. The title of this discovery sounds perfect and seems to promise just the type of evidence I have been looking for. I access my library’s online resources, call up the appropriate journal, and enter the title from the citation. The article isn’t there. Maybe the volume or the year of publication isn’t correct. I enter the title in Google Scholar to do a search and related articles appear, but there is no match for the specific paper I want. The citation that generated my excitement is very likely an AI hallucination.

I first wrote about this issue several years ago when AI was itself less sophisticated and this problem was probably more common. I include the link to this previous post because it contains multiple examples of what such hallucinations look like. I decided to revisit the topic after reading a recent Nature article examining this issue. The recent article did a good job of explaining why such hallucinations seem so real, but also raised questions related to how such hallucinations could appear in newly published research and how and why scholars might end up citing and developing arguments in their own papers related to some literature that does not exist.

The structure of a citation and why it results in hallucinations

The Nature study included a visual representation of a citation that I found helpful. I did not want to just cut and paste their examples so I had an AI tool develop something similar.

Think of a citation as consisting of several elements and understand that AI is not itself cutting and pasting what it offers in response to a prompt, but generates content. When this happens, some of the possibilities can result in fake outcomes.

Author may have published in this general area
Authors may have published together but not this paper
Words in the title are consistent with some of the work the author has done so are used to create the title
Pages fit with the date for this journal but are not appropriate
DOI (digital object identifier) – does not point to anything, but is similar to other DOIs for this journal

Ironically, trying to have an AI tool generate a plausible citation and identify its components also resulted in hallucinations (compare the image below with the one above). I tried multiple iterations to get what I wanted, but finally, I just had the tool generate the figure without lines, then used a different app to manually add them myself.

What are the responsibilities of an author?

How hallucinated citations appear in published work raises other serious issues. Possibly, the author who submitted the paper used a tool to build the reference list, but did not then check the final product. More seriously, the author used AI to write sections of a paper complete with citations and did not actually read the original papers.

Check your references

In my own efforts to explore relevant courses of action, I learned that many publications now rely on services that verify citation authenticity. I checked on the services and did not find anything that would be financially feasible for individuals. I did find that there are tools, some free, that will check a reference list.

CiteTrue

CiteTrue is a free online tool that accepts a list of citations and checks each component for accuracy. The following image shows what this looks like. I used part of the list of hallucinated citations I included in the previous post on this topic I describe above, and pasted these into the input box. The output indicated all were inaccurate and speculated about what was incorrect.

Personal Comment

This is not an issue I personally worry about, as I am no longer an active researcher. I do cite sources in some of my posts when a reader cannot follow a link to the source. I admit that not all of my sources follow the APA (American Psychological Association) format. This is due to my laziness. I do read all of the papers I cite, but putting together a citation is sometimes a manual process of accurately pulling together different pieces of information from the pdf for that source. I often copy the title from the pdf and paste it into Google Scholar and then use the citation for that source provided by Google. I am unclear how Google assembles citations in its systems, but they do not always follow the most recent APA guidelines. For example, many do not include a DOI or list the authors in different ways. I know the titles work because that is how I find the citations.

Lessons From the Classroom for Lifelong Learners

My interest in lifelong learners as a category of learners or as an expected focus of research activity has been frustrating. I have spent considerable time proposing that the research on note-taking and other study strategies can be translated, especially when applied to learners functioning independently outside of classrooms or in response to classroom-based assignments. When reading efforts to encourage strategies associated with concepts such as “personal knowledge management” (PKM) or “second brain”, I was surprised to be unable to find research associated with these proposals and often not even references to the literature I studied and tried to contribute to throughout my academic career. While at some level “learning is learning”, efforts to propose strategies for practice must consider differences in learning contexts and evaluate specific translations of controlled research as operationalized for these contexts.

Much of this post is based on the book “Make it Stick” (Brown, Roediger & McDaniel). I’ll explain why this book, rather than other sources, is used later. The following list of learning activities, which have been found to improve learning outcomes, is paraphrased from this source.

Purposeful learning should require:

Finding and retrieving information from memory (retrieval practice)
Spacing learning activities (spaced practice)
Mixing the focus of learning activities (interleaving)
Building personal knowledge – use existing knowledge structures to integrate new information, use new information to modify or extend existing structures
Testing understanding – recognizes failures of understanding

If you are unfamiliar with this terminology or the cognitive processes that are argued to be influenced by them, I offer the following set of links to extended posts I have written.

Retrieval practice

Spaced practice and desirable difficulty

Generative tasks and building understanding

Interleaving and desirable difficulty

When I think about useful learning activities, I do so from a different perspective. I focus on generative tasks. I define a generative task as an external task that increases the probability of productive internal (cognitive) activities. The recommendations proposed by Brown and colleagues would fit this more general category.

Brown and colleagues (Make it stick) define generation as an attempt to answer a question or solve a problem before being shown the answer or solution, and note that it makes the mind more receptive to new learning. This is a very different and I think less common use of how cognitive types use the phrase. See Fiorella and Mayers (see sources at the end) for a list of generative activities that have some overlap with those Brown and colleagues provide. The way I describe a generative activity makes sense to me. It is like asking a teacher what she has assigned a classroom activity. The teacher might use phrases such as “it gets my students to think” or “it requires my students to use”, but these are just different ways of saying internal (cognitive) behaviors are required.

Differences in the classroom and lifelong learning contexts

When the authors of Making it Stick argue that similar learning strategies can be applied by both students and lifelong learners, they note that the contexts in which these strategies are used differ and that the strategies are adjusted accordingly. What are some of these differences?

Structure

Classroom learning is typically more structured, with teachers deliberately designing activities that encourage generative activities. Teachers mostly select the goals and related tasks – lectures, syllabi, and tests. The time course of learning is driven by external decisions informed by the needs of the group and not individual learners.

Lifelong learning is personally structured and depends on the challenges of daily life or personal interests. Task selection and the continuation of the effort of learning is based on individual insights, accomplishments or motivation.

Most consistent source of motivation

With classroom learning, the educators shaping the tasks build in base-level, external sources of motivation (grades, evaluations).

Lifelong learners rely on personal challenges and goals to motivate their effort.

Task Initiation

In classrooms, the teacher often presents or assigns material first, then asks students to act on this material in some way.

In lifelong learning, it often begins with “learning from experience” – you face a problem, try to solve it, and only then seek answers. Learning as part of daily life is often referred to as experiential learning. Learning activities may also be driven by personal goals and interests.

Feedback

In classrooms, feedback is typically immediate and provided by the teacher (graded quizzes, comments).

In lifelong learning, the consequences of responding to life challenges are a source of feedback. Personal goals are typically evaluated based on reflection or in some cases the reaction of others if the goal is to produce a public product. Tasks that result in feedback can also be self-initiated.

Time Frame

The time frame for classroom learning is short and predictable. Exams and assignments have clear performance rate and these evaluations tend to be a couple of weeks or at most a couple of months.

When lifelong learners commit to a learning goal they may or may know when any knowledge or skill will be applied. In many situations, there is no guarantee a learning accomplishment will ever be applied.

The Insight from Make It Stick

The benefits to me of reading Make It Stick were in making the connection between the efforts to identify and evaluate learning strategies and both classroom and lifelong learners. As my own interests from shifted from classroom to self-directed learning I tried to find other educational writers who addressed this relevance and made some effort to explore how the same core of strategies might apply in each learning environment. My efforts to use search tools focused on scholarly research yielded close to nothing that I would label research focused on adults learning on their own. To bridge this gap I followed the speculation about the best approaches to personal knowledge management and second brain creation and use and translated research focused on classroom learning to interpret the underlying bases for the ideas being developed to guide personal knowledge management and learning.

Brown and colleagues were of interest because they argued that the same principles are key to all learning with emphasis and tactic of implementation varying with structural differences. The assumption of a commonality seems reasonable and not surprising, but the identification of lifelong learners and the recognition of structural differences in their learning tasks was unique.

This brings me to an effort to identify structural differences and how these differences alter strategy implementation.

Interaction with structural differences

In school, students rarely practice these strategies spontaneously; those who do “will need more than encouragement if they are to practice them effectively”. Teachers must build retrieval, spacing, etc. into the course design (cumulative quizzing, peer instruction, daily summarizations, etc.). Secondary students can be taught to take some responsibility for these skills using tactics such as flashcards, peer quizzing, and note-reworking techniques, such as those encouraged by the Cornell note-taking system.

Lifelong learners typically don’t have external structure, but they do have autonomy and clearer intrinsic goals. Application becomes designing their own schedules and systems, e.g., setting spaced reviews, using self-testing, and writing brief reflective summaries. Digitally based note systems often have tools encouraging random reviews and embedded AI allows the generation and evaluation of content-related questions.

Students’ metacognition and self-regulation are still developing, so these learners are more likely to misjudge what works and gravitate to easier illusions of learning like rereading. Imposed strategies and purposeful skill instruction are important. Adults often have stronger self-regulation, but must deliberately create constraints and routines or the strategies don’t happen.

Existing knowledge and life experiences

Students tend to have less developed background knowledge, so adding complex strategies can create working memory overload when they’re already struggling. So in K–12/undergrad, teachers often need to scaffold: start with simpler retrieval (short low-stakes quizzes, guided questioning) and gradually move toward more independent study behavior.

Lifelong learners usually work from richer life experiences and probably more formal education, so elaboration, self-questioning, and application can be more generative right away, and spacing can be stretched further because there is more prior knowledge to attach to. Goals are also based on a longer time frame for potential application.

Why: The same strategy (e.g., self-testing) places different cognitive demands depending on knowledge and fluency; novice students need more support and tighter feedback loops.

Motivation and time span

For students, motivation is often extrinsic (grades, exams). This external focus is based on a predetermined curriculum and external expectations for what must be learned. This can make the challenges of learning feel like unfair obstacles, so teachers must explain why they’re spacing, mixing, and testing—making assessment “a positive learning experience” rather than just judgment. I have a long-term interest in what is often called “mastery learning” which is a competency-based system that allows multiple opportunities to demonstrate competence. This approach would fit well with promoting the benefit of feedback, but unfortunately is not a common approach.

Lifelong learners are generally motivated by relevance (job, hobby, citizenship). This aligns well with an emphasis on generative activities such as write-to-learn activities and linking and adding to original notes.

Why: With adults, you can lean hard on immediate application and self-explanation because the “why this matters” is obvious; with students, you often must cultivate that connection. The lifelong learner accepts that what they learn may not have immediate value and that skills and knowledge often find an application, impact motivation, and create a focus on improving the retrievability of notes and on linking, reviewing, and reworking notes over time as interests change and goals become evident.

Summary

This post builds on a short section of “Make it stick” that recognized that lifelong learners and students in formal educational settings likely apply similar learning strategies adapted to unique characteristics of their settings. To me, just the mention of lifelong learners and students in the same publication seemed unique. Certainly, the brief effort to analyze a few interactions of learning strategies with these different settings is very uncommon. I admit that research of the same quality would be very difficult to evaluate the effectiveness of proposed strategies for lifelong learners, but at least the effort to speculate about how techniques of proven value in classrooms apply elsewhere seems useful.

Sources

Brown, P. C., Roediger III, H. L., & McDaniel, M. A. (2014). Make it stick: The science of successful learning. Harvard University Press.

Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. Educational Psychology Review, 28(4), 717-741.

Less Expensive Equipment Alone Won’t Take Down Chromebooks

I have been an Apple in education guy since the 1980s. The potential of the new NEO for that market immediately caught my attention. So many podcasts have speculated about the potential of the Neo in the marketplace and one of the participants on Macbreak Weekly changed my mind concerning the Neo as a Chromebook killer. She argued that the price point of the Neo or lower-end iPads is only part of what the school-based tech people consider. Apple or Microsoft has no viable alternative to Google Classroom and the structure and security issues addressed are worth a lot to school tech decision makers.

How popular are different tech products in K12?

You might think providing data on the recent history of school purchases of Apple, Windows, and Chromebook devices would be simple. Some source must have found this topic to be of interest. I have tough expectations for what this would look like with Chromebooks showing a sharp rise in recent years. What I was less certain I understood was the comparative tracks of Apple and Windows equipment. I have been retired now for about a decade so I have spent only a little time in schools. I expected popularity comparisons would show Apple somewhere between Chromebook and Windows machines.

There are no “official statistics,” and some of the most carefully acquired have value to businesses interested in the education market, and require you to purchase the reports (e.g., Futuresource). Data I could locate was inconsistent and no source fit my expectations. After multiple searches, I asked Perplexity to generate a graph for the 20 year period I wanted and that graph appears below. I did find similar general descriptions elsewhere to my surprise and the issue here is whether the new and less expensive Apple computer will change this trend.

Why Chromebooks will likely continue the tech of choice?

So, Google’s Chromebooks have maintained a stranglehold on K-12 classrooms. While the Neo is a device that brings the prestige and power of macOS to a price point that schools can actually afford, the decisions those who make purchases depend on more than the cost of the equipment.

The real reason Chromebooks will likely still be preferred isn’t just the lower price tag; it is the infrastructure of Google Classroom and the Google Workspace for Education ecosystem. Until Apple builds a direct, functional competitor to Google Classroom, the Neo is just a nice laptop in a room where everyone is already tied into a different system.

When a school district buys a large number of devices, often in the thousands, they are looking beyond a reasonable price point and hardware sophistication. Organizations also consider manageability and deployability – how to oversee how devices are used and how to set them up quickly and efficiently. Apple has always focused on individual users. When the machine assigned to an individual has an issue or selects one from a classroom card, each student simply signs in to the new machine, it is personalized, tabs, documents, and settings like the last time they connected. The design of Apple equipment maintains an individual’s priorities and content on that individual’s machine.

The Google Classroom

Google Classroom was designed with an understanding of the school day and classroom tasks. It connects to Google’s online services – Google Drive, Calendar, and Meet and gives the teacher some level of immediate access to student accounts. Teachers have a way to distribute and grade assignments. Google Classroom allows a teacher to “make a copy for each student” with one click, see real-time progress on an essay, and provide instant feedback. The system works great because Google owns both the productivity suite (Docs/Sheets) and the management layer (Classroom).

Built for Collaboration

Google Docs was built for the web and for collaboration; it was built for twenty students to be in the same document at the same time without the system crashing or creating “conflicted copies.” Students can peer-edit, work on group slides, and share data in real-time. These capabilities can be accessed on the devices from other companies, but if classroom tasks are heavily dominated by tasks Google makes easy why add the complications of equipment that is not as easily integrated?

The Cost Issue at the Level of the System

As I understand it, Google Classroom itself is generally provided at no additional cost to schools as part of Google Workspace for Education. The cost of Google Workspace depends on what schools want. There is a free tier for eligible institutions that provides the core tools such as Docs, Drive, Gmail, and Meet. Schools can also pay for other features that meet common needs, such as advanced security, analytics, admin controls, and additional teacher tools. I couldn’t find pricing details as the charge to the schools depends on the number of classrooms and teachers.

Additional Comments

Teachers don’t choose platforms; districts do. Procurement decisions happen at the IT/admin level, not the classroom level. In considering Apple versus Microsoft in the business environment, the same issue seems to apply. In schools. Google won those relationships early and aggressively in the 2010s, and switching costs are now enormous — not technically, but now in terms of retraining staff and migrating years of curriculum materials. It seems to me Windows and Apple equipment will continue to have value in specific school applications but overcoming the inertia of a device and services suited to the most common tasks would be a massive challenge. If we get to AI on devices, perhaps things will change.

Other companies have attempted to develop competing services. For example, Microsoft has Teams for Education. This product is regarded as capable, but teachers reportedly find Google Classroom much easier to use.

Summary

Understanding why schools adopt a given type of tech hardware may not be as apparent as some may assume. This post argues that even given a similar price point other factors are important and Google has a lead when it comes to the basics of common classroom practice.