Blurts

Educators should find this segment from 60 Minutes informative. The focus is on how the Kahn Academy makes use of Ai. Despite the claims that AI does the work for students, Kahnmigo is designed to guide students as the learn. 

60 Minutes AI in Education segment

One insight I took away from the year or so we suffered with COVID was that the public has a poor understanding of what I would describe as the process of science. This failed understanding allowed opportunistic politicians to attack scientists and sowed a resentment of expertise, academic institutions, and well-established scientific findings that persist to this day. Although I was involved in research as part of my profession, I had never really thought much about the assumptions nonscientists make about science. I have since decided most folks think of science as a body of knowledge – facts, causal relationships – and not as the process of accumulating such insights. The idea of best guesses (hypotheses) that get refined over time is missing. When a hypothesis is applied in a situation such as the COVID epidemic and fails, the insight that the best evidence to that point was incomplete and best guesses based on what was known are shown to be wrong is not a failure, but what some might call a feature. Science advances when what is known is refined and made more specific so that a broader range of questions can be accurately answered.

Humans are complex and their behavior, which involves personal decisions, is difficult to study

I was an educational psychologist and understand that some think of the social sciences as a different kind of science, if a science at all. I was trained as both a biologist and a psychologist and have some sense of the challenges of at least these different fields. When I taught the Introduction to Psychology course, I used to challenge the hierarchical ranking many students had about the sciences by noting that when a chemist predicts how a particular chemical reaction will proceed, she doesn’t have to give a thought to whether the chemicals might not feel like reacting today. The challenge I suggested is that human behavior is complex and can depend on so many factors that identifying useful principles to guide useful applications sometimes feels like it is impossible. 

Why can’t psychologists conclude the best way to do something? Why do research studies reach conflicting outcomes?

Here is an example of this challenge that those who follow my posts may recognize. I am interested in the use of digital technologies in studying. Several recent subtopics have involved questions of whether it is better to perform a certain task using technology or a more traditional method. Is it better to take notes on paper or using a laptop? Is it better to read a paper version of a textbook or a digital version? 

These research questions have gained enough popularity to reach the popular press, and in such sources, the general conclusion has been that reading and note-taking on paper are the preferred approaches. If you read the secondary sources, you might conclude that the paper/screen issue has been decisively resolved. The primary sources – the journal articles describing individual studies and also the integration of these results through what are called meta-analyses – provide a more nuanced perspective. I support the use of technology in reading and note-taking, and I think I can make a solid case based on a careful reading of many of the original studies. I make this claim because I pay attention to the methods used in the individual studies. The popular press does not describe the methods of the research associated with their summaries.

Why review the Method Section?

The methods section of a psychology paper provides a detailed description of how the study was conducted, including the participants, materials, design, and procedures used. This section is crucial because it allows other researchers to understand and potentially replicate the study.

The details can lead to important differences in the results of a study and must be considered in the interpretation of these results. For results that advocates want to suggest should guide practice, do the characteristics of the Methods match important characteristics that apply to practice? 

What follows are some thoughts on topics that are addressed in most Methods descriptions that apply to my sample paper or screen topic.

Participants – The description of participants addresses who took part in the study, how they were selected, relevant characteristics (e.g., age, gender), and the sample size. I am often interested in how adults outside of a formal education setting learn from reading and attempt to retain what they have read. I also studied college students and their study behavior. There are many ways in which the characteristics of these groups differ from the learners in other situations. Their goals are different; e.g.,  performance on learning related to externally selected content and externally determined expectations vs. personally selected content and goals. There may be differences in previous experiences that are relevant. Have participants read many books in a digital format before participation in the research study, and have they highlighted and taken notes while reading in this way? 

Materials – If you read the research coming out of different labs, you may note that a particular lab will use the same story or chapter from a given book over and over in a series of experiments. I have had the experience of trying to extend the research from such labs and find that with different materials, I cannot even get what seems like a direct replication to produce the same results. My experience has not been that a completely opposite outcome occurs (a different treatment is found to be superior), but rather that no difference is demonstrated in my attempt at a replication. This is one of the reasons that replication and meta-analyses are important. Consider the different types and topics of reading material that are involved in educational settings and how differences may be important. For example, you can take notes more quickly using a keyboard. Would transcription speed differences matter when taking notes in a biology and a philosophy class, where factual density and level of abstraction could be very different?

Procedure – The procedure outlines the step-by-step process of how the study was conducted, including instructions given to participants and any interventions or tasks performed. I think an important consideration is whether the procedure involves the participant in a task with characteristics similar to those of the task to which results will hopefully generalize. For example, if a note-taking study is to generalize to the study process of students preparing for a course exam, the time between the note-taking activity and the review activity in preparation for an examination may be important. Many studies include the note-taking, note-reviewing, and assessment activity within the same research setting. Obviously, running a study in which participants would take notes during one session, review notes a week later, and then complete an examination the following day would be cumbersome and very rare. Yet, notes are obviously more useful in prompting recall after a delay, and if this factor is not part of the Method the results may not be as useful. Older students can type faster than they can write by hand, resulting in more notes when using a digital device. It makes sense that more notes would provide a greater advantage after a delay, encouraging the expectation that taking digital notes would be an advantage. Yet, this seems a less frequent research outcome. Perhaps the mismatch between the common research procedure and the intended application (what approach should college students take) leads to flawed advice. 

Nearly all of the research involves a procedure intended to mimic the classroom setting in which the content to be learned is provided, and the examination of learning is designed to assess mastery of this assigned content. Those of us who take notes outside of the classroom have far less certain goals. The concepts of Personal Knowledge Management and the creation of a Second Brain involve the recording of notes at one point in time without a clear use for such notes and an uncertain timeline. The application of the notes is something as vague as “I might find this concept or idea useful in a future writing project”, but the type of ideal note taken under such circumstances (e.g., atomic notes or permanent notes) might have different characteristics than notes taken for next week’s examination. There is very little research focused on non-academic note-taking, and it seems possible that using conclusions from the copious literature on academic note-taking should be applied with care.

One final point about Methods

There is one further point of a more general nature I would like to offer. We are in a time when summarization tools are being promoted as an alternative to reading the studies scholars cite or even just the entirety of any article one might encounter. Yes, the actual reading of say the 20-30 studies cited at the end of a research paper or textbook chapter takes a very long time and summaries are often good enough. However, the details do matter and the details revealed in a careful analysis of the Method section are often quite useful in understanding the frequent inconsistencies reported in the literature. If you have only a vague understanding of the Methods that were involved, you have only a vague understanding of why the results turned out as they did. 

*****

Sample studies related to the paper vs keyboard topic

Clinton, V. (2019) Reading from paper compared to screens: A systematic review and meta-analysis. Journal of Research in Reading, 42: 288–325. https://doi.org/10.1111/1467-9817.12269.

Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23-38.

Kong, Y., Seo, Y. S., & Zhai, L. (2018). Comparison of reading performance on screen and on paper: A meta-analysis. Computers & Education, _123_, 138-149.

Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological science, 25(6), 1159-1168.

There is an opinion that those of us who teach psychology often have to tolerate. I would translate this opinion as the belief that the strategies we spend our time researching could be identified by common sense. Some assume that what we teach and test is about the theory and research behind such practical applications and these topics are not especially valuable because most folks would function in their daily lives according to common sense and achieve very similar results. My specialization, which includes courses intended to prepare future teachers but also talking to students about their learning strategies, frequently confronts this attitude. Both future teachers and college students have experienced the educational process as learners for many years and so often feel well-informed when it comes to the strategies they should apply for success. Is this the case?

Many examples demonstrate that the behavior of learners is far from optimal. An obvious example would be how students spend their time. Studies of massed versus distributed practice clearly show the long-term benefit of distributing the investment of study time. Yet, cramming for exams is common. I understand that the issue here is often one of motivation rather than the understanding of learning. Simply put, long-term retention may not be what many students prioritize. Their goal may be to maximize their performance on tomorrow’s test with far less interest in whether they can recall the content next year. If students understand this and don’t care, then cramming may not represent a good test of the claim I am making. The problem I intend to describe here seems different, but even cramming ignores the reality that courses build on each other and background knowledge has a significant impact on success with future related courses.

When it comes to studying and even to what has become my most recent interest, nonacademic note-taking and the application of knowledge gleaned from personal reading, there are examples more easily related to the lack of understanding of how learning works. Such examples come from situations in which a learner decides between two strategies that require the same commitment of time and chooses to focus on the less productive strategy. When such examples can be identified, this is not the end of what researchers want to learn. Why are poor choices common? Are learners unaware of their poor choice and the issue might then be resolved through greater awareness or are other more challenging issues involved? To get back to the issue I first identified, why doesn’t common sense solve this challenge? 

The value of retrieval practice represents an example of the situation I have just described and this phenomenon has received a great deal of recent attention. An easy way of operationalizing one task that involves retrieval practice is to describe this task as testing. The purpose is not testing for a grade, but just for benefits generated by the experience of testing. This task might be controlled by an educator who spends some class time having students use clickers or their cell phones to respond to questions based on assigned reading or classroom presentations. Testing might be self-imposed with one common strategy being the use of learner-created “flash cards”. 

So, to implement the type of research scenario I have just claimed demonstrates that learners make poor choices, it is necessary to first know that one strategy is more productive than the other. A simplified description of such research would involve two groups assigned at random to conditions that either involve the review of notes and book highlights or the same content with some of the time spent completing test items. When both groups work for the same amount of time, the group that included retrieval practice (testing) performs better on a delayed test of what was learned. 

Once established, the focus becomes what do learners do when study “in the wild” or when offered the option of a study session including testing or a study session focused on the study content without testing? A surprising proportion avoid the inclusion of some form of testing as part of their study sessions.

The perception of effort required and perceived effectiveness

Retrieval practice has attracted a great deal of recent emphasis and again the question is why learners underutilize tasks that involve retrieval. One hypothesis relates to how learners interpret the difficulty of different study methods. Here is how this has been investigated. 

Participants are involved in several studies in which they rate two study strategies (i.e., study involving or not involving testing) for the mental effort required and the perceived effectiveness (Kirk-Johnson and colleagues, 2019). It appears the more learners perceive a study strategy as mentally demanding the less effective they perceive it to be and the less likely they would select it for future use. The researchers described this as the misinterpreted-effort hypothesis.

How do learners interpret testing?

Morehead and colleagues (2015) investigated knowledge of study strategies by asking both educators and college students about different methods. The researchers asked college students how frequently they used a list of study methods including rereading, flashcards, testing, studying with friends, reworking your notes, etc., and in a second study they presented students with scenarios in which two study options were listed and asked the participants to select which they would most likely apply. When the researchers focused on teachers, they asked about the value of different study methods. These methods determined that both college students and instructors recognized testing as productive, but follow-up questions indicated the value was perceived to be the identification of which topics still needed more work rather than recognizing that the experience of trying to recall information to answer test questions results in better understanding and retention. Of course, the identification of what to focus on is important to be efficient in the use of time, but this perspective underestimates the impact of retrieval practice as an efficient learning experience.

What to make of all of this?

It seems important to me that more advanced students appreciate the value of testing even if they focus on a different and valid explanation for why testing experiences are valuable. Effective metacognition partly depends on personal insights into how learning works. It seems more obvious that testing identifies failures of understanding than that testing builds retrieval strength so recognizing the direct benefits of retrieval practice is much less likely to be apparent from daily life experiences. Educators are in the best position to address this situation. While present understanding of the values of testing reflects a limited perspective on the part of educators, the formal training of educators should address this issue and educators should then be in an ideal position to include retrieval practice tasks as they interact with students and to encourage students to make greater use of retrieval practice as part of the time they devote to self-directed study.

The message that “what doesn’t kill you makes you stronger” is a popular way to counter difficult things in other parts of our lives and perhaps a variant of the advice can be promoted when it comes to how we try to learn. The academics have been pushing what they describe as “desirable difficulty” as a more cognitively-based interpretation of this principle. Difficulty is not an end in itself, but in situations in which tasks require certain important cognitive activity the engagement of these activities may be perceived as more demanding than not engaging such behaviors. The need is to interpret this effort appropriately.

References:

Kirk-Johnson, A., Galla, B. M., & Fraundorf, S. H. (2019). Perceiving effort as poor learning: The misinterpreted-effort hypothesis of how experienced effort and perceived learning relate to study strategy choice. Cognitive Psychology, 115,101237. http://dx.doi.org/10.1016/j.cogpsych.2019.101237

Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin & Review, 14, 219 –224. http://dx.doi.org/10.3758/BF03194055

Morehead, K., Rhodes, M. G., & DeLozier, S. (2016). Instructor and student knowledge of study strategies. Memory, 24, 257–271. http://dx.doi.org/10.1080/09658211.2014.1001992

Willingham, D. T. (2023). Outsmart your brain: Why learning is hard and how you can make it easy. Simon and Schuster.

Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399-435.

It is ironic that despite reading and writing on Medium for several years now, my notes and highlights from Medium were the last source I was able to bring into my personal knowledge management system. As I have explained in a previous post, I want the long-term storage and organization of my reading to end up in Obsidian, and Readwise plays an important in acting as an intermediary to that goal. Readwise can be used to automatically collect annotations and highlights from certain sources (e.g., Kindle books) and then automatically export this content to Obsidian. Recently, these processes were enabled for Medium.

The process of moving highlights and annotations from Medium to Readwise is relatively easy. The import from Medium option now appears among the Readwise options (first image). 

To export automatically, there is an export option for Readwise that sends content to Obsidian. Which of these options you use may depend on what you want to store in Obisidian. If you read a large variety of stories in Medium, you may have a specific focus for Obsidian and prefer to select just that content from Readwise that is consistent with this focus. 

Suppose you want to export the additions from a specific imported source. In that case, the stored content in Readwise will appear within the list of imported articles and you then use the inverted caret to reveal the options and select “Export highlights”. A file containing the highlights and notes will be downloaded as a file to your device and you can open this file and add to Obsidian or any other storage tool.

One more comment about the material you have stored in Readwise. Both Readwise and Medium are online tools. You can move from a stored item in Readwise back to Medium. To do this, use the inverted caret that appears at the top corner of a specific item in Readwise and select the “View in Medium” option.