One insight I took away from the year or so we suffered with COVID was that the public has a poor understanding of what I would describe as the process of science. This failed understanding allowed opportunistic politicians to attack scientists and sowed a resentment of expertise, academic institutions, and well-established scientific findings that persist to this day. Although I was involved in research as part of my profession, I had never really thought much about the assumptions nonscientists make about science. I have since decided most folks think of science as a body of knowledge – facts, causal relationships – and not as the process of accumulating such insights. The idea of best guesses (hypotheses) that get refined over time is missing. When a hypothesis is applied in a situation such as the COVID epidemic and fails, the insight that the best evidence to that point was incomplete and best guesses based on what was known are shown to be wrong is not a failure, but what some might call a feature. Science advances when what is known is refined and made more specific so that a broader range of questions can be accurately answered.
Humans are complex and their behavior, which involves personal decisions, is difficult to study
I was an educational psychologist and understand that some think of the social sciences as a different kind of science, if a science at all. I was trained as both a biologist and a psychologist and have some sense of the challenges of at least these different fields. When I taught the Introduction to Psychology course, I used to challenge the hierarchical ranking many students had about the sciences by noting that when a chemist predicts how a particular chemical reaction will proceed, she doesn’t have to give a thought to whether the chemicals might not feel like reacting today. The challenge I suggested is that human behavior is complex and can depend on so many factors that identifying useful principles to guide useful applications sometimes feels like it is impossible.
Why can’t psychologists conclude the best way to do something? Why do research studies reach conflicting outcomes?
Here is an example of this challenge that those who follow my posts may recognize. I am interested in the use of digital technologies in studying. Several recent subtopics have involved questions of whether it is better to perform a certain task using technology or a more traditional method. Is it better to take notes on paper or using a laptop? Is it better to read a paper version of a textbook or a digital version?
These research questions have gained enough popularity to reach the popular press, and in such sources, the general conclusion has been that reading and note-taking on paper are the preferred approaches. If you read the secondary sources, you might conclude that the paper/screen issue has been decisively resolved. The primary sources – the journal articles describing individual studies and also the integration of these results through what are called meta-analyses – provide a more nuanced perspective. I support the use of technology in reading and note-taking, and I think I can make a solid case based on a careful reading of many of the original studies. I make this claim because I pay attention to the methods used in the individual studies. The popular press does not describe the methods of the research associated with their summaries.
Why review the Method Section?
The methods section of a psychology paper provides a detailed description of how the study was conducted, including the participants, materials, design, and procedures used. This section is crucial because it allows other researchers to understand and potentially replicate the study.
The details can lead to important differences in the results of a study and must be considered in the interpretation of these results. For results that advocates want to suggest should guide practice, do the characteristics of the Methods match important characteristics that apply to practice?
What follows are some thoughts on topics that are addressed in most Methods descriptions that apply to my sample paper or screen topic.
Participants – The description of participants addresses who took part in the study, how they were selected, relevant characteristics (e.g., age, gender), and the sample size. I am often interested in how adults outside of a formal education setting learn from reading and attempt to retain what they have read. I also studied college students and their study behavior. There are many ways in which the characteristics of these groups differ from the learners in other situations. Their goals are different; e.g., performance on learning related to externally selected content and externally determined expectations vs. personally selected content and goals. There may be differences in previous experiences that are relevant. Have participants read many books in a digital format before participation in the research study, and have they highlighted and taken notes while reading in this way?
Materials – If you read the research coming out of different labs, you may note that a particular lab will use the same story or chapter from a given book over and over in a series of experiments. I have had the experience of trying to extend the research from such labs and find that with different materials, I cannot even get what seems like a direct replication to produce the same results. My experience has not been that a completely opposite outcome occurs (a different treatment is found to be superior), but rather that no difference is demonstrated in my attempt at a replication. This is one of the reasons that replication and meta-analyses are important. Consider the different types and topics of reading material that are involved in educational settings and how differences may be important. For example, you can take notes more quickly using a keyboard. Would transcription speed differences matter when taking notes in a biology and a philosophy class, where factual density and level of abstraction could be very different?
Procedure – The procedure outlines the step-by-step process of how the study was conducted, including instructions given to participants and any interventions or tasks performed. I think an important consideration is whether the procedure involves the participant in a task with characteristics similar to those of the task to which results will hopefully generalize. For example, if a note-taking study is to generalize to the study process of students preparing for a course exam, the time between the note-taking activity and the review activity in preparation for an examination may be important. Many studies include the note-taking, note-reviewing, and assessment activity within the same research setting. Obviously, running a study in which participants would take notes during one session, review notes a week later, and then complete an examination the following day would be cumbersome and very rare. Yet, notes are obviously more useful in prompting recall after a delay, and if this factor is not part of the Method the results may not be as useful. Older students can type faster than they can write by hand, resulting in more notes when using a digital device. It makes sense that more notes would provide a greater advantage after a delay, encouraging the expectation that taking digital notes would be an advantage. Yet, this seems a less frequent research outcome. Perhaps the mismatch between the common research procedure and the intended application (what approach should college students take) leads to flawed advice.
Nearly all of the research involves a procedure intended to mimic the classroom setting in which the content to be learned is provided, and the examination of learning is designed to assess mastery of this assigned content. Those of us who take notes outside of the classroom have far less certain goals. The concepts of Personal Knowledge Management and the creation of a Second Brain involve the recording of notes at one point in time without a clear use for such notes and an uncertain timeline. The application of the notes is something as vague as “I might find this concept or idea useful in a future writing project”, but the type of ideal note taken under such circumstances (e.g., atomic notes or permanent notes) might have different characteristics than notes taken for next week’s examination. There is very little research focused on non-academic note-taking, and it seems possible that using conclusions from the copious literature on academic note-taking should be applied with care.
One final point about Methods
There is one further point of a more general nature I would like to offer. We are in a time when summarization tools are being promoted as an alternative to reading the studies scholars cite or even just the entirety of any article one might encounter. Yes, the actual reading of say the 20-30 studies cited at the end of a research paper or textbook chapter takes a very long time and summaries are often good enough. However, the details do matter and the details revealed in a careful analysis of the Method section are often quite useful in understanding the frequent inconsistencies reported in the literature. If you have only a vague understanding of the Methods that were involved, you have only a vague understanding of why the results turned out as they did.
*****
Sample studies related to the paper vs keyboard topic
Clinton, V. (2019) Reading from paper compared to screens: A systematic review and meta-analysis. Journal of Research in Reading, 42: 288–325. https://doi.org/10.1111/1467-9817.12269.
Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23-38.
Kong, Y., Seo, Y. S., & Zhai, L. (2018). Comparison of reading performance on screen and on paper: A meta-analysis. Computers & Education, _123_, 138-149.
Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological science, 25(6), 1159-1168.
You must be logged in to post a comment.