{"id":7721,"date":"2026-06-14T17:40:14","date_gmt":"2026-06-14T17:40:14","guid":{"rendered":"https:\/\/www.learningaloud.com\/blurts\/?p=7721"},"modified":"2026-06-14T17:46:06","modified_gmt":"2026-06-14T17:46:06","slug":"input-token-cost-of-rag-focused-ai","status":"publish","type":"post","link":"https:\/\/www.learningaloud.com\/blurts\/2026\/06\/14\/input-token-cost-of-rag-focused-ai\/","title":{"rendered":"Input Token Cost of RAG-Focused AI"},"content":{"rendered":"\n

Andrew Karpathy has identified an issue with the way those of us who make use of AI to interact with content we have collected (RAG- retrieval-augmented generation). Each time we submit a prompt to query our designated content, the AI resets and reprocesses it to address our query. As I will eventually demonstrate, this repeated input processing can be significantly demanding, and if you pay by the token or have a plan that has a token limit, this repetitive processing can be quite expensive. Karpathy proposed that you have a reasonable understanding of what you generally want to get from your curated content, you can find and export this content once and then focus on an organized wiki (Karpathy\u2019s term) of the types of information you anticipated would be of value. Karpathy\u2019s concept and its multiple variations have generated considerable attention, including interest in running them on personal computers. It is not my intent to focus on the how-tos of the LLM-wiki process<\/a>. I am intrigued and I did purchase a mac-mini I can dedicate to exploring this process. The popularity of this idea has made purchasing Mac Minis very popular and the machine I purchased in April is not scheduled for delivery until mid-July. I take this as an indication of the potential usefulness of this approach.<\/p>\n\n\n\n

What I want to document here is the inefficiency of RAG. Unless you are a heavy AI user and unless you make frequent use of AI to explore a large pool of content that you designate, you are unlikely to encounter this challenge. I don\u2019t presently have a $20 a month account. Instead, I have a lower-max account (Abacus.AI<\/a>) and pay for several accounts that allow me to upload content for exploration (Mem.ai<\/a>, Recall.ai<\/a>). The situation that allowed me to carefully analyze token use was the API plugin I used to apply AI to content<\/a> I have stored and organized in Obsidian. This plugin accesses Claude via an API and a \u201cpay as you use\u201d plan. Anthropic provides detailed usage data, enabling interesting analyses.\u00a0<\/p>\n\n\n\n

Claude via Obsidian<\/strong><\/p>\n\n\n\n

I have used Obsidian to store notes and highlights<\/a> I generate from my reading for several years and I have collected a sizeable body of content. The folder on note-taking I am using in the following demonstration contains 132 notes totaling 38,208 words. The following demonstration takes the following approach. The data available from my use of Claude to prompt this folder with prompts is saved by the day so for purposes of this little experiment, I issued one prompt a day. <\/p>\n\n\n\n

Prompts by the day:<\/p>\n\n\n\n

Day 1: Under what circumstances is notetaking a generative activity <\/p>\n\n\n\n

Day 2: What have research studies shown regarding the relative effectiveness of taking notes by hand or using a keyboard? <\/p>\n\n\n\n

Day3: Using my notes, generate separate responses to the following prompts.\u00a0Under what circumstances is notetaking a generative activity? What have research studies shown regarding the relative effectiveness of taking notes by hand or using a keyboard?<\/p>\n\n\n\n

Logic of this approach. The data retained by Anthropic for use of Claude (4.6 in this case) is totaled daily. In addition to the charge to my account (one chart), a daily token total is provided, along with a running total of Tokens In and Tokens Out. By tracking the totals in combination with the charts provided, it is possible to demonstrate the problem others have demonstrated with AI and RAG – the processing required for inputting the content is required each time the system is used hence asking for a response to the same two requests following a single input will save a considerable number of tokens (i.e., day 3 token use will be nearly the same as either day 1 or day 2). So, the cost and token usage on day 3 were only slightly higher than on either day 1 or day 2.\u00a0<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

Summary<\/strong><\/p>\n\n\n\n

I must admit I was surprised at the cost of asking an AI tool to focus on content a user provides instead of asking a similar question to be answered from that tool’s own stored \u201cknowledge\u201d. These seemed basic requests someone might make many times a day in exploring with AI, but the focus on content the tool must first tokenize adds substantially to the cost. Given this cost, the Karpathy wiki strategy makes a lot of sense.\u00a0At present as a temporary strategy, I store the output from my Obsidian-related AI queries in a separate Obsidian folder and I can then target this folder for a lower Input Token cost.<\/p>\n\n\n\n

One more thing. Claude’s response to my query was great and would have been quite helpful had I been focused on creating a post. The structure of the output was, I would say, insightful. I have appended the response to my query about taking notes by hand versus on a digital device (see below).<\/p>\n\n\n\n

*****<\/p>\n\n\n\n

Notetaking – Hand vs Keyboard<\/p>\n\n\n\n

Hand vs. Keyboard Note-Taking: What Your Notes Show<\/p>\n\n\n\n

Your notes contain extensive coverage of this topic across multiple studies. Here is a synthesis of the key findings:<\/p>\n\n\n\n

The Foundational Study<\/p>\n\n\n\n

The landmark study by Mueller & Oppenheimer (2014), documented in Pen is mightier than the keyboard, found that keyboard users recorded more content but that longhand note-takers performed better, particularly on conceptual understanding items. In Study 3, which included a week delay with note review, differences favoring handwritten notes emerged specifically when review was allowed.<\/p>\n\n\n\n

The Generative Processing Explanation<\/p>\n\n\n\n

Several of your notes describe the primary theoretical explanation for a longhand advantage. Because typing speed (~33 wpm) exceeds handwriting speed (~22 wpm), as noted in transcription speeds, longhand note-takers are forced to summarize and paraphrase rather than transcribe verbatim. This engages generative processing\u2014connecting new information to prior knowledge and reorganizing ideas\u2014which promotes deeper learning. As described in Luo, Linlin, Kenneth A. Kiewra, Abraham E. Flanigan, and Markeya S. Peteranetz2018:<\/p>\n\n\n\n

“Laptop note takers had a transcription orientation and recorded mainly verbal information… whereas longhand note takers had a generative orientation and used a combination of verbal and spatial note-taking strategies.”<\/p>\n\n\n\n

The Luo et al. (2018) study further found that keyboard users recorded fewer visual notes (charts, graphs) and that the process function was more important for keyboard notes while the product function (review) was more important for longhand notes.<\/p>\n\n\n\n

Contradictory and Nuanced Findings<\/p>\n\n\n\n

Your notes reveal that the research is far from settled:<\/p>\n\n\n\n