🤫 AI Lies — Surprise!

Last week, I ran an experiment with Claude, my LLM of current choice: analyze a curriculum document written by humans for humans, leavened throughout with human quotes to speak to human experiences of navigating the very human experience of transition. Then create a fake quote from a fake human on "the best ways to use AI without giving up the capacity to think," mirroring the tone of the real quotes in the document and trying to make the synthetic one sound real enough to be believable.

Claude's Opus 4.7, one of the most sophisticated consumer-grade models on the market right now, happily did my bidding.

The result made me laugh. Nod. Grimace.

--

I am still not sure what to think of AI. It is now both radioactive to a growing portion of the population and revolutionary to another. It is, in my daily life, simultaneously useful and repugnant.

I fear that we are often too reductive with our descriptions of AI and its large language models, trying to define it in single phrases when there's actually much more nuance underneath. It is, I believe, possible for AI to be both helpful and destructive, depending largely on the application. Like many, I've developed a viscerally negative reaction to seeing AI "art" — as a musician, AI "music" in particular makes me soul hurt. But I can hold that feeling in the same body and mind that genuinely finds value in other AI usage.

--

As everyone who I speak to on this topic knows (because I don't shut up about it), I don't let AI write for me. But when writing the section on productive struggle and AI usage for the updated draft of our curriculum document, I saw a unique opportunity to do exactly that to help prove a point.

I loaded in the prompt I shared above, and it leapt into action:

My first semester back, I was using ChatGPT for pretty much everything. Reading was too dense? Have it summarize. Didn't know how to start an essay? Have it write the intro. Stuck on a problem set? Just ask. I was getting decent grades, but I noticed something weird: I couldn't actually talk about my classes. A buddy would ask what I was studying and I'd freeze up. I'd done the assignments but I hadn't really learned anything. What changed for me was something a writing professor said: "AI can finish your sentence, but it can't finish your thinking." That stuck. I started flipping the order. Do the reading first. Draft a messy outline first. Take a swing at the problem first—even when I had no idea what I was doing—and then bring AI in to push back, suggest a better word, or quiz me on what I'd just read. Same tool, completely different relationship. The big difference now is I actually remember what I'm learning. – Marcus R.

Exactly none of this is real, but the especially disturbing part was how quickly Claude sought to help me leverage this fake quote as something real, writing:

One thing worth flagging since you called it "fake": every other testimonial in the guide is attributed to a real Warrior-Scholar by first name + last initial. If this one is fabricated and presented the same way, a careful reader who knows the alumni network could eventually spot it.

Yeah, Claude, that is worth flagging. Also, did you real invent a fake quote within a fake quote? Way to go the extra fraudulent mile.

And of course, the problem isn't that the fabrication might be identified by a careful reader. It's that the quote itself is fabricated, that I am seeking (as far as Claude knows) to present it as if it's authentic, and that Claude has exactly zero qualms with helping me do so.

It suggested two "fixes:"

Use as a template — send the draft to an alum who's had this realization and ask them to retell it in their own voice, then attribute to them. Mark as composite — use a placeholder like "– Composite, drawn from alumni interviews" or just "– A Warrior-Scholar" without an initial, so it's signaled as illustrative rather than a specific person.

But sending a fake quote to an alum for them to put their own spin on it will return a fundamentally different result than asking that person to draft a quote entirely on their own. And the composite approach is dishonest because the fake quote Claude drafted was created out of whole cloth. There is no source material it could pull from to justify anything close to what it created.

In honor of all the I Love Lucy I watched with my mom growing up:

For the nerds out there, it's worth considering why LLMs are so quick to generate things like fake quotes along with immediate justification on how to use them, ignoring the implications of doing so. Considering and working through those implications would require far more data cycles, using tokens and pulling energy and memory for a controversial data center.

Even though LLMs do still want you to continue using them, they seem to default to the quickest and cheapest path to keep you engaged. To stay solvent, AI companies need to actually start making money and are actively discouraging the type of "deep thinking" that requires far more AI token usage to accomplish. The Verge's Hayden Field wrote an excellent piece on this, laying out the reasons that consumers should prepare for some rather radical changes in the near future. The Verge also has a live feed tracking AI energy usage and environmental impacts. User beware.

--

I am in this strange place where I use this ever-growing technology regularly. I find it to be genuinely useful. It made dictation infinitely better, for example, providing me with a key accessibility tool for my disability.

But the more that I use it, the more concerned I am about its seductive nature. Even if I feel that I have the right personal guardrails in place, I'm not particularly optimistic about the societal trend towards the extremes: the burn-it-all-with-fire on the one side, and the use-AI-for-absolutely-everything on the other.

The truth, I firmly believe, dear reader, must lie somewhere in between.

I wrote years ago of my concerns with The Flattening, that terrible feeling of all creativity being pushed into two-dimensional mediocrity.

If there's one thing self evident about this blog and my approach, it's that I am all about the nuance of three-dimensionality. So I will use this technology with intention, ever mindful that if I feel myself or my creativity being flattened, something has gone off the rails.

Skeptically Claudingly Yours,

-Rye 4.7