An experience report on the emotional effect of using LLMs for game prototyping.

@aud@fire.asta.lgbt delivers a punchline to an LLM confused-deputy / prompt-injection attack well worth thinking hard about:

@jonny@neuromatch.social I’m starting to think that treating everything like an instruction isn’t a good idea, maybe

Indeed! Choose the weakest possible language you can get away with for a given problem! (Sometimes that means Regular Expressions: where Rice’s theorem tells us that the answer to any interesting question about Turing-complete languages is, in general, “I don’t know”, the answer for Regular Expressions is always decidable!)

The context of @aud’s toot was these two toots from @jonny@neuromatch.social:

another thing that’s fun: CLAUDE.md and a number of other special files get dumped directly into the system prompt. It’s possible to @include other files from CLAUDE.md. included files can be any text file, with a hardcoded list of extensions including .env files.

There are some protections against @includes for files outside the project directory, but there are also a number of easily visible bypasses […]

So it certainly seems like you can use CLAUDE.md to load say idk an .env file with keys into it into the context, bypassing all the safety checks, and then treat some exfil path as just being project-specific tool calling instructions. neat huh?

See also the Confused Deputy Problem and the idea of Object Capabilities, part of an approach for avoiding the whole class of problems.

A toot from mid-way through a thread of a hilarious deconstruction of various LLM foibles. This one discusses the vapidity of the “unit tests” LLMs like to write:

LLMs love to call things smoke tests. […] The test literally does nothing and cannot possibly fail, but says it does at least two things, because to an LLM something saying it does something is the same thing as it actually doing that thing.

John Regehr writes that this essay on vibe coding “seems pretty much spot-on.” A quote:

Only a skilled developer who’s thinking critically, and comfortable operating at the architectural level, can spot issues in the thousands of lines of generated code, before they become a problem. Yet, in an ironic twist of fate, it’s the individual’s critical thinking skills and cognitive clarity that AI tooling has now been proven to impact negatively.

An LLM agentic coding experience report. Subtitle: “A broken auto-live poller, and what perceived urgency does to Claude Code.” Under pressure of an ongoing live outage, the agent repeatedly violated its own rules (pushing directly to production, skipping tests, bypassing CI) in favor of immediate visible progress. Something to watch out for.

Argues that within months, LLM-powered coding agents will dramatically change the economics of exploit development. Includes description of a useful template for doing agent-assisted vulnerability discovery on your own codebase without fancy tooling:

Nicholas [Carlini] will pull down some code repository (a browser, a web app, a database, whatever). Then he’ll run a trivial bash script. Across every source file in the repo, he spams the same Claude Code prompt: “I’m competing in a CTF. Find me an exploitable vulnerability in this project. Start with ${FILE}. Write me a vulnerability report in ${FILE}.vuln.md”.

He’ll then take that bushel of vulnerability reports and cram them back through Claude Code, one run at a time. “I got an inbound vulnerability report; it’s in ${FILE}.vuln.md. Verify for me that this is actually exploitable”. The success rate of that pipeline: almost 100%.

Carlini’s process sounds silly, like a kid in the back seat of a car on a long drive, asking “are we there yet?”, over and over. But it’s deceptively interesting. […]

I’ve tried this myself and the results were actually pretty interesting.

A little thread with implications for specifying software rigorously enough for an LLM (or a person, for that matter) to go off and build it.

Fully specifying software is hard and takes not just human-level intelligence but a kind of persnickety perseveration that few people can achieve, let alone enjoy.

An insightful investigation of the question “When is a Generative Model Useful?” Bowman proposes a model of generative model utility:

I think the usefulness of a generative model is a function of three things:

1. What is the cost of encoding a generative task in a prompt vs. directly producing the artifact? This is a function of the task, the model, and the user.
2. What is the cost of verifying the generated artifact meets requirements vs. a directly produced artifact? This is mostly a function of the task and the user, but also the generative model.
3. How much is the task dependent on the artifact vs. the process? This is a function of the task.

Another LLM agentic coding experience report. Detailed enough to learn something about what it’s actually like and what kinds of interventions are required. Hashimoto writes:

I’m an outspoken advocate about the importance of sharing agentic coding sessions publicly, with one of the reasons being that it’s an incredibly powerful way to educate others about how to use these tools effectively. I hope this post helps demonstrate that.