Watching a playtest takes as long as the playtest. A 20-minute session is 20 minutes of video, so small teams skip it or skim it, and the moment a player got lost is never seen by the person who built the room.
Playtest Lens watches it for me. It's my solo entry to the Meta Global AI Developer Hackathon, built in nine days and submitted on 4 November 2026. Meta's Muse Spark watches the recording and listens to the player thinking out loud. A second pass has to check every moment it found against the clip and the game's own event log. What comes out is one page: where the player was confused, stuck or surprised, on the level's map, with the clip and the player's own words.
You can open the live demo (two real playtests, no install), read the code, or watch the 3-minute video:
I like building systems that could help me do better work, and I enjoy trying new tools. I hadn't worked with video and audio analysis before, so this was a fun challenge that produced something potentially useful, either to use as it is or to keep building on.
What you get
One page per playtest, in two modes. Read is a page to scroll: a one-line verdict, the top problems beside the level's map, then every moment in order under the room or puzzle it happened in, and last the moments the check threw away, each with its reason. Watch opens from any moment. The clip fills most of the screen with the player's words as its headline. Beside it, the map shows where the player is, where they walked in the last 20 seconds and the last thing the game logged.
Each moment also says whether the friction was by design (learning the controls, exploring, a planned surprise) or unplanned (stuck, or blocked with no way forward). On Player 1's session that split matters: of 24 moments kept, 16 are by design and 8 are real friction. A switch hides the first kind, so the page starts with what to fix.
One moment, in full
Player 1 played the Lobby level of the COGITO demo for 21 minutes and 20 seconds, thinking out loud. She kept coming back to the Archive's code lock, three attempts in all. Her page puts it first: Archive, Code Lock, 4 moments, 1:22 stuck, abandoned.
Moment 21 is at 19:17. Its clip runs from 19:07 to 19:27, and its headline is what she said at the keypad: "So I still don't know what is the code". The check's note says she keeps pressing keypad inputs and the door doesn't open. The likely cause it gives is plain: the Archive door keypad gives no hint to the code.
Then the event log takes over. At 20:27 the duct door in the toilet stall opened. At 20:33 the quest step for getting into the Archive was done. She was stuck on the puzzle, not on progression, because the toilet route let her into the Archive.
That difference is what I'd want to know as the designer of that room, and the video alone doesn't show it. Open moment 21 in her report to see the clip, the map and the log together.
How it works
- Record. The tester plays the Godot build and records the screen with the microphone on, with OBS or any screen recorder. A small Godot add-on writes an event log as they play: inputs, position and view, rooms and puzzles entered, doors, pickups, quest steps. It only listens. It never changes gameplay, timing or input, and it logs nothing that names a person or a machine.
- Find. Muse Spark 1.2 takes each 2-minute window of the recording as video with its sound in one request. So there's no frame sampling and no separate speech model at this step. It lists moments worth a look, each with a time and a reason.
- Check. For each moment, a second Spark call gets a 20-second clip, the player's words and that slice of the event log. It has to keep or drop the moment, with evidence.
- Report. One findings file holds every moment, the route, the rooms and the summary. The page only draws that file and computes nothing. Every model reply is cached by a hash of its request, so a rerun on the same input costs nothing.
The fiddly part was lining up the video and the log. The add-on flashes the screen white and beeps for a quarter of a second, and writes the time of that frame to the log. The first brief for finding the flash was wrong: the recorder captured the mouse pointer over the white, and the game ran in a window, so the flash frame was never fully bright. The recorder also starts only once the game window is open, so a flash right at launch would be missed. Now it flashes at New Game and at each level load, and the one command checks the log for a flash before it makes any model call.
The level side is plain level design work. Telemetry volumes in the editor name the rooms and puzzles, and the page groups moments by them. Another small tool draws a top-down map of each floor.
The check has to prove it
The first pass is allowed to guess. The second one isn't. It's the same model with a different job: back each claim with the clip, the player's words and the log, or drop it.
The player's words come from Muse Voice Transcribe, a separate speech-to-text model, and a quote reaches the page only if that transcript holds it. This rule came from the first review of the check. Back then, the quote on a card was the checking model's own string, not what the player said. A model can paraphrase a player and present it as a quote, and on a page that's meant to be evidence, that's the worst kind of mistake.
Dropped moments aren't hidden. They're listed at the end of the page, each with its reason. On Player 1's page one candidate went partly because its quote added words the transcript didn't have.
The check isn't perfectly stable, and I'd rather say so. Rerunning the unchanged check on Player 1's candidates flipped 3 of 24 cards by chance. One prompt change pushed too many moments to "stuck" (from 4 to 9 of 24 on her session), so it got a guard. I judged every card that changed between the last two versions of the prompt, on her session and my own, in both modes, and agreed with the final one on 17 of 21.
A session nothing was tuned on
Before any tuning, the sessions were split. My own runs and Player 1's were for tuning. Player 2's was held out: no prompt or threshold was tuned on it. I watched him play, wrote my notes blind, and compared them with the page only at the end.
- Found: the places where he was slowed. The Archive code door (he never found the code), the CEO Office door (he got in with the key instead), and the toilet duct he used to reach the Archive.
- A bonus: a freeze when he shot the Archive door, logged as a suspected bug. The log shows a frame taking 141.92 ms right after the shot. I think it can be a real one.
- Overrated: the friction. I saw no one stuck. The page calls three moments blocked, the code door among them, which I'd count as a problem but not a blocker.
- Missed, and can't see: he never opened a drawer, so he never found the lockpicks or the code note. Nothing in the video or the log shows a drawer that was never tried.
I left the overrating in. The session is held out, so tuning on it would have made the score mean less. Player 2's report is on the demo site as it came out.
Spark and open weights
The main mode runs on Muse Spark 1.2 in contributor mode, through Meta's API. Contributor mode is cheaper, and Meta may train on what you send, so the testers' consent text said so. The same pipeline also runs on Muse Spark Open: the open-weights version, whose weights are public, so other providers can host it. I ran it on Baseten.
Open finds more, keeps more and costs more. Of the moments Spark kept, Open also kept 20 of 24, 9 of 10 and 7 of 8, within 15 seconds. Spark's pages were easier to read. Open's catch more small confusions.
Getting the open model to hear the player took trial and error. The first version sent eight still frames per window with no sound, and the answers came back empty. A run with the real video but no voice quoted nothing: 0 of 22 moments had a quote, and it did worst against my own notes. On Baseten, the audio inside the video file was simply ignored. A 20-second clip on its own came back saying it heard nothing. The same clip with its audio sent as a separate track quoted the player. That's how the open path works now, and its quotes are still checked against Voice Transcribe.
For a playtest, the voice carried more than the picture. The video helps with motion and timing, but the player saying they don't get it is the signal.
What it doesn't do yet
- It can't see what a player never tried. Player 2 never opened a drawer, and nothing records a drawer nobody touched. A log line for objects looked at but never used could feed it. That's only an idea for now.
- It rates friction higher than I do, at least on the held-out session. I didn't tune that away.
- It has seen one level of one game, three times: two testers and me. That's enough to show the idea, not to trust it on your game.
- The map draws one scene per report. A session that visits two mapped levels loses one of them.
- The add-on logs a pickup only when it lands in a new inventory slot. Items that stack aren't logged.
- On a phone, the timeline pins pile up.
One person, several models
I built it alone, with coding agents in T3 Code, under the same rule as in my post on agentic game development. I decide what gets built, every word a user reads, the look, the prompts' goals and the scope. Agents propose, build what I approved, and prove it works. Each of my decisions is recorded with my words quoted; without a quote, it's a proposal.
Every task has a brief in the repo: who did it, the goal, what came back and what was changed before commit. Every review has its own report, and a reviewer is always a different model from the author. Each commit names its author model. There are 45 task briefs and 41 review reports, and they're all public.
The split changed twice. Muse Code on Muse Wonder wrote some of the first days' code. From 28 to 31 October, DeepSeek on Modal and Grok through Cursor wrote parts of the analyzer, the report and the Godot side. From 31 October, Claude Opus 5.5 wrote most of the code, Claude Fable 5.1 designed the report page, Grok reviewed the code and Claude Sonnet 5.5 reviewed the page.
What went wrong, from the records:
- One agent spent all 30 of its turns exploring and changed nothing.
- Usage limits cut agents off mid-task more than once. One day a review couldn't start because of Cursor's limits, so a different model reviewed it.
- My review rule stops after round 2 and brings me the approach question. The review of how moments get grouped into rooms still ran six rounds before I stopped it. A seventh, after another model's fix, passed.
- A half-screen layout of the report was built, failed review because it broke on short windows, and was reverted.
- A review caught bug cards that named the wrong engine. They said Unity; the game is Godot.
- The brief for the review before going public leaked the very details it was meant to keep out. The commit was amended and the history rewritten before anything went public.
The report page went through three rounds of mockups before the layout settled: one current moment, the map and the clip side by side, and a timeline across the foot of the page.
What it cost
A 21-minute playtest costs about $0.18 on Muse Spark 1.2 Contributor, including Voice Transcribe. The whole project came to about $30.40 in model calls, from my own cost log. Every call goes through one client that caches the reply and logs its tokens and cost, which is how I know.
What I learned
I was surprised by the difference between working with the open model and the closed-source Spark, and by how quickly today’s AI tools let me build a prototype like this. It feels increasingly possible to build custom software around your own needs. That doesn't automatically make it useful: sometimes the result isn't useful, sometimes it's sloppy. Learning to use the tools and working through those attempts is part of the process.
The human still has a job to do. You're the one who identifies the problem, iterates on the solution and decides what the product should be. The tools can do a lot of the implementation, but that doesn't replace your judgement.
If I did this again, I'd find more people to playtest and build my own test level in advance rather than use the COGITO demo. I didn't have time for that during this hackathon.
What's next
This stays a hackathon experiment for now. I want to see whether other people are interested in picking it up and improving it.
Tools and thanks
- Player 1 and Player 2, for playing, thinking out loud and letting me share their sessions for the submission and the demo.
- COGITO by Phazorknight (MIT), the immersive-sim template whose demo is the test level. My changes live in a fork. The demo's own asset credits are in THIRD_PARTY.md.
- Godot 4.7.2, Python 3.13 and ffmpeg.
- Meta's models: Muse Spark 1.2 (contributor), Muse Voice Transcribe, and Muse Spark Open, hosted on Baseten.
- To build it: T3 Code, Claude Code, Cursor and Muse Code, with DeepSeek and GLM on Modal.
Aleksandr Nariadchikov · @sashanaria
