This workflow comes from Pieter Levels (@levelsio), who shared the experiment on X in November 2026. Original post: https://x.com/levelsio/status/2116959423117857026 . His reply detailing the toolchain: https://x.com/levelsio/status/2116961807315169718 . What follows is our breakdown and rebuild guide, not his writeup.
Pieter Levels handed raw DJI footage to Claude Code and asked for a roughly 10-minute vlog. He made only minor adjustments afterward, and learned what tools it had used only after the fact.
That detail is what makes this worth unpacking. It turns the vague idea of "AI edits video" into an engineering pipeline you can copy.
What actually ran
The chain runs in this order:
- Transcription: mlx-whisper converts audio into timestamped text
- Frame sampling: frames are grabbed at fixed intervals and composited into a single grid image
- Speaker separation: SpeechBrain's ECAPA model identifies who is talking
- Decision: Claude reads the transcript and the frame grid together and picks which segments survive
- Output: a JSON edit list, not an edited video
- Render: ffmpeg cuts and joins according to that list
- Extras: OpenCV with YuNet for face thumbnails, Pillow for the opening title card
Steps 4 and 5 are the pivot. The model is not editing video. It is reading two structured inputs and emitting an editing decision.
Why this design works
Could you just feed the video to a model? Technically yes, but cost explodes. An hour of footage at frame granularity is an astronomical token count.
This pipeline compresses twice.
Compression one, on audio. An hour of talking becomes roughly ten thousand words of text. Cheap to read, and timestamps are preserved so the model knows where each sentence sits in the video.
Compression two, on visuals. Frames sampled at intervals and tiled into a grid let the model see multiple moments in one image. It can judge "this part is walking" or "this part has scenery" without watching continuous frames.
After both compressions, hours of footage become input a model can read in one pass.
Rebuilding it yourself
Step 1: Get transcription right first
Separate the audio track and produce timestamped text. Everything downstream depends on this.
For talking-head vlogs, transcription accuracy decides whether the model can find the boundary where a thought ends. Noisy footage loses words, and segments get cut mid-sentence.
Step 2: Design the edit list schema
Have the model emit a fixed-schema JSON object, not a natural-language instruction like "keep 00:12 to 01:47":
{
"segments": [
{"start": 12.4, "end": 107.2, "reason": "arriving at the rental, room tour"},
{"start": 340.1, "end": 402.8, "reason": "seaside sunset timelapse"}
],
"total_duration": 596.1
}
With a fixed schema, a script consumes it directly. The reason field matters: it tells a human reviewer why each segment survived.
Step 3: Let ffmpeg render
The model only decides. Cutting and joining go to ffmpeg. Every stage then debugs independently: bad selections mean prompt or model changes, bad frames mean ffmpeg flags.
Minimal cut-and-concat:
# cut each segment per the list
ffmpeg -ss 12.4 -to 107.2 -i input.mp4 -c copy seg_01.mp4
# concatenate
printf "file 'seg_01.mp4'\nfile 'seg_02.mp4'\n" > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy out.mp4
-c copy skips re-encoding and runs orders of magnitude faster. It requires matching codec parameters across segments, so normalize once before cutting.
Step 4: Route the decision through an API
This is where a gateway key helps. Configure Claude Code against HiAPI (see Claude Code and Codex on HiAPI), and the reasoning runs on Opus 5.5 at $2.02 per million input tokens and $10.08 per million output tokens.
For one hour of footage, the transcript plus a few dozen frame grids lands around 20,000 to 30,000 input tokens. At Opus 5.5 input pricing, a full decision costs cents. Against an hour of human editing time, that is not a close call.
The same skeleton, different inputs
Swap the input source and you have a different product:
| Scenario | Input | Editing goal |
|---|---|---|
| Podcast clips | Long audio plus transcript | Find self-contained arguments |
| Meeting recaps | Screen recording plus transcript | Keep decisions and action items |
| Course edits | Lecture video | Remove pauses and repetition |
| Interview reels | Multi-camera | Reorganize by theme |
Prompts differ per category. The skeleton, transcribe, compress, decide, render, stays the same.
Where it breaks
Timestamp drift. Different tools anchor timestamps differently, sometimes by tens of milliseconds, enough to clip the first syllable. Calibrate on a short clip first.
Models keep too much. First runs typically undershoot the trim, returning more than the target length. State the target duration explicitly and instruct it to re-select if it overshoots.
Cuts land mid-sentence. Give the model sentence boundaries from the transcript, not just raw timestamps.
Re-encoding drags. Mismatched codec parameters break -c copy and force a full re-encode, which takes a long time on an hour of footage. Normalizing parameters first is not optional.
FAQ
Do I have to use Claude Code?
No. Any model that reads long text and emits schema-conforming JSON can handle the decision step. Claude Code's edge is that it writes scripts, runs ffmpeg, and inspects results itself, saving you the manual glue work.
Does mlx-whisper work without Apple silicon?
No, mlx is Apple's framework. On other machines use faster-whisper or whisper.cpp. The logic is identical.
How long does the whole run take?
Transcription dominates. An hour of footage takes anywhere from tens of minutes depending on model size on Apple silicon. The decision step is minutes. Rendering with -c copy is fast.
Is the output ready to publish?
Levels described minor manual adjustments. The pipeline solves the expensive part, choosing ten minutes out of hours of footage. Pacing and transitions are still faster by hand.
Bottom line
What makes this work is not model intelligence. It is turning video into data a model can afford to read. Transcription compresses audio, frame grids compress visuals, the model only picks segments, and ffmpeg does the rendering. To rebuild it, start with timestamped transcription. The rest is engineering.
Sources
- Original share: Pieter Levels (@levelsio), https://x.com/levelsio/status/2116959423117857026
- Toolchain details (author's reply): https://x.com/levelsio/status/2116961807315169718
- Pipeline breakdown, code samples, and caveats here are written by HiAPI, not the original author
