From the paper Training Vision Models for LLM-Driven Weight-Space Programming · Ropinski & Kranz · Ulm University
1,986 recordings of LLM agents editing vision models directly in their weights: no gradients, no fine-tuning, only probes, weight reads and weight writes through our WeightLab tools.
Narrated overview of the paper (2:43): Edit-Ready Training, WeightLab, results, and the trajectory wall.
@misc{ropinski2026training,
title = {Training Vision Models for LLM-Driven Weight-Space Programming},
author = {Ropinski, Timo and Kranz, Sebastian},
year = {2026},
eprint = {2611.26408},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
Each agent receives an edit request (relabel a concept, fix a confusion between two classes, or fix a misclassified subgroup) with a few support images. It may change any weight, under a budget of 20 probes, 30 reads and 8 edits, and never learns how the network was trained. Every edit is scored on held-out images as success minus side effect.
Agents mostly write sparse, rank-one edits to the classifier head. Edit-Ready Training (ERT) rehearses exactly such edits during training and backpropagates through them, so the features become easy to edit. ERT gives the best mean net score for both frontier editors: GPT-6 Astra 0.74 (standard training 0.63) and Claude Opus 5.5 0.72 (0.60), with the largest gains where standard networks are hard to edit.
(a) ERT simulates edits during training. (b) An LLM edits the network through WeightLab. (c) Mean net edit score over all nine settings.