Giving coding agents their own screen

IThe interruption

When a coding agent opens a browser on my desktop, it can cover my work, move the cursor, or take focus. I wanted it to test interfaces while I continued using the machine.

I use hyprcage for this. The original project gives an agent a separate, in-memory screen through the Cage compositor. I maintain a fork with additions for interaction, emulation, and recording.

IIRead before taking a screenshot

A screenshot is useful for checking appearance, but many actions only need a control’s name, role, and location. I added a text snapshot that lists those details with references an agent can act on.

The tool tries the browser’s DevTools protocol, then the desktop accessibility interface, AT-SPI, then OCR. The available source depends on the application and its setup. An action tool clicks, types, or scrolls against a reference; a find tool waits for an element.

Text does not replace visual review. When it cannot explain the layout, the agent still needs a screenshot.

IIITest and show the result

I added device emulation for viewport size, pixel ratio, touch, user agent, network conditions, CPU slowdown, and color scheme. These controls help exercise responsive behavior. They do not replace testing on a physical phone.

I also extended recording with a browser-viewport crop, an option to cut idle time, and pointer movement with click rings. The aim is a short demonstration that makes the interaction easy to follow.

A mirror window lets me watch the agent’s screen. It does not accept input, so watching does not interfere with the agent’s work.

IVBoundaries and limits

The fork also provides tools for the real desktop, including accessibility-based actions and window control. Those tools refuse actions while the session is locked.

A separate screen prevents desktop interruptions; it is not a claim that launched applications are isolated from the host filesystem or network.

Each screen still consumes memory, so I cap concurrent screens. Accessibility coverage also varies: I recorded a case where Thunar stopped appearing on the accessibility bus after its Create Folder dialog opened.

The useful change is practical: interface testing can happen beside my work, and I can inspect the result when I need to.