DEV Community

Cover image for Agentic Desktop Testing with tamesu: Test Native Apps the Way a Person Uses Them
Xavier Portilla Edo
Xavier Portilla Edo

Posted on Originally published at xavidop.me

Agentic Desktop Testing with tamesu: Test Native Apps the Way a Person Uses Them

Introduction

Desktop apps are the hardest apps to test from the outside. A web page has a DOM and a dozen good test tools. A desktop app has a window.

So desktop UI tests usually go one of two ways. A script clicks at screen coordinates and compares pictures. Or you learn a different test framework for every operating system.

A person does neither. A person reads the window, finds the button that says "Unlock", and clicks it.

tamesu (試す, Japanese for to try, to test) is an open-source CLI written in Go that tests software that way. Version 1.22 adds a desktop target for macOS, Windows and Linux. You write a step in plain language. An agent reads the app through the accessibility tree, the same tree a screen reader uses, and acts on the elements it finds there. There are no screenshots to interpret and no coordinates.

This is the fourth article about tamesu. The first three cover web apps, AI agents and A2A agents. This one covers desktop apps.

tamesu drives agent-desktop, which reads your app through the accessibility API of the operating system

Why Desktop UI Tests Break

Coordinate and picture scripts tamesu
A step A click at x, y, or a match on a picture A sentence that says what the user wants
What the test reads Pixels The accessibility tree, as text
A renamed or moved button The test fails The agent finds the element by its role and name
The mouse pointer Moves across the screen Stays where it is: a click is an accessibility press
Cost of AI steps Every run pays for every step Plans are cached, so a stable window replays with no model calls for its goals
Passwords In the script Typed at the last moment, never sent to the model
macOS, Windows, Linux One tool per system One YAML format for the three

A tamesu test mixes three kinds of steps:

  • goal: a sentence. The agent acts until it is met.
  • expect: a sentence. The model says if the window satisfies it.
  • action: an exact, deterministic step, such as a click on a named button or a check on a text. No model is involved.

Use a goal where a person would say what they want. Use an action where the test must be exact.

How tamesu Reaches a Desktop App

tamesu does not talk to the app itself. It runs agent-desktop, a CLI from Vercel Labs that speaks the accessibility API of each system:

  • the AX API on macOS,
  • UI Automation on Windows,
  • AT-SPI on Linux.

tamesu pins one version of that CLI (1.2.0 in tamesu 1.22.0). On first use it downloads the binary into ~/.cache/tamesu/agent-desktop. It checks the file against a SHA-256 that ships inside tamesu before it runs it. To use your own binary, set TAMESU_AGENT_DESKTOP.

A goal step runs in a loop of three stages:

  1. Observe. The window becomes one line per element. A line with a number is something the agent can act on. A line without a number is text it can read.
  2. Plan. The agent gets the goal and those lines. It calls one tool: click, type, select, press, wait or done.
  3. Act. The tool runs on the app. The agent gets the new window and plans again.

The loop ends when the agent calls done. It also ends after max_goal_steps actions (15 by default), and then the step fails.

This is what the agent reads when the demo app of this guide starts:

    window "Mochi Stock"
      text "Mochi Stock"
      text "Enter the staff PIN to open the stock room"
[1]   textfield "PIN" (focused, secure)
[2]   button "Unlock"
[3]  button "Close"
[4]  button "Full Screen"
[5]  button "Minimize"
     text "Mochi Stock"
Enter fullscreen mode Exit fullscreen mode

That is the whole window: nine short lines of text. A small, fast model can drive a desktop app from that.

What You Will Build

You will test Mochi Stock, a small native Mac app for the stock room of Mochi Shoes, the demo shop of the earlier articles. It has a PIN screen, a product list with a search box and a form to receive stock. A command in the menu bar exports a CSV file.

The suite has four tests:

  1. Unlock the app and receive stock, written as goals.
  2. A wrong PIN keeps the app locked.
  3. The search box narrows the list.
  4. The export command in the menu bar shows a confirmation.

Then you will change the app twice. First you rename a button, and the test still passes. Then you add a real bug, and the test fails with the reason.

What You Need

  • A Mac with macOS 13 or newer, for the demo app of this guide. The desktop target also runs on Windows 10 (1809) or newer and on Linux (Ubuntu 24.04 or newer).
  • tamesu 1.22.0 or newer.
  • The Xcode Command Line Tools (xcode-select --install), only to build the demo app.
  • A Gemini API key in GEMINI_API_KEY. tamesu calls models through Genkit, so OpenAI, Anthropic and local models work too.

Step 1: Install tamesu and Give It Access

Install the CLI

brew install xavidop/tap/tamesu      # macOS
go install github.com/xavidop/tamesu/cmd/tamesu@latest   # any platform
Enter fullscreen mode Exit fullscreen mode

Linux packages and a Scoop bucket for Windows are on the install page.

Give the terminal access

A desktop test needs a desktop: the session of a signed-in user, where the app can open a window. Each system asks for something more:

System What it takes
macOS The Accessibility permission for the app that runs tamesu (your terminal, your editor or your CI agent). It is in System Settings, Privacy & Security. Add Screen Recording to get screenshots.
Windows The interactive session of the signed-in user: not a service, not a locked screen.
Linux A graphical session with a window manager and the accessibility bus (at-spi2-core), on X11 or Wayland.

On macOS, grant the two permissions now. You will check them with tamesu doctor in Step 3.

Step 2: Build the App Under Test

Mochi Stock is one Swift file. Create a folder named mochi-stock and save this as MochiStock.swift:

import SwiftUI

struct Product: Identifiable {
    let id: Int
    let name: String
    var units: Int
}

final class Store: ObservableObject {
    @Published var unlocked = false
    @Published var wrongPIN = false
    @Published var exported = false
    @Published var pin = ""
    @Published var query = ""
    @Published var selected = 0
    @Published var units = ""
    @Published var products = [
        Product(id: 0, name: "Trail runner", units: 12),
        Product(id: 1, name: "City sneaker", units: 30),
        Product(id: 2, name: "Canvas slip-on", units: 3),
        Product(id: 3, name: "Leather boot", units: 8),
    ]

    var total: Int { products.reduce(0) { $0 + $1.units } }

    func unlock() {
        unlocked = pin == "2468"
        wrongPIN = !unlocked
    }

    func receive() {
        products[selected].units += Int(units) ?? 0
        units = ""
    }
}

@main
struct MochiStockApp: App {
    @StateObject private var store = Store()

    var body: some Scene {
        WindowGroup("Mochi Stock") {
            Group {
                if store.unlocked { StockView() } else { LockView() }
            }
            .environmentObject(store)
            .frame(width: 560, height: 440)
        }
        .commands {
            CommandMenu("Stock") {
                Button("Export CSV") { store.exported = true }
                    .disabled(!store.unlocked)
            }
        }
    }
}

struct LockView: View {
    @EnvironmentObject var store: Store

    var body: some View {
        VStack(spacing: 12) {
            Text("Mochi Stock").font(.largeTitle)
            Text("Enter the staff PIN to open the stock room")
            SecureField("PIN", text: $store.pin).frame(width: 160)
            Button("Unlock") { store.unlock() }
            if store.wrongPIN {
                Text("Wrong PIN. Try again.").foregroundColor(.red)
            }
        }
        .padding()
    }
}

struct StockView: View {
    @EnvironmentObject var store: Store

    var shown: [Product] {
        store.query.isEmpty ? store.products : store.products.filter {
            $0.name.localizedCaseInsensitiveContains(store.query)
        }
    }

    var body: some View {
        VStack(alignment: .leading, spacing: 12) {
            TextField("Search products", text: $store.query)
            List(shown) { product in
                HStack {
                    Text(product.name)
                    Spacer()
                    if product.units < 5 { Text("Low stock").foregroundColor(.orange) }
                    Text("\(product.units) in stock")
                }
            }
            HStack {
                Picker("Product", selection: $store.selected) {
                    ForEach(store.products) { Text($0.name).tag($0.id) }
                }
                TextField("Units", text: $store.units).frame(width: 80)
                Button("Receive") { store.receive() }
            }
            Text("Total: \(store.total) units")
        }
        .padding()
        .alert("Exported 4 products to stock.csv", isPresented: $store.exported) {
            Button("OK") {}
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Three details matter for the tests:

  • The PIN is 2468, and the PIN field is a secure field. It never shows its value.
  • "Canvas slip-on" starts with 3 units, so it shows a Low stock label. The label goes away at 5 units or more.
  • All the state lives in memory. Every launch starts locked, with 53 units in total.

Build and install it

Save this next to the Swift file as build.sh:

#!/bin/sh
set -eu
APP="Mochi Stock.app"

rm -rf "$APP"
mkdir -p "$APP/Contents/MacOS"
swiftc -O -parse-as-library MochiStock.swift -o "$APP/Contents/MacOS/MochiStock"

cat > "$APP/Contents/Info.plist" <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>CFBundleName</key><string>Mochi Stock</string>
  <key>CFBundleIdentifier</key><string>dev.mochi.stock</string>
  <key>CFBundleExecutable</key><string>MochiStock</string>
  <key>CFBundlePackageType</key><string>APPL</string>
  <key>LSMinimumSystemVersion</key><string>13.0</string>
</dict>
</plist>
EOF

mkdir -p "$HOME/Applications"
rm -rf "$HOME/Applications/$APP"
cp -R "$APP" "$HOME/Applications/"
echo "installed $HOME/Applications/$APP"
Enter fullscreen mode Exit fullscreen mode
chmod +x build.sh
./build.sh
Enter fullscreen mode Exit fullscreen mode
installed ~/Applications/Mochi Stock.app
Enter fullscreen mode Exit fullscreen mode

The script copies the app to ~/Applications for a reason. tamesu starts an app by its name or its bundle id, not by a path. So the app must live where macOS looks for apps.

Step 3: Configure the Project

Create a second folder for the tests. Add a file named tamesu.yaml:

version: 1
suites: ["suites/*.yaml"]

genkit:
  plugins: [googlegenai]
models:
  default: googleai/gemini-3.8-flash
  judge: googleai/gemini-3.8-flash
  browser_agent: googleai/gemini-3.8-flash

defaults: { timeout: 3m, max_goal_steps: 10 }
budgets: { max_cost_usd: 2 }
reporters: [console, junit, html]

secrets:
  GEMINI_API_KEY: env:GEMINI_API_KEY
Enter fullscreen mode Exit fullscreen mode
  • models.browser_agent is the model that drives the app in a goal step. The name comes from the browser target; the desktop target uses the same setting.
  • models.judge grades the whole session at the end.
  • budgets stops the run when the model spend passes the limit.
  • secrets are references, never values. env:GEMINI_API_KEY tells tamesu where the key lives.

Step 4: Write the First Test with Goals

Create suites/stock.yaml:

suite: stock
tags: [desktop]

secrets:
  STOCK_PIN: env:STOCK_PIN

targets:
  stock:
    type: desktop
    app: Mochi Stock

tests:
  - name: unlock and receive stock
    target: stock
    steps:
      - goal: "Unlock the app with the PIN {{secrets.STOCK_PIN}}"
      - expect: "The stock list is visible and the Canvas slip-on is marked as low stock"
      - goal: "Receive 5 units of the Canvas slip-on"
      - expect: "The Canvas slip-on has 8 units in stock and no low stock warning"
      - action: { assert_text: "Total: 58 units" }
    assert:
      - judge: "The steps show the app being unlocked and 5 units of the Canvas slip-on being received"
      - metric: { name: goal_steps, lte: 8 }
Enter fullscreen mode Exit fullscreen mode

The target is two lines: the type and the name of the app. The test reads like a note to a colleague. Two goals do the work, two expect steps check the meaning, and one action pins the exact total.

The assert block runs after the steps:

  • judge asks a model to grade the whole session against one sentence.
  • metric checks numbers that tamesu measured. Here the agent may use 8 actions at most.

Check the setup

export STOCK_PIN=2468
tamesu validate
tamesu doctor
Enter fullscreen mode Exit fullscreen mode
1 suite, 1 test, 0 errors, 0 warnings
ok   config: tamesu.yaml (version 1)
ok   suites: 1 file, 1 test
ok   secret GEMINI_API_KEY: env (link 1 of 1)
ok   secret STOCK_PIN: env (link 1 of 1)
ok   models: default=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), judge=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), browser_agent=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved)
skip speech: no speech models configured (speech.tts / speech.stt); audio mode and sip need them
ok   chromium: Google Chrome 157.0.8103.79 (/Applications/Google Chrome.app/Contents/MacOS/Google Chrome); no browser target configured
ok   agent-desktop: 1.2.0 (~/.cache/tamesu/agent-desktop/1.2.0/agent-desktop)
ok   desktop permissions: the desktop can be read and captured
skip livekit: no livekit target configured
skip sip: no sip target configured
Enter fullscreen mode Exit fullscreen mode

The last two ok lines are the desktop checks. agent-desktop shows the binary a run will use. desktop permissions confirms that macOS lets tamesu read and capture the desktop.

Before the first run, the agent-desktop line says that version 1.2.0 "is not in the cache yet". The first tamesu run downloads it.

Run it

tamesu run
Enter fullscreen mode Exit fullscreen mode
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-43-00
PASS   stock / unlock and receive stock  1m13s

1 passed, 0 failed, 0 errors in 1m13s · 5 model calls, 6801 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Enter fullscreen mode Exit fullscreen mode

Mochi Stock opens, unlocks, receives the stock and closes. Nobody touched the mouse.

Mochi Stock after the test received 5 units: the Canvas slip-on has 8 in stock and the total is 58

What the agent did

Every step is in steps.json, in the run directory. This command lists the actions of the two goals:

jq -c '.[] | select(.kind == "goal") | .actions[] | [.name, .args.element, .args.text]' \
  .tamesu/runs/*/tests/unlock-and-receive-stock/run-1/steps.json
Enter fullscreen mode Exit fullscreen mode
["type","textfield \"PIN\"","{{secrets.STOCK_PIN}}"]
["click","button \"Unlock\"",null]
["click","combobox \"Product\"",null]
["click","menuitem \"Canvas slip-on\"",null]
["type","textfield \"Units\"","5"]
["click","button \"Receive\"",null]
Enter fullscreen mode Exit fullscreen mode

Two things to notice:

  • The agent found everything by role and name. Nobody told it that the product is picked in a dropdown named "Product".
  • The PIN is still a placeholder. The model planned with {{secrets.STOCK_PIN}} and never saw 2468. tamesu puts the value in at the moment of typing, and passes it to agent-desktop on standard input, never as a command-line argument.

After the unlock, the window is richer. This is what the agent reads before the second goal:

    window "Mochi Stock" (focused)
[6]   textfield "Search products"
[7]   scrollarea
[8]    outline
[9]     treeitem "Trail runner 12 in stock"
[10]      cell "Trail runner 12 in stock" (focused)
          text "Trail runner"
          text "12 in stock"
[11]     treeitem "City sneaker 30 in stock"
[12]      cell "City sneaker 30 in stock" (focused)
          text "City sneaker"
          text "30 in stock"
[13]     treeitem "Canvas slip-on Low stock 3 in stock"
[14]      cell "Canvas slip-on Low stock 3 in stock" (focused)
          text "Canvas slip-on"
          text "Low stock"
          text "3 in stock"
[15]     treeitem "Leather boot 8 in stock"
[16]      cell "Leather boot 8 in stock" (focused)
          text "Leather boot"
          text "8 in stock"
      text "Product"
[17]   combobox "Product" value="Trail runner" (readonly)
[18]   textfield "Units"
[19]   button "Receive"
      text "Total: 53 units"
[3]  button "Close"
[4]  button "Full Screen"
[5]  button "Minimize"
     text "Mochi Stock"
Enter fullscreen mode Exit fullscreen mode

A row of the list is a treeitem. The dropdown is a combobox with its current value. The total is plain text.

Good to know:

  • A click presses the element through the accessibility API. The pointer does not move.
  • Text is normally set the same way, with no key pressed. Some fields take text from the keyboard only. tamesu then focuses the field and types with the real keyboard, and the run records a keyboard event in result.json. The three text fields of Mochi Stock are of that kind.
  • The real keyboard types into whatever has the focus. Leave the keyboard and the mouse alone while a suite runs.
  • An expect step sees what the window shows. The secure field shows no value: textfield "PIN" (focused, secure).

Step 5: Run It Again: the Plan Cache

Run the same command a second time:

tamesu run
Enter fullscreen mode Exit fullscreen mode
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-44-13
PASS   stock / unlock and receive stock  31.3s

1 passed, 0 failed, 0 errors in 31.3s · 3 model calls, 1660 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Enter fullscreen mode Exit fullscreen mode

The first run took 1m13s, 5 model calls and 6,801 tokens. This one took 31 seconds, 3 model calls and 1,660 tokens.

When a goal succeeds, tamesu saves its actions as a plan in .tamesu/cache/actions. This is the heart of the plan of the first goal:

{
  "goal": "Unlock the app with the PIN {{secrets.STOCK_PIN}}",
  "url_path": "desktop://Mochi Stock",
  "key": "1487a403320f0602b9600cc0",
  "actions": [
    {
      "name": "type",
      "args": {
        "element": "textfield \"PIN\"",
        "ref": 1,
        "text": "{{secrets.STOCK_PIN}}"
      }
    },
    {
      "name": "click",
      "args": {
        "element": "button \"Unlock\"",
        "ref": 2
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The next run replays the plan and does not call the agent. The three calls that remain are the two expect steps and the judge, the steps that need judgment. The metrics of the second run say the same:

{
  "cache_hits": 2,
  "cache_misses": 0,
  "desktop_actions": 6,
  "goal_steps": 6,
  "llm_cost_usd": 0,
  "llm_tokens": 1660,
  "observations": 10,
  "turns": 0
}
Enter fullscreen mode Exit fullscreen mode

A goal with the same window replays its plan with no model call; a changed window makes the agent plan again and save the new plan

The key of a plan is the goal, the app and the set of elements in the window (their roles and names). When the window changes, the key changes, and the agent plans again. You will see that in Step 9.

Two flags control the cache:

  • tamesu run --update-cache ignores the saved plans and plans every goal again.
  • tamesu run --replay uses only saved plans. A goal with no plan fails. This is the mode for CI.

Commit the .tamesu/cache folder with the suite.

Step 6: Pin Exact Behavior with Actions

A goal is flexible. Some checks must not be. Add two tests to suites/stock.yaml:

  - name: wrong PIN keeps the stock room closed
    target: stock
    steps:
      - action: { fill: { name: PIN, text: "0000" } }
      - action: { click: { role: button, name: Unlock } }
      - action: { assert_text: "Wrong PIN. Try again." }
      - expect: "The app is still locked and shows no products"

  - name: search narrows the list
    target: stock
    steps:
      - action: { fill: { name: PIN, text: "{{secrets.STOCK_PIN}}" } }
      - action: { click: Unlock }
      - action: { wait: { text: "Total: 53 units" } }
      - action: { fill: { name: "Search products", text: "boot" } }
      - action: { assert_text: "Leather boot" }
      - expect: "The list shows the Leather boot and no other product"
Enter fullscreen mode Exit fullscreen mode

No goal here, so no agent drives the app. Each action names its element the way a person would: by the text on it. The expect at the end of each test still asks the model one question.

The lock screen after a wrong PIN: the app shows "Wrong PIN. Try again."

The desktop target has six actions:

Action Argument
click An accessible name (click: Unlock), or { role, name }
fill { name, text }: empties the field, then types
press A key or a combination: Enter, Escape, Tab, cmd+s
wait Milliseconds, or { text } to wait until the window holds a text
screenshot A name: the window, saved under screenshots/
assert_text A text the window must hold, or { contains }, { equals }, { matches }

Good to know:

  • Names are matched exactly, but upper and lower case do not matter.
  • Add the role when two elements share a name.
  • A fill takes secrets too. {{secrets.STOCK_PIN}} is replaced at the moment of typing.
  • wait: { text: ... } is the safe way to wait for a screen change. The search test uses it before it types.
  • A desktop app has no URL, no page script and no network of its own to control. navigate, assert_url, eval_js, mock, a11y and visual belong to the browser target only.

Step 7: Drive the Menu Bar and a Native Alert

Mochi Stock has a Stock menu in the menu bar with one command, Export CSV. It answers with a native alert. Add the fourth test:

  - name: export from the menu bar
    target: stock
    steps:
      - action: { fill: { name: PIN, text: "{{secrets.STOCK_PIN}}" } }
      - action: { click: Unlock }
      - action: { click: Stock }
      - action: { click: "Export CSV" }
      - action: { assert_text: "Exported 4 products to stock.csv" }
      - action: { screenshot: export-alert }
      - action: { wait: 700 }
      - action: { press: Enter }
      - expect: "No alert is open and the stock list is visible"
Enter fullscreen mode Exit fullscreen mode

A menu item only answers when it is showing. So the test opens the menu first, then clicks the item. A name alone is the portable way to name a menu, because its role differs between systems.

The alert is not part of the window. tamesu lists what is open on top of the window after it, and searches it first. This is the end of the observation while the alert is open:

      text "Total: 53 units"
[3]  button "Close"
[4]  button "Full Screen"
[5]  button "Minimize"
     text "Mochi Stock"
    sheet "alert" (focused)
     text "Exported 4 products to stock.csv"
[20]  button "OK"
Enter fullscreen mode Exit fullscreen mode

The screenshot action saves what the test saw:

The native alert of Mochi Stock: "Exported 4 products to stock.csv" with an OK button

Good to know:

  • Use actions for the menu bar. With tamesu 1.22.0 on macOS 27, the agent does not get the titles of the menu bar in a goal step. A goal such as "export from the Stock menu" fails there. Two click actions work.
  • Close an alert with a key. press: Enter answers the default button. A click on the alert's own button ends in an error: agent-desktop reports that it could not verify the press.
  • Wait a moment before the key. Without the short wait, the key press can fail with a STALE_REF error.

Step 8: Run the Whole Suite

tamesu run
Enter fullscreen mode Exit fullscreen mode
tamesu run: 4 tests, output in .tamesu/runs/2026-11-17T14-44-45
PASS   stock / export from the menu bar  22.2s
PASS   stock / wrong PIN keeps the stock room closed  16.1s
PASS   stock / unlock and receive stock  30.7s
PASS   stock / search narrows the list  18.2s

4 passed, 0 failed, 0 errors in 1m27s · 6 model calls, 2685 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Enter fullscreen mode Exit fullscreen mode

Four tests, 1m27s and six model calls. Desktop tests take turns. A machine has one desktop, so tamesu runs them one at a time, whatever --parallel says.

Tags and filters select a part of the suite:

tamesu run --filter "wrong PIN"
tamesu run --tags desktop
Enter fullscreen mode Exit fullscreen mode

The app starts clean every time

Each test launches the app, and tamesu quits it at the end of the test. That is why every test of this suite starts at the lock screen with 53 units.

  • An app that was already running is yours. tamesu joins it and leaves it open at the end.
  • An app that refuses to quit, because it asks to save, is terminated.
  • State that an app keeps on disk is yours to reset. Use desktop.args on the target, or before_each and after_each hooks.

What a run leaves behind

File What it holds
screenshots/step-NN.png One screenshot of the window per step
steps.json Each step, its result, its reason and the agent's actions
agent-desktop.jsonl agent-desktop's own trace of what it read and did
transcript.json The steps as a conversation, for judge, contains and regex assertions
result.json Assertions, metrics and events
report.html The whole run as one page, next to junit.xml

The HTML report of the run: 4 passed, 0 failed, 6 model calls, and the history of each test

The screenshots and the trace cover the app under test only, never the rest of your desktop.

Step 9: Rename a Button

Now change the app the way a designer would. Rename the Receive button:

-                Button("Receive") { store.receive() }
+                Button("Add units") { store.receive() }
Enter fullscreen mode Exit fullscreen mode

Build the app again and run the first test:

./build.sh
tamesu run --filter "unlock and receive"
Enter fullscreen mode Exit fullscreen mode
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-47-51
PASS   stock / unlock and receive stock  56.5s

1 passed, 0 failed, 0 errors in 56.5s · 4 model calls, 5300 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Enter fullscreen mode Exit fullscreen mode

The test passes, and nobody edited the suite. Look at the model calls: four, not three.

The lock screen did not change, so the first goal replayed its plan. The stock screen has a new button name, so the key of the second goal changed. The agent planned that goal again, clicked "Add units", and tamesu saved the new plan. The metrics show it:

{
  "cache_hits": 1,
  "cache_misses": 1,
  "desktop_actions": 6,
  "goal_steps": 6,
  "llm_cost_usd": 0,
  "llm_tokens": 5300,
  "observations": 11,
  "turns": 0
}
Enter fullscreen mode Exit fullscreen mode

The next run replays both plans again.

Step 10: Catch a Real Bug

A rename is not a bug. This is one. In Store.receive(), a += becomes a =:

-        products[selected].units += Int(units) ?? 0
+        products[selected].units = Int(units) ?? 0
Enter fullscreen mode Exit fullscreen mode

The app now replaces the stock with the units you receive. Build and run:

./build.sh
tamesu run --filter "unlock and receive"
echo "exit code: $?"
Enter fullscreen mode Exit fullscreen mode
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-48-52
FAIL   stock / unlock and receive stock  32.5s
    x step[4] expect: The Canvas slip-on currently has 5 units in stock according to treeitem "Canvas slip-on 5 in stock", not 8.
    x judge "The steps show the app being unlocked and 5 units of the Canvas slip-on being received": Although the app was unlocked, 5 units were not properly received; starting from 3 units in stock, receiving 5 units should have resulted in 8 units, but only 5 units in stock were recorded.
    last turns:
      agent: replayed 4 cached actions (plan 60274b032ed3361120575328)
      user:  The Canvas slip-on has 8 units in stock and no low stock warning
      agent: The Canvas slip-on currently has 5 units in stock according to treeitem "Canvas slip-on 5 in stock", not 8.

0 passed, 1 failed, 0 errors in 32.5s · 3 model calls, 1578 tokens, $0.0000 · llm cache: 0 hits, 1 misses
exit code: 1
Enter fullscreen mode Exit fullscreen mode

Both goals replayed without a problem. The buttons are all there, and every click works. A test that only clicks would pass.

The expect step reads the window and fails: the Canvas slip-on has 5 units, not 8. The judge says the same about the whole session. The exit code is 1, so a pipeline stops here.

Put the += back and build again before you go on.

Step 11: Run It in CI

Replay in the pipeline

tamesu run --replay --out out
Enter fullscreen mode Exit fullscreen mode
tamesu run: 4 tests, output in out
PASS   stock / export from the menu bar  22.2s
PASS   stock / wrong PIN keeps the stock room closed  12.1s
PASS   stock / unlock and receive stock  32.5s
PASS   stock / search narrows the list  16.3s

4 passed, 0 failed, 0 errors in 1m23s · 6 model calls, 2724 tokens, $0.0000 · llm cache: 0 hits, 1 misses
Enter fullscreen mode Exit fullscreen mode

With --replay, every goal comes from the saved plans. The six model calls are the five expect steps and the judge, so the job still needs the API key. The exit codes are the same as for browser tests: 0 passed, 1 a test failed, 2 config error, 3 infrastructure error, 4 over budget.

Give the runner a desktop

The runner needs what your laptop needed in Step 1:

  • macOS runner: grant Accessibility to the process that starts tamesu.
  • Windows runner: run the job in the interactive session of the signed-in user.
  • Linux runner: most have no screen, so start a session first.

This is the Linux recipe from the tamesu documentation:

sudo apt-get install -y xvfb openbox dbus x11-utils at-spi2-core libatk-bridge2.0-0
export DISPLAY=:99
Xvfb :99 -screen 0 1280x800x24 -nolisten tcp &
openbox &
dbus-run-session -- tamesu run
Enter fullscreen mode Exit fullscreen mode

tamesu's own pipeline tests the desktop target this way on every pull request: against a GTK text editor in a headless Linux container, and against Notepad on a hosted Windows runner of GitHub Actions.

Split a large suite

Desktop tests run one at a time on one machine. To go faster, give each job its own machine and split the suite:

tamesu run --replay --shard 1/2    # job 1
tamesu run --replay --shard 2/2    # job 2
Enter fullscreen mode Exit fullscreen mode

Also New in 1.22: agent-browser for Web Tests

The same release adds a second engine to the browser target. Set browser.engine: agent-browser, and the agent-browser CLI reads the page and acts for the agent. The browser stays tamesu's own Chromium. Mocks, the accessibility audit, visual baselines and recordings work the same under both engines, and chromium stays the default.

Conclusion

A desktop test should say what the user wants, and fail only when the user would fail. The accessibility tree makes that possible. It turns a window into a few lines of text that a model can read and a test can assert on.

The rename and the bug show the difference. A renamed button costs one extra model call, once. A wrong number in the stock list fails the test with a sentence that says what is wrong.

Start small. Take the one flow of your app that someone still checks by hand before every release. Write it as two goals and two expect steps, and run it on your own machine first.

Further reading

Top comments (0)