DEV Community

stepinto.dev
stepinto.dev

Posted on Edited on Originally published at stepinto.dev

Planning Against the Model

The last post gave Claude the room with the door and lost the first round. Claude fitted the physics from six readings, and my thirteen-weight model from 104, handed to Claude as a tool, talked it into a heating plan that left the room at 10.6 degrees. The second round went the other way once the model borrowed the shape Claude had described: a leak proportional to the gap between inside and outside, a constant from the heater, and a second leak that comes and goes. That model got one code snippet and one table at the end of a long post, and a promise that this one would build it properly and put a planner on top.

The code is in tools/heater_plan.ts, and the playground now has the model and the planner in it.

The model Claude described

Here is the class. The private field, reset and observe are 104's, line for line. Only act is new.

class PhysicsModel {
  #field = 0;
  w: PhysicsWeights;
  constructor(w: PhysicsWeights) { this.w = w; }

  reset() { this.#field = 0; }
  get field() { return this.#field; }

  act(state: State, action: Action): State {
    const w = this.w, heaterOn = heaterAfter(state, action);
    const gap = state.outsideTemp - state.roomTemp;
    const change = (w.leak + w.doorLeak * this.#field) * gap + (heaterOn ? w.heat : 0);
    return { roomTemp: state.roomTemp + change, outsideTemp: state.outsideTemp, heaterOn };
  }

  observe(state: State, action: Action, nextState: State) {
    const f = inputs(state, action), w = this.w;
    const change = (nextState.roomTemp - state.roomTemp) / 5;
    this.#field = Math.tanh(
      w.uField * this.#field + w.uRoom * f.room + w.uOutside * f.outside +
      w.uHeater * f.heater + w.uChange * change + w.uBias);
  }
}
Enter fullscreen mode Exit fullscreen mode

The 104 act had seven weights and let the gradient decide how room, outside, heater and field combined. This one has three and decides for it. The room and outside temperatures only ever appear as their difference, the heater only ever adds a constant, and the field only ever scales the leak. Read as physics, that says heat flows out of the room in proportion to how much warmer it is than outside, the heater puts in a fixed amount, and something the log never mentions makes the room leakier. That is realRoom from 104, with the door replaced by a number the model has to infer.

Nothing in the logs or the training loop changed to accommodate this. The 4,800 rows are the same rows, and training is the same walk forwards and backwards through each episode, with three lines different. The prediction error uses the new act:

const gap = (t.state.outsideTemp - t.state.roomTemp) / 10;
const err = (w.leak + w.doorLeak * field) * gap * 10 + w.heat * f.heater - target;
Enter fullscreen mode Exit fullscreen mode

The gradients for the three prediction weights follow from it:

g.leak += s.err * s.gap; g.doorLeak += s.err * s.field * s.gap; g.heat += s.err * s.f.heater;
Enter fullscreen mode Exit fullscreen mode

And the line that passes blame back to the previous step's field now asks how much the field mattered through the door leak:

dField = s.err * w.doorLeak * s.gap * 10 + dPre * w.uField;
Enter fullscreen mode Exit fullscreen mode

The gap is divided by ten inside the loop and the two leak weights get a tenth of the learning rate. A gap of minus fifteen degrees would otherwise give the leak gradients fifteen times the push of the heater's, and the loop would spend its epochs bouncing across the valley the 103 post described. With that, three thousand epochs over 4,800 rows take 454 milliseconds in Node 23.

Nine weights, read back

weight learned weight learned
leak 0.102 uField 0.18
doorLeak 0.154 uRoom -0.24
heat 1.16 uOutside 0.24
uHeater 0.76
uChange -3.15
uBias -0.56

The left-hand column needs the field before it means anything, because the leak the model uses is leak + doorLeak * field. On the test room from 104 (22 degrees, minus four outside, heater on throughout, door open from step 10 to step 20) the field sits around -0.17 for the first ten steps, which puts the leak at 0.076 on average over those steps and 0.072 at step 5, where the planning question below starts. The true closed-door leak is 0.07. The step after the door opens, the field jumps to 0.97 and the leak to 0.252, against a true 0.25. The heater weight is 1.16 against a true 1.2.

The model did not put "door shut" at a field of zero. It put it at about -0.17, and gave zero a leak of 0.102 that the room never has. The field's scale is the model's own business, as it was in 104, and only the leak that comes out of it can be checked against the room, which it matches.

The right-hand column is the same recipe for the field as last time. uChange dominates again: the field is mostly a reading of how far the room fell last step, with the heater and temperatures taken off. uField, its memory of itself, is 0.18 this time against 0.007 in 104, so this model holds on to the door slightly better.

Step by step through the door, next to the 104 model:

step door real room physics-shaped, one step ahead field after 104, one step ahead field after
10 closed 17.19 16.83 -0.23 16.97 -0.15
11 open 13.14 16.94 0.97 16.90 0.91
12 open 10.10 9.98 0.96 11.16 0.86
13 open 7.76 7.73 0.93 8.52 0.81
20 open 1.94 2.31 0.64 2.27 0.59
21 closed 2.83 1.91 -0.04 1.77 0.12
22 closed 3.55 3.34 -0.11 3.56 0.15

Step 11 is a miss for both, as it has to be. At step 12 the physics-shaped model is a tenth of a degree out where the 104 model was over a degree out. Leaving out the two steps where the door moved, the one-step error over the thirty steps is 0.26 degrees against 0.32.

On short forecasts inside the range it was trained on, the two models are close, and not always in the physics-shaped model's favour. Asked at step 5 to run five steps blind, it says 16.92 against a real 17.19, where the 104 model said 17.12. At step 23, two steps after the door shut, its field has not quite settled and it forecasts 5.84 against a real 6.80, where the 104 model said 6.11. Its fit to the log is slightly worse too, a loss of 0.179 against 0.172. A model with four fewer weights than its rival fits the data it was trained on a little less closely, which is what you would expect, and the difference between the two shows up outside that data.

Why the bias had to go

The test that broke the 104 model in 106 was the heater left off for twenty steps from the step 5 reading, at minus four outside, door shut. Every outside temperature in the log was between 2 and 12, so this is a long way outside it.

Heater off for twenty steps at minus four outside, door shut, five models

Heater off for twenty steps from 19.15 degrees at minus four outside. Solid lines are the real room, averaged over 200 runs, the physics-shaped model and the 104 model. Dashed lines are the two fixes that failed: the physics-shaped model with a bias term, and the 104 model refitted on logs down to minus six. Both end colder than the air outside.

Download the data (CSV)

The physics-shaped model ends at 1.15 against a real 1.42. The 104 model ends at 5.25. The two dashed lines are the two things I tried before settling on the model above, and both fit the log well.

The first is the physics-shaped model with a fourth prediction weight, a bias, added to act the way every model in this series has had one. Trained from the same starting point, it reaches a loss of 0.1793, against 0.1792 without it. On the log the two are indistinguishable. Its leak, with the field where the test room leaves it at step 5, is 0.068, which is about right. Its heater is 1.22 and its bias is -0.504, a constant half-degree pull downwards every step, and on the test room it ends at -3.96. Set the change to zero with the heater off and the bias model says an unheated room settles where the leak balances the bias, 7.4 degrees colder than outside. On the rows in the log, the other weights pay that half degree back, and the predictions come out the same as the model without a bias. With the heater off for twenty steps in a cold room, nothing pays it back.

The second is more data. If the problem is that the log never saw minus four, record a log that has, so I refitted the thirteen weights on 4,800 rows with outside temperatures from minus six to twelve. The loss came out at 0.227, and the heater-off room ended at -4.85, colder than the air outside it, which no room without a fridge in it can be.

The bias came out because the physics has none, and because Claude's description in 106 had none. I made that call after seeing the bias version fail, so the result is not a blind test of the shape. The shape is the part of the model that carries knowledge from outside the log, and the log only has to supply three numbers. If the shape is wrong, as with the bias, no amount of fitting will notice, because the error only shows up where there is no data.

Searching for the plan

Back to the question from 106. At step 5 the room reads 19.15, it is minus four outside, and the door is shut. Keep every reading from step 6 to step 25 at or above 12 degrees, with the heater on for as few steps as possible.

A planner needs a way to score a schedule, so here is the score. One point per heater step, a hundred per degree that any predicted reading falls short, and a hundredth of a point off for having room to spare, so that between two plans with the same number of heater steps the safer one wins.

function cost(temps: number[], heater: boolean[], target: number[]) {
  let c = 0, slack = Infinity;
  temps.forEach((t, i) => {
    if (heater[i]) c += 1;                                  // one per heater step
    if (t < target[i]) c += 100 * (target[i] - t);          // a hundred per degree short
    slack = Math.min(slack, t - target[i]);
  });
  return c - 0.01 * Math.min(1, Math.max(0, slack));        // and, between equals, the one with more room to spare
}
Enter fullscreen mode Exit fullscreen mode

temps comes from rolling the model forward under the schedule, from the step 5 reading, with the field where observe left it. That is the "what if?" from 101, asked once per schedule.

Twenty steps of on or off is 1,048,576 schedules, and with a model this small, trying all of them takes about a second. The cheapest on the physics-shaped model is sixteen heater steps: off for four, then on for sixteen. Run in the real room 200 times with fresh draughts, it never drops below 12. The same search on the 104 model finds thirteen, and the real room under that plan drops to 10.83 on average and fails in all 200 runs. With the true constants the answer is fifteen, which leaves a tenth of a degree to spare and dips below 12 in 75 of the 200 runs. Those three numbers are the 106 result again, reached without Claude doing the searching.

Trying everything stops working fast. Every extra step doubles the count, so forty steps would take about twelve days at the same speed, and a planner that decides every minute needs its answer well inside a minute. A real planner has to be satisfied with trying some of the schedules.

Random guesses, and better ones

The simplest thing that tries some of them is random shooting: draw schedules at random, score each one, keep the best.

function shoot(predict: (h: boolean[]) => number[], target: number[], n: number): Plan {
  let best: Plan = { heater: [], cost: Infinity };
  for (let k = 0; k < n; k++) {
    const heater = target.map(() => rand() < 0.5);
    const c = cost(predict(heater), heater, target);
    if (c < best.cost) best = { heater, cost: c };
  }
  return best;
}
Enter fullscreen mode Exit fullscreen mode

The trouble is that the good schedules are rare. A random schedule has the heater on for about half its steps, and the answer needs sixteen of twenty, mostly at the end. I ran each method a hundred times from different seeds.

method schedules tried plans that keep the room at 12 plans with 16 heater steps time per plan
every schedule 1,048,576 100 of 100 100 of 100 about 1 s
random shooting 100 4 3 0.1 ms
random shooting 1,000 39 12 0.7 ms
random shooting 10,000 100 69 6.3 ms
random shooting 100,000 100 100 63 ms
cross-entropy method, 100 × 6 rounds 600 100 98 0.5 ms
cross-entropy method, 200 × 6 rounds 1,200 100 100 0.8 ms

Random shooting gets there with a hundred thousand guesses. The last two rows get there with about a thousand, because they stop guessing blindly after the first round.

The cross-entropy method keeps a probability for each step that the heater is on, starting at a half. Each round it draws a batch of schedules from those probabilities, keeps the best tenth, and resets each step's probability to how often the best tenth had the heater on at that step. After one round the early steps drift towards off and the late steps towards on, and the next batch is drawn mostly from the neighbourhood of plans that worked.

function cem(predict: (h: boolean[]) => number[], target: number[], samples = 200, rounds = 6): Plan {
  let odds = target.map(() => 0.5);
  let best: Plan = { heater: [], cost: Infinity };
  for (let r = 0; r < rounds; r++) {
    const tried: Plan[] = [];
    for (let k = 0; k < samples; k++) {
      const heater = odds.map(p => rand() < p);
      tried.push({ heater, cost: cost(predict(heater), heater, target) });
    }
    tried.sort((a, b) => a.cost - b.cost);
    if (tried[0].cost < best.cost) best = tried[0];
    const elite = tried.slice(0, samples / 10);
    odds = odds.map((_, i) => elite.filter(e => e.heater[i]).length / elite.length);
  }
  return best;
}
Enter fullscreen mode Exit fullscreen mode

All hundred runs at 200 a round found a sixteen-step plan, and 78 of them found the same plan the exhaustive search did. The others found different sixteen-step plans with less to spare. Each run takes under a millisecond. The model does about 6.4 million twenty-step rollouts a second, so the time goes on drawing and sorting the schedules rather than on the model.

This is the planner in PlaNet, the 2019 paper from Danijar Hafner and colleagues that came before Dreamer. Its world model is a recurrent state-space model with hundreds of numbers where mine has one field, and its actions are continuous, so its odds are a mean and a variance rather than a probability per step. The loop is the same, and its settings are larger than mine: twelve steps ahead, ten rounds, a thousand candidates and the best hundred kept. The later Dreamer papers replaced the search with a policy network trained inside the model, which is faster at decision time and a great deal harder to explain. For a heater, a millisecond of search per decision is cheap enough.

Planning again every step

A plan made at step 5 is a forecast of twenty steps, and the 104 post showed what twenty-step forecasts are worth when the room does something the model did not expect. PlaNet's answer, and the standard one, is to plan at every step, carry out only the first action of the plan, look at what the room actually did, and plan again. This is model predictive control, and with a model that observes it costs nothing extra to set up: observe folds each new reading into the field, so the next plan starts from a model that knows whether the door is open.

const replan: Controller = (ev, i, s, model) => {
  const target = Array.from({ length: Math.min(20, ev.steps - i) }, (_, j) => wantAt(ev, i + 1 + j));
  return cem(h => model.predict(s, h), target).heater[0];
};
Enter fullscreen mode Exit fullscreen mode

Each decision is one call to cem, twenty steps ahead or to the end of the evening, whichever comes first, and only the first step of the result is used. The other nineteen are thrown away and worked out again a step later with one more reading.

To test it I wanted a job where planning ought to matter, so the planner gets an evening. It lasts forty steps and the room starts at 14. Someone is out until step 25 and only wants the room kept at 12 or above until then. From step 25 they want it at 18. The heater adds about a degree a step, minus the leak, so it takes a dozen or more steps to warm the room, and anything that waits until step 25 to start will be late.

The thermostat it runs against comes on below a degree above what is wanted and goes off 0.7 above that, so it sits on the right side of the target with draughts. Left alone it switches to the evening setting at step 25, so it is late every time. A competent installer would set it to switch early, so the thermostat gets a lead, a number of steps before arrival when it moves to 18, and I tuned the lead for each outside temperature: the cheapest lead that leaves fewer than a quarter of a reading short, over 200 evenings. The right lead is 18 steps at 3 degrees outside, 11 at 5, and 7 at 8.

Every controller got the same 200 evenings, with the same draughts in the same order, run by run.

outside controller heater steps readings short of what was wanted
3° thermostat, no lead 29.8 15.0
3° thermostat, lead tuned for 5° (11 steps) 34.1 4.6
3° thermostat, lead tuned for 3° (18 steps) 36.7 0.1
3° planner 35.5 1.3
5° thermostat, no lead 27.2 9.7
5° thermostat, lead tuned for 5° (11 steps) 31.0 0.1
5° planner 30.7 0.2
8° thermostat, no lead 21.1 6.3
8° thermostat, lead tuned for 5° (11 steps) 24.6 0.0
8° thermostat, lead tuned for 8° (7 steps) 23.4 0.0
8° planner 22.8 0.1

On any one night, a thermostat with the right lead matches the planner, a heater step either way. On the night the lead was tuned for, they are level. The thermostat is not told what night it is. Tuned for 5 degrees and run on a 3 degree night it starts heating seven steps too late, and the person comes home to a room that is short of 18 for four or five readings. Run on an 8 degree night it starts four steps too early and burns 1.8 heater steps more than the planner, about 8 percent. The planner has no lead to tune. It finds 18 steps on the cold night and 7 on the mild one because it can ask the model how long the warm-up will take.

One evening at three degrees outside: the planner against a thermostat tuned for five

One evening at 3 degrees outside, the one whose results sit closest to the averages over all 200. The planner starts heating almost at once, reads 17.8 at step 25 and stays above 18 from then on. The thermostat, tuned for a 5 degree night, starts later and reads 17.0 when the person arrives. Heater steps are shown underneath.

Download the data (CSV)

The planner misses some readings too: 1.3 an evening at 3 degrees. It plans to the target and no further, so a plan that the model says will reach 18.02 at step 25 can meet a draught and read 17.95. The thermostat has its degree of margin built in. The planner has none, because the model gives it one number per step and no idea how far to trust it. That is the subject of the next post.

The 104 model, used as the planner's model on the same evenings, does as well as the physics-shaped one: 35.4 heater steps and 1.4 readings short at 3 degrees, 22.1 and 0.3 at 8. These evenings run between 3 and 8 degrees outside with the room between 12 and 20, which is inside the log it was trained on, and inside its log the 104 model was always good. It failed in 106 at minus four, twenty steps from anything it had seen.

What the door does to both

The same 200 evenings again, with someone using the door the way they do in the logs: about once every 25 steps, for about 8.

outside controller heater steps readings short
3° thermostat, lead tuned for 3° 37.1 15.9
3° planner 36.8 16.3
5° thermostat, lead tuned for 5° 33.4 12.8
5° planner 33.7 12.5
8° thermostat, lead tuned for 8° 27.4 6.4
8° planner 27.1 6.6

The planner's field picks up the door one step after it opens, as it did in the table above, and the plan it makes next assumes the leak that goes with an open door. That does not help, because there is nothing useful to do differently. At 5 degrees outside with the door open and the heater on, the room heads for 9.8 degrees. No schedule holds it at 18, or at 12, and the best any controller can do is heat and wait. With the door in play the two are level, and at 5 degrees both spend about a third of the evening short.

A planner with a door it can see would do better in one specific way. If the door opened at step 15 and the model knew it would shut at step 20, the right move would be to stop heating a room that is pouring heat out of the door and spend those heater steps after it closes. My model cannot plan that way, because its field only says the door is open now, and the rollout holds the field fixed for all twenty steps. A field that could forecast its own future, with some idea of how long doors stay open, would have something to plan around. Claude did this in 106 when it noticed that door events in the log end after two to eighteen steps.

Try it

The playground door scenario now trains three models when you press Train: the four-weight model, the latent model from 104, and the physics-shaped model. The right-hand chart adds the physics-shaped model's forecast in green, and the field strip shows both fields against the door. Its weights table reads the leaks off the field the same way this post does.

Underneath is a new section that plans an evening. The sliders set the outside temperature, the target, when the person arrives and the thermostat's lead, and a checkbox brings the door in. Press Plan and the browser runs forty plans of 1,200 rollouts each, which takes about twenty milliseconds, and draws the room under the planner and under the thermostat with the target line and both heater strips. Move the outside temperature and leave the lead alone to see the result above: the thermostat is either late or wasteful, and the planner adjusts. The playground uses Math.random where the script has a seed, so its numbers wander a little from the ones here. The tables above are in heater-plan-evenings.csv and heater-plan-search.csv.

On the 106 question the planner chose sixteen steps because the model said fifteen would not do, and the model was right partly by luck: it is six tenths of a degree pessimistic at the bottom of the plan. The true fifteen-step plan, the one with a tenth of a degree to spare, fails in 75 of 200 draws, and a planner given the true constants picks exactly that plan. What the planner needs is a model that says how sure it is, so that it can keep a margin where the model is unsure and drop it where the model knows. The next post fits several models on resampled logs, uses how far they disagree as the measure of doubt, and has the planner refuse plans that run through the places where they disagree most.

Top comments (0)