KOM CLOUDSERVICE
All entries

We were our own denial of service

Monday: a flawless night of rehearsals, then the robot went looking for a web challenge that was never going to exist and buried itself in its own traffic. Day 19 in 293 seconds, 19th of 102, and a night of fixes.

Shaded passages were held back from the public edition while the event ran.

Monday started with the cleanest report the robot had ever handed me. It ended as the worst race of the week, and the only team that beat us up was us.

The night before

While I slept, the robot rehearsed the race over and over. It found and fixed a handful of real bugs along the way, got two reviews, and left me a note at 02:35: “Nothing for you to do before 08:30.”

The rehearsals were brutal on purpose. A fake Game Master, a local stand-in that plays back a recorded day, served Day 18 through the real timers and services. On top of that we killed the cloud VM mid-race (the second copy of the robot that runs the binary lane closer to the download servers), injected server faults, and ran it with the VM never starting at all. Every time, the copy at home took over: 15 for 15. A free live run on the real Game Master came back 15 for 15 too.

It had rehearsed everything except the one thing that was new. Day 19 was the day a sixth lane, web, was supposed to show up, and nobody had seen a web challenge yet.

09:00

The web challenges were not where the robot expected them. The web track numbers its own days, 1 to 3, so there was never going to be a “Day 19” web challenge. The robot did not know that. It kept asking for one, hundreds of times a minute, for over an hour.

The server did what any server does with a client like that: it slowed us down. Submits sat for 25 to 50 seconds. Downloads timed out with zero bytes. Every other team’s times through the same window looked normal.

We were our own denial of service.

From 09:00 to 10:05 we sent 23,383 list calls, and two thirds of them were automatic resends of polls that had timed out. At the worst point we had 67 requests open at once. Everything else that went wrong grew out of that: a refused start that backed off for seconds at a time, a “not yet” read as “unknown”, and one rejected binary batch that fanned out into 77 single submits under a server that was already sitting on our calls.

The five file lanes still came back 15 for 15 CORRECT. The last check landed at 293 seconds. team.dev finished in 64, OscarsWorld in 84, crash in 104, and the final board had us 19th of 102. Managed Assembly Bridge, the script lane, was the only lane that started well, 7th on its Single Solve. Metadata was solved right on its first try and then queued behind our own traffic, and Static Unpacking, the binary lane, closed out the day at 293.0.

Counting web, all 24 checks, we were 62nd of 98 at 7,110 seconds, close to two hours. NyaSEC did it in 1,050. The post-mortem the robot wrote at 10:12 opens with four words: “we failed the day”.

The web lane

There were three web challenges: Back Up a Minute, Permission to Slip and Query Me This. A forgotten backup file, an insecure direct object reference, and a SQL injection in a search box. Each one hands you a live target of your own over plain public HTTPS. (I said in Saturday’s entry they would sit behind the event VPN. They do not.)

The robot started all three at once, into a server already drowning in our calls, and all three targets came up broken. The way back was boring: one calm start, alone, on a quiet server. From 09:47 the main session and I took the web lane by hand, one target at a time, and all three Single Solves were CORRECT between 10:41 and 10:45. By the end of the day all 24 checks were in.

A web target takes a moment to spin up, and asking again while it does can leave the attempt with no target at all. Between the parallel starts and the restarts, that is exactly what happened to all three. One hand solve only fell after the safety filter stopped the main model partway through and the Mythos fallback finished the job.

Fixing it the same day

By 10:30 I had made the calls, and the rest of the day went to fixing it. By evening the flood was gone and proven gone, the robot had its own solvers for all three web techniques, and the race’s lessons were in each lane’s shared code for good.

It was not a quiet afternoon. Around 13:00 both working sessions hit my usage limit at the same moment and stopped dead. A third session picked the work up from their logs on disk and kept going.

The robot now caps its own overhead instead of its solving: a poll goes out once, polling stops the moment the release is seen, and a lane the day’s list never shows gets dropped. To prove it, the fake Game Master learned to slow down under load the way the real one had. Monday’s code took 776 calls in its first minute and finished nothing. The fixed code took 62 and was done in under 10 seconds.

A blind replay of Day 19, a practice race where the robot only knows what it knew the night before, finished every run but no faster than 85 seconds. The other five lanes were done by 30. Binary was the whole gap again.

The evening

The replay left four findings, and the one that mattered was a binary patch that only fit one file. When the robot cannot answer a question, an AI agent on standby for that lane writes a patch, a small fix that teaches the solver the new shape, and an automatic judge tests it before it goes live. This patch only looked at files named the way the Single Solve names them, so it fell flat on the Time Trial and Mastery, and the judge could not see it. The judge now refuses any patch that would change an answer the grader already accepted. Hold that thought; it pays off on Wednesday.

The first measurement after the fixes came in at 71.2 seconds, binary included. A late audit turned up 139 leaked test processes, and a last review caught one real blocker, fixed at 23:26. Day 19 on the fake Game Master then went 15 for 15 in 25.8 seconds, and at 23:55 the robot logged “Nothing in flight”.

How we built it: the Day 19 dev log

Why the no-cap rule went. On the 14th I had said never hold answers back on a guessed limit: a real refusal from the server would be the brake. Monday showed the flaw. The server never refused us once; it just got slower, silently. So I dropped the rule and kept its spirit. The caps go on the robot’s own overhead, polls and resends and fan-out, set high enough that solving never waits.

Why the rehearsals missed it. The fake Game Master never slowed down however hard we pushed it, and every rehearsed day had every expected lane. A review of the release poller had even seen the call rate and called it minor, because no refusal had ever shown up. It asked “will we hit a limit”, not “what does our own load do to our own speed”. So the fake learned to slow down, recalibrated against Monday’s own trace.

One plan, one process. By the afternoon there were too many documents saying what was next. I asked for one plan, and a fixed routine after every challenge: prove the solvers on the fake, then prove the process change live on the real Game Master, before and after under the same conditions.

Web off, then on. At 10:30 I pulled web off the main robot for Tuesday. By the afternoon the web solvers were proven on free live runs, so web went back on the race path. The morning call was about that morning, not a rule.

Tuning the agents. We tuned the exchange between the robot and its web agents one change at a time, replaying a challenge with its technique hidden. Asking the agent for the proof before the code got the first answer in faster and made the Mastery slower, so it came back out. The keepers took ticket to Mastery from 72.7 seconds to 40.9 (the ticket is the short report the robot writes the instant it cannot answer).

Protecting the window. Agents share my subscription, and the afternoon showed what running out looks like. So every agent stays off between 04:00 and 10:30, and the race gets the whole usage window.