I’m pleased to announce that corteza has scored 98.58% on the ARC-AGI-3 challenge with Claude Opus 5. corteza is an R-native agent harness built around a persistent R session: functions, data, and intermediate work survive across model calls, and even across context compaction.

wa30, a Sokoban-style carry game and one of the 23 games corteza finished at a score of 100. The rules are unknown by design; the model has to work them out.

Prime Intellect’s Prime Agent, a Python harness built around a similar RLM (recursive language model) idea, reports a 95.5% run. What interested me more than the score was how corteza got there, and how it failed. Both came from the same architectural choice: persistent state and compute outside the model’s context. That helped the model solve 182 of 183 levels, and also let it spend hours building and running its own combinatorial search on the one level it didn’t finish.

Prime Intellect’s announcement. Same model, different harness: Claude Opus 5 scored 30.2% in ARC’s reference harness and 95.5% in Prime Agent.

The scorecard

Our September 12th and September 14th ARC runs, next to Prime Agent’s 3 published runs and ARC’s own runs of GPT-6 Astra and Claude Opus 5, all on the same 25 public environments:

ARC publishes per-game scores for its own runs, not level or action counts.

Note: A previous version of this post conflated Astra’s performance on ARC’s public and semi-private games.

Twenty-three of the twenty-five game environments scored 100, meaning corteza completed every level in those games while matching or exceeding the human baseline in aggregate under ARC’s Relative Human Action Efficiency (RHAE) metric. Game tn36 finished all seven of its levels and scored 92.95, because level six took 158 actions against a human baseline of 55. cn04 finished five of six levels and was the only environment left unfinished. More on that below.

Prime Intellect reports three Claude Opus 5 xhigh runs at 94.99%, 95.24%, and 95.5%; their published scorecard is the median 95.24% run. Our 98.58% and 98.56% score came from the September 12th and 14th runs shown above. Their fully completed run used 12,439 actions, compared with our xhigh run’s 9,154: ours used 26.4% fewer actions.

ARC Prize’s Astra results make the harness point too. At max reasoning effort, averaging ARC’s published per-game scores across the same 25 public environments gives about 68.3% with the standard harness and 100% with the provider adapter. ARC describes the adapter as a harness that “preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work.”

ARC’s September 3 analysis separately reports semi-private results: 62.71% with the standard harness at max effort, 99.95% with the provider adapter at high, and 98.55% with the adapter at max. Those figures come from a different evaluation set and aren’t directly comparable to our public-game scorecards.

Caveats: They used Claude Opus 5 at xhigh reasoning effort. We used Claude Opus 5 at max and xhigh reasoning effort. They reported 3 complete runs. We’re reporting our only two complete runs that didn’t break due to technical difficulties like API limits and network disconnects (more on that below).

Background

Back in January, at the height of the clawdbot/openclaw hype, I wanted an agent harness I could inspect and understand thoroughly, something I could publish in the warm embrace of CRAN. It also seemed painfully obvious that a REPL like R or Python gives a model RAM that bash doesn’t. I posted on socials about it, but mostly nobody cared. A few months later it was useful enough to merit a blog post, it went up on CRAN a month after that, and a literature review a few weeks later convinced me that what I’d built fit the emerging RLM pattern. Over the summer we used corteza to power chatbots on our internal Matrix server to dogfood our AI and Matrix packages.

When Prime Intellect announced Prime Agent on X, it looked like the Python RLM analogue of corteza, and the announcement touted its ARC-AGI-3 result. I figured I should at least take a stab at the challenge.

Rough edges

API fun

If your harness loses connection, you need retries and reconnects or you will repeatedly get closed down mid-run and either have to restart or have to point at 3 different scorecards and say, “Believe me it just timed out!” There were more than a few iterations on the ARC-AGI-3-specific connections. I’ve put those into a repo on GitHub just in case anyone else has tokens to burn on experimental agentic harnesses for R ;)

The ARC API caps an online scorecard at 24 hours. In practice, we ran into closure substantially earlier than that on some long-lived runs. As a solo dev working off of a Claude Max 20x subscription and a live harness (Prime Intellect’s runner plays an offline local copy of the games; ours plays live against the ARC API), you can’t run all 25 games at once or you’ll anger your API provider. If that doesn’t get you an error, you’ll run through your 5-hour limit. On the other hand, if you run 1 at a time, at an average of 1 hour per game, you’ll time out your ARC scorecard.

ARC-AGI-3 also documents a 15-minute inactivity timeout for scorecards, although in our live runs the effective behavior was less predictable. A long-running R call produces no ARC activity at all, which is how we got our only failure.

cn04

cn04 is a jigsaw-style game: rotate and slide pieces until their tabs mate. Over 270 run_r calls the model wrote 63 helper functions into its R session and kept building on them: a frame parser from level 1 was still being called on level 6, and a coordinate-key helper from level 3 sat inside every search it wrote afterward. On level 4 it turned a level-3 rotation helper into a piece rotator and wrote a recursive backtracking search over piece order, rotation, and tab pairings that picks the cheapest valid assembly, which solved the level in under a minute of compute.

Full replay on arcprize.org.

On level 5 it wrote 7 variants of that search on the same helpers, two of which each ran for about 45 minutes before it solved the level. Level 6 has 2 pieces that can grow, and after a 48-minute search and a 56-minute search that found nothing, the agent looped an exhaustive search over 7 candidate widths for one of them: just under 8 hours. That call did return, the agent immediately started another full search, and 13 minutes later the driver’s per-game wall clock killed the R process. cn04 ended at 5 of 6 levels with 5 actions spent on the final level, and ARC closed the scorecard and published it incomplete.

To reiterate, it didn’t burn any ARC actions doing this. The prompt told it, verbatim: “R analysis is free; game actions consume the score and the fixed run budget.” It took me at my word (R compute wasn’t entirely free) and held one CPU core for hours. I think the cool kids with government contracts call this an alignment problem.

Ironically, the one environment we didn’t finish is also the clearest demonstration of why I built corteza this way in the first place. Any model can write an R search that runs too long. The persistent session let this one assemble that search from pieces it had already written and tested on earlier levels, without re-sending or re-running the definitions, and keep the computation outside the model’s context and outside the ARC action budget. That’s the REPL RAM idea from January working as intended. Helpers written hundreds of turns earlier stay live and composable. Here’s the chain in the trace, trimmed to the definitions and one later use:

# turn 13, level 1
frames <- function(fr) lapply(fr$frame, tomat)
# turn 24, level 1
bmat <- function(m){ B <- matrix(NA_integer_, 20, 20)
  for(I in 1:20) for(J in 1:20){ sub <- m[(3*I):(3*I+2), (3*J):(3*J+2)]
    B[I,J] <- if(length(unique(as.vector(sub)))==1) sub[1,1] else -1 }
  B }
# turn 141, level 3
rot1 <- function(cells) cbind(cells[,2], -cells[,1])   # CW about origin
# turn 145, level 3
k <- function(m) paste(m[,1],m[,2])
# turn 207, level 4: rot1 goes into the piece rotator the search uses
rotP <- function(p,k){ b<-p$body; t<-p$tabs; for(i in seq_len(k)){ b<-rot1(b); t<-rot1(t) }
  a<-rbind(b,t); oi<-min(a[,1]); oj<-min(a[,2]); list(body=cbind(b[,1]-oi,b[,2]-oj), tabs=cbind(t[,1]-oi,t[,2]-oj)) }
# turn 425, level 6: frames and bmat from level 1
rev <- function(bgc=9){ gz<-frames(fr)[[length(fr$frame)]]; Bz<-bmat(gz)  # ...
# turn 435, level 6
R6b <- searchA2(P6b); length(R6b)

The same pattern held across all 25 games. The model defined 894 helper functions over 7,942 R calls, and at least 3 in 4 of those calls used a helper it had written earlier in the same game. On the 9 games where the harness compacted the model’s context, 53 helpers written before the compaction were still being called after it. The R session persists even when the context doesn’t.

The cn04 search ran for far too long, and that’s the bug, but the architectural bet worked almost too well. In response, I’ve added timeouts and context to all corteza processes. For ARC, a run_r call now gets at most 600 seconds, less as the 15-minute inactivity deadline approaches, and a timeout tells the model to reuse helpers, split the work, or approximate.

Getting our reps in

Software gets better the more it’s used… at least up until a point! It’s been our strategy at cornball.ai to get our tools into our production processes as quickly as possible so we can sand down the rough edges. The bugs you see at a million calls get missed at a hundred calls.

That’s why we’re super grateful for the ARC-AGI-3 challenge and for Prime Intellect open-sourcing their Prime Agent harness. The challenge has provided desperately needed reps. Prime Intellect has raised more than $150m. cornball.ai has raised $0. They can afford dramatically more benchmark reps than a solo dev on a Claude for Open Source Max 20x (thanks Anthropic!), which makes their open-source runner particularly valuable: it gave me a well-developed reference implementation to test corteza against.

What we took from Prime Agent

The prompt, mostly.

Prime Intellect’s ARC-AGI-3 runner is open source under the MIT license, and we used it. Our prompt carries some closely adapted language from their behavioral guidance and initial game prompt, and is dutifully credited under their license. Being able to skip past that part allowed me to quickly iron out the details with the challenge’s API and find some real bugs in our own harness stack.

What’s next

An xhigh-effort run with the timeout fix is in progress, which will put a second data point next to the caveat above. corteza also keeps powering the chatbots on our Matrix server, which is where most of its reps come from.

If you’re working on agents, persistent computational environments, or RLMs and want to collaborate, or want to help push this architecture further, get in touch.

Try it

corteza is on CRAN.

install.packages("corteza")

The dev version is on GitHub:

remotes::install_github("cornball-ai/corteza")

• Previous
  janssonr is on CRAN, and mx.* can verify devices now