Independent AI research lab · Austin, Texas
Think.dev researches how intelligent systems should be built: the primitives of AI engineering, so an agent's behavior stays legible to the engineer who is accountable for it, and the structure of statistical learning, so a model's effects can be read and controlled rather than merely scored. We publish the tools as open source. We partner with a small number of teams whose problems require original thinking.
from thoughtflow import LLM, MEMORY, DECIDE, ACTION # Memory is the state. Every step reads it and returns it. llm = LLM("openai:gpt-4o") memory = MEMORY() memory.add_msg("user", "Why did the churn model flag account 4471?") # A DECIDE is one LLM call that lands on a named choice. route = DECIDE(name="route", llm=llm, prompt="Given {last_user_msg}, pick the next step.", choices=["explain_flag", "answer"]) # An ACTION is a plain Python function, recorded into memory. explain = ACTION(name="explain_flag", fn=explain_flag) # The loop is yours. Nothing runs that you did not write. for turn in range(4): memory = route(memory) if memory.get_var("route_result") == "answer": break memory = explain(memory) print(memory.render(format="conversation"))
The thesis
A language model is the most capable component ever made available to an engineer, and it is also the least inspectable. The lab exists in the gap between those two facts. Everything we publish is an attempt to put capability inside a form that a person can read, test, and be accountable for.
The unit of value in an intelligent system is a decision that holds up: a choice made with a reason, at a cost, under a constraint. Completions are the raw material. The system that turns raw material into a defensible decision is what we build, and it is not the model.
Prompting is a fine skill and a poor foundation. What happens before the call (memory, retrieval, the shape of the question) and after it (verification, action, record) determines whether the system deserves to be trusted. That is an engineering discipline, and it is mostly missing.
A black-box model can be extremely useful. A black-box system cannot be deployed responsibly for long. Our research question is how to house learned capability inside structures where the engineer can see the state, name the transitions, and replay the run.
The research programs
In AI engineering the question is where the control flow lives: in a runtime someone else wrote, or in code you can read. In statistical learning the question is where the structure lives: in a score you can only compare, or in effects you can name. Both programs are about the same refusal: we do not accept a capable component as an excuse for an illegible system.
The two programs feed each other. Agents need models whose reasons can be surfaced. Models need systems that decide when, and whether, to act on them. A lab that only did one half would keep tripping over the other.
What is the smallest set of parts from which any agent can be assembled, in ordinary code, with the state always visible? ThoughtFlow is the running answer. The Foundation Agent program tests it against real behavior.
How do you embed machine-learned effects inside a formula a domain expert can read, and keep the whole thing causally coherent as it converges? CGEM is the running answer.
Program A · AI engineering · ThoughtFlow
Most agent frameworks sell top-level simplicity by hiding the loop. ThoughtFlow does the opposite: fourteen small primitives, a memory object that is the whole state, and a loop you write yourself. It looks like more code. It is less system.
The engineer who will be accountable for the behavior should be the one who wrote the control flow.
Hidden runtimes are pleasant on day one. On day forty, when a customer asks why the agent did something, the answer is buried inside a scheduler you cannot step through. ThoughtFlow keeps every decision point in code you own. The trade is simple: you write the loop. In exchange, there is no behavior in the system that you did not author.
The same pattern played out once before. Multilayer perceptrons became useful when engineers could see and shape every layer; the frameworks that abstracted the architecture away did not survive contact with hard problems. We expect agent engineering to rhyme.
# Every primitive has the same shape: memory in, memory out. # So the control flow is plain Python, and it is yours. while not memory.get_var("done"): memory = plan(memory) # THOUGHT memory = act(memory) # ACTION memory = reflect(memory) # THOUGHT memory = check_done(memory) # DECIDE # What happened? Every step is in the log, in order. for event in memory.get_events(): print(event["type"], event["content"])
from thoughtflow import LLM, MEMORY from thoughtflow.eval import Harness, TestCase # Record once against a live model... recorded = MEMORY() live = LLM("openai:gpt-4o").record(recorded) support_flow(MEMORY(), live) recorded.save("fixtures/refund.json") # ...then replay forever: offline, no keys, same decisions. replay = LLM.replay(MEMORY.from_json("fixtures/refund.json")) def escalates(memory): return memory.get_var("route_result") == "escalate" case = TestCase(name="broken item above authority", messages=[{"role": "user", "content": "Refund order 8812, it arrived broken."}], check=escalates) results = Harness([case]).run(lambda m: support_flow(m, replay)) assert results.passed_count == results.total_count
Models do not give the same answer twice, so tests that check the answer are flaky by design. ThoughtFlow tests check the decision instead: which branch was taken, which tool was refused, which commitment was kept. Record one run against a live model, save the memory, and replay it as a fixture from then on. The suite runs in milliseconds, offline, with no API key.
This is why memory is a log. A log can be replayed, and a replay can be tested. Anything less means trusting the model to behave the same way tomorrow.
Program A · Legibility as a design requirement
We accept opacity inside the model because we have no alternative yet. We refuse opacity in the system around it because there the alternative is simply discipline. Every transition should be named. Every tool call should leave a receipt. Every decision to act, wait, or escalate should be a line in a log that a person can read the next morning.
This is not a preference for tidy code. It is the only path we know to accountability: the property that lets an engineer stand behind a behavior, fix it when it is wrong, and prove the fix held. Legibility is what makes iteration cheap. Opacity is what makes every incident a research project.
The full argumentProgram A · Research · The Foundation Agent
Foundation models gave every engineer the same raw capability. What differentiates a system now is its form: how it decides when to act and when to rest, what it remembers, what it refuses, and how it explains itself. The Foundation Agent program treats that form as a research object: an immutable loop template (triage, plan, execute, reflect, rest) iterated across generations, with every version diffable, replayable, and comparable.
Each generation is evaluated by simulation: a dozen or more LLM-driven personas run long multi-turn conversations against the agent, and every failure is traced back to the exact event in memory that caused it. The point is not a leaderboard. The point is a design doctrine that accumulates, because each rule is attached to the failure that earned it.
In a 214-turn simulation battery against the previous generation, the current Foundation Agent cut major behavioral failures from 16.0% to 11.7% of turns and p90 turn latency from 17.5 s to 10.4 s, with zero silent turns. We report it here because it is the kind of number the program exists to produce. We label it as internal because it is our battery, our taxonomy, and our judgment of what counts as a failure. A write-up of the method is on the research page.
Read about the evaluation methodProgram A · Memory, restraint, and receipts
Any competitor can rent the same model you rent. What they cannot rent is the record of your system behaving well over time. That record is built from small, unglamorous properties: memory that survives the session and shapes the next one, restraint that lets the system choose to do nothing when nothing is the right move, and receipts for every action so the system's word can be checked against its deeds.
We think of memory as the behavioral substrate, not a cache. An agent without durable memory is a very articulate stranger. An agent that cannot rest will always be doing something, and most of it will be wrong. An agent without receipts is asking for trust it has not earned. These are architecture decisions, and they are the ones we spend our time on.
Memory as substrateProgram B · Statistical learning · CGEM
Collaborative Generalized Effects Modeling writes the model as a formula a domain expert can read, then lets each term be anything from a constant to a gradient-boosted learner. The terms are fit collaboratively until the whole converges. What you get is a structured model with machine-learned parts, not a black box with a feature-importance chart taped on.
from cgem import CGEM # The structure is stated, not discovered. formula = "UNITS = STORE_EFF * PRICE_EFF * SEASON_EFF" terms = { "STORE_EFF": {"model": "CatRegModel()", "xvars": ["STORE"], "ival": 1}, "PRICE_EFF": {"model": "OLS()", "xvars": ["PRICE", "PROMO"], "ival": 1}, "SEASON_EFF": {"model": "GAMTerm()", "xvars": ["WEEK"], "ival": 1}, } model = CGEM() model.load_df(train) model.define_form(formula) model.define_terms(terms) model.fit(25) # iterate to coherence preds = model.predict(test) print(model.calc_r2(test["UNITS"], preds))
A gradient-boosted model of demand will score well and tell you nothing you can act on. A CGEM model of the same demand says: this store lifts baseline by 1.3x, this price curve bends here, this season effect peaks in week 47. Each effect is a term you can inspect, constrain, freeze, or replace with a smarter learner without disturbing the rest.
Additive, multiplicative, and nested relationships, stated in one line.
A term can be a constant, a regression, a GAM, or a boosted learner.
Terms are refit collaboratively so no single effect swallows the others.
The body of work
A lab that only publishes positions has no way to be corrected. Every idea here is attached to a system someone can install, call, or measure. Some are libraries. One is a running product. All of them are the argument, in executable form.
Fourteen primitives for building agents in ordinary Python. Memory as the whole state, record and replay for tests, seven providers, an MCP client, zero dependencies.
Collaborative Generalized Effects Modeling. Interpretable structured models whose terms can be statistical or machine-learned, fit iteratively to coherence.
Compact 512-dimension embeddings with a five-year endpoint commitment. A production test of the lab's ideas about latency, cost, and infrastructure you can depend on.
An immutable agent-loop template, iterated through 27 generations with an append-only doctrine, a closed activity registry, and simulation-based behavioral evaluation.
Also on PyPI under the same license: thoughtbase (deploy ThoughtFlow agents as serverless APIs), taskatlas (agent-readable project state), membase (a cloud filesystem for agents), and apimagic. Roughly 42k downloads across the six.
Working with the lab
Partner work funds part of the research and keeps it grounded in what practitioners actually need. The projects that fit best are the ones where no standard answer exists yet: a new market, a new kind of product, or a system that has to be explainable to the people accountable for it.
When there is no playbook to copy, you need people who work from first principles. That is what we do.
Most tools and most advice are built around the average case, and for most problems the average case is exactly right. But when you are entering a market that does not exist yet, or shipping a product whose promise depends on an intelligent system behaving in a way nothing off the shelf does, the average answer is the wrong one. Those are the problems we want to work on.
In practice that means designing an agent system or a retrieval pipeline from the ground up, building an interpretable model where a black box would not be defensible, or standing up a lean production path on serverless infrastructure. Every project starts with a short architecture sprint so both sides know what we are building before anyone commits to more.
What you get is a working system, a written rationale for each decision, and a test suite that proves the behavior. Then we hand it over completely, documented well enough that your team owns it without us.
When there is no incumbent architecture to copy, a principles-first design is faster than a borrowed one, because nothing has to be unlearned later.
Products whose core promise depends on an intelligent system behaving in a way no off-the-shelf agent does.
Regulated, high-stakes, or high-visibility deployments where "the model did it" is not an acceptable answer.
Track record, in brief
Agent platforms for defense and financial services on serverless infrastructure; a production vector-search system at a $5B super-app credited with more than 500k orders a month; recommendation and data platforms at venture-backed startups; and the lab's own products, running in production. Details on request.
Positions
A three-line agent that hides a ten-thousand-line runtime has moved complexity, not removed it. Real simplicity is when the whole system fits in an engineer's head.
Everything that crosses a boundary in an intelligent system should be printable, diffable, and serializable exactly as it appears in code. Objects with hidden state are where bugs go to live.
A system that must always produce output will produce bad output. Choosing to wait, and being able to say why, is part of judgment.
Interpretability bolted onto a black box explains the explainer. Structure the model so its reasons are its parameters.
The written rationale is what lets the next engineer, human or otherwise, change the system without breaking its intent. We write more English than code.
Every position here is attached to code someone can run. A position that cannot be tested is a mood.
Who runs the lab
James Rolfsen has shipped machine learning in production since 2012 and led data and ML teams since 2015, most recently as Global Head of Data at Rappi, a $5B super-app across nine countries, where his team's vector-search system was credited with more than 500,000 additional orders a month. Before that: recommendation and data platforms at venture-backed startups, one of which he co-founded, and agent systems for defense and financial-services partners.
He is the author of ThoughtFlow and CGEM, the builder of Similar.dev, and the principal of the Foundation Agent program. His first two peer-reviewed papers (2014, 2015) simulated how social networks and international regimes evolve, a decade before agents became the industry. He writes at rolfsen.ai and works from Austin, Texas.
Start a conversation
Every submission lands as a structured note in James's inbox, and he replies personally. Partnership ideas, research questions, contributions to the tools, and plain hellos are all welcome. If you prefer email, info@think.dev reaches the same place.
Bring a problem without a playbook. We scope a first step together in one conversation, usually a short architecture sprint.
pip install thoughtflow or pip install cgem. Both are MIT. Issues and pull requests on GitHub are read and answered by the author.
Read the philosophy. If you think a position is wrong, say so; the ones that survive contact are the ones we keep.
info@think.dev. No ticketing system, no autoresponder. Austin, Texas, Central Time.