What poteto's pstack teaches about AI engineering

Summary

Lauren Tan (@poteto) and her public pstack workflow offer a useful practitioner thesis: AI engineering is less about eliciting code from an LLM and more about designing the surrounding system—context, tools, constraints, evaluators, runtime access, and feedback loops—so a fallible model can produce evidence-backed outcomes.

The deepest idea is not “use more agents.” It is “make one agent reliable enough that parallelism becomes safe.” That reframes the engineering problem from generation to controlled execution.

Open the visual deep-research explainer →

Scope and source posture

This note separates three kinds of statements:

• Practitioner evidence: Lauren's public posts, podcast discussion, and the design of pstack. These reveal a working method, but they are not controlled experiments.
• Research-backed mechanisms: results and frameworks from ReAct, Reflexion, SWE-agent, Anthropic's agent engineering guidance, and OpenAI's internal data-agent report.
• Consulting recommendations: engineering judgments about how a team should adopt these ideas under constraints of correctness, maintainability, security, cost, and stakeholder trust.

The relevant social-media source is Lauren's article “How I Use Cursor”. The implementation-level source is the Cursor pstack repository, which encodes the approach as playbooks, principles, verification skills, and automation. The research sources are listed at the end and linked at the point where they support a claim.

The core mental model: an LLM is a component, not the engineer

A language model can synthesize plausible code, select tools, explain a hypothesis, and revise its plan. It does not automatically possess the repository's intent, the product's actual state, permission boundaries, a trustworthy oracle, or the organization's definition of “done.” Those must be supplied by the surrounding system.

AI engineering = model
               + task contract
               + curated context
               + agent-computer interface
               + executable constraints
               + evaluator / oracle
               + runtime evidence
               + human escalation
               + regression memory

This is why the distinction between a chatbot and an agent matters. Anthropic describes an agent as a model that plans, acts through tools, observes environmental feedback, adjusts, and repeats, with explicit stopping conditions and human checkpoints. Their practical recommendation is to start with simple composable workflows and add autonomy only when it improves measured outcomes. See Building Effective AI Agents.

What transfers from poteto / pstack

1. Verification is the first scaling primitive

Lauren's most transferable claim is that the agent should be able to use the product in the same meaningful way a user does: run the app, navigate to a feature, reproduce an issue, inspect traces or snapshots, and show the resulting evidence. Her later verification-skill posts describe a “feature map” that records user-facing routes, controls, shortcuts, and stable selectors so a screenshot-only bug report can become an executable investigation.

This is stronger than “ask the model to check its work.” It supplies an oracle and an observation channel. The difference is visible in the loop:

task → hypothesis → tool action → runtime observation → evaluation
                                      ↑                  │
                                      └── repair / stop ┘

The research analogy is direct. ReAct interleaves reasoning and action so the agent can gather information and correct course through the environment; SWE-agent finds that the interface given to the agent changes its ability to navigate repositories, edit files, and execute tests. Anthropic likewise emphasizes ground truth from tool results or code execution at each step.

Engineering consulting implication: before choosing an agent framework, identify the oracle. It may be a unit test, database invariant, screenshot comparison, browser journey, CPU trace, generated SQL result, human rubric, or user-confirmed decision. If no credible oracle exists, the first deliverable is not autonomy; it is observability and evaluation design.

2. Externalize tacit engineering judgment

pstack turns repeated failure modes into named playbooks and principles: binary-search a bug, prove behavior, sequence verifiable units, guard the context window, encode lessons in structure, and build a rerunnable lever. This is a form of procedural memory for a system whose model context resets, compacts, or changes across runs.

The important distinction is between advice and infrastructure:

Weak memory	Durable memory
“Remember to check authorization.”	A test, policy, lint rule, or CI gate that fails without authorization.
“Use the feature correctly.”	A feature map plus a runnable user journey.
“Do not repeat this bug.”	A regression case linked to the failure and its fix.
“Keep agents focused.”	Bounded task contracts, ownership fences, and a verification ledger.

The public repository's “build the lever” principle makes the same argument: a deterministic script or rerunnable check is easier to review and rerun than a hand-applied change. This connects to the consulting practice of making recommendations operational: a client should receive a mechanism that survives the original conversation.

3. Go deep before going broad

Lauren argues that parallel orchestration is most useful when it increases depth on one or a few problems: best-of-N attempts, adversarial review, or multiple reproduction attempts. Broadly assigning many agents to unrelated problems creates a human coordination bottleneck. The hidden resource is not just model tokens; it is the operator's attention and ability to remember what each agent changed.

This is a resource-allocation principle:

parallelism is safe when:
  independent work × clear ownership × cheap verification
  -------------------------------------------------------
  context switching + coordination + error correlation
  is favorable

The formula is a reasoning aid, not a benchmark. Measure it locally. A single strong agent with a clear tool surface may beat a committee of agents whose outputs require manual synthesis. Anthropic recommends adding multi-step complexity only when it demonstrably improves outcomes; the same discipline should apply to multi-agent fan-out.

4. Evaluate the workflow, not just the model

Lauren's podcast discussion describes treating skills like code: curate representative tasks, define a rubric, test across models, compare results, and preserve observed failure cases. The useful unit is not whether a prompt produces an impressive transcript. It is whether the complete workflow satisfies a task contract under a real harness.

OpenAI describes a comparable pattern in its internal data agent: curated question-and-answer cases, manually authored “golden” SQL, execution of generated SQL, comparison of result sets, grading with explanations, and continuous regression checks. The important detail is semantic evaluation: equivalent SQL should not fail merely because its text differs, while a plausible-looking query should fail if its result is wrong. See Inside OpenAI's in-house data agent.

For a team, a useful eval record includes:

1. task input and starting state;
2. expected behavior or acceptable answer shape;
3. permitted tools and permissions;
4. observable success and failure signals;
5. cost, latency, and number of tool calls;
6. whether a human had to intervene;
7. the model, prompt, skill, and tool versions;
8. links to the trace, artifact, or regression test.

5. Design the agent-computer interface

The prompt is only one part of the interface. Tool descriptions, parameter names, absolute paths, return formats, error messages, sandbox boundaries, and examples all shape behavior. SWE-agent calls this an agent-computer interface; Anthropic reports spending more effort optimizing tools than the overall prompt in its software-engineering agent, including changing file tools to require absolute paths after observing path errors.

This gives a practical review checklist:

• Does each tool do one clear thing?
• Are invalid states rejected at the boundary?
• Are names meaningful to a model and a human reader?
• Do errors say what failed and what can be tried next?
• Are dangerous operations permissioned, bounded, and reversible?
• Are tool outputs small enough to preserve context?
• Can the tool be tested independently from the model?

These are ordinary engineering-consulting habits—explicit contracts, guard clauses, defensive programming, least privilege, observability—applied to the model's interface.

6. Use constraints as executable architecture

Lauren's Dune example uses architectural rules and CI enforcement to narrow the space of agent mistakes: directory boundaries, import checks, lint rules, and selected banned patterns. The general lesson is sound: constraints that execute are more reliable than constraints that merely appear in prose.

The specific rules are not universal. Banning useEffect, comments, or another construct may be sensible in one codebase and harmful in another. A consulting recommendation should therefore phrase constraints as hypotheses:

“We observed failure mode X. We will add guardrail Y, measure false positives and regressions, and remove or refine it if the rule harms legitimate work.”

That preserves fail-fast behavior without turning a local taste into dogma.

Academic calibration: what the literature supports

Supported mechanisms

• Reasoning plus action: ReAct gives a formal research precedent for interleaving internal reasoning with actions and environmental observations.
• Feedback as memory: Reflexion shows that language-based feedback stored between attempts can improve agent behavior without updating model weights.
• Interface effects: SWE-agent demonstrates that an agent-oriented computer interface materially affects software-engineering performance.
• Simple workflows plus evals: Anthropic recommends composable patterns, explicit evaluation, tool testing, ground truth, sandboxing, and stopping conditions.
• Task-level continuous evaluation: OpenAI's data-agent report illustrates how executable semantic checks can function as regression tests for an evolving agent.

Practitioner hypotheses, not settled laws

• “Depth first” is a strong default when context and verification are expensive, but its value is workload-dependent.
• A trust curve is a useful operating model, but trust must be measured through defect rates, intervention rates, cost, latency, and reversibility—not inferred from confidence or PR count.
• Multi-model review may provide useful diversity, but models can share blind spots and can increase cost or correlated error.
• More rigid architecture may help agents while reducing human flexibility. Rules need local validation.

Important limits

Verification proves only what the oracle observes. Passing tests can coexist with a wrong requirement, insecure authorization, poor maintainability, data leakage, or a missing failure mode. Human review remains necessary for system intent and trade-offs.

Long-horizon computer-use research also tempers the autonomy story. OSWorld 2.0 reports that frontier agents still struggle with hidden state, changing constraints, information arriving mid-task, and skipped verification. The research frontier is therefore reliable execution in changing environments, not simply longer context or more parallel agents.

A consulting playbook for adoption

Phase A — Frame the outcome

Write one sentence that a stranger could execute: “Given starting state S, produce artifact A, satisfying criteria C, within cost and permission limits P.” Clarify the stakeholder's actual goal: cycle time, quality, accessibility, operating cost, or learning.

Phase B — Establish a narrow oracle

Choose one representative task and define observable success. Include negative cases, not only the happy path. Decide what must be escalated to a human and what can be automatically retried.

Phase C — Harden the interface

Expose the smallest useful tool set. Add explicit types and schemas, boundary validation, least-privilege credentials, sanitized errors, structured logs, and idempotent operations. Treat async I/O, caching, and concurrency as production engineering concerns rather than agent magic.

Phase D — Turn failures into leverage

After the first failure, ask what durable artifact would prevent its recurrence: a better tool, a feature map, a test, a lint rule, a policy, a skill, or a clearer task contract. Keep the artifact small and rerunnable.

Phase E — Earn autonomy gradually

Use a staged trust ladder:

1. read-only investigation;
2. proposed change with human approval;
3. bounded write in a sandbox;
4. automatic merge behind tests and review gates;
5. production action only with explicit permissions, monitoring, rollback, and an audit trail.

At each stage, inspect correctness, security, cost, latency, intervention rate, and escaped defects. Do not promote a workflow because it is fluent, fast, or prolific.

What to learn if building general AI-engineering depth

1. LLM fundamentals: tokenization, context limits, sampling, structured outputs, tool calling, embeddings, retrieval, and model/version behavior.
2. Agent systems: state machines, workflows versus autonomous loops, planning, observation, retries, stopping conditions, and human-in-the-loop design.
3. Evaluation: datasets, golden answers, execution-based checks, graders, pairwise comparison, regression tracking, cost/latency measurement, and eval contamination or awareness.
4. Agent-computer interfaces: tool contracts, context shaping, permissions, sandboxing, error design, trace capture, and UI/browser/device control.
5. Classical engineering: types, modular boundaries, idempotency, tests at the right level, observability, secure defaults, graceful degradation, and operational runbooks.
6. Product and consulting judgment: identify the real decision, quantify failure cost, make uncertainty visible, involve stakeholders, and ship the smallest mechanism that creates durable value.

Bottom line

The durable lesson from Lauren Tan is not that everyone should copy pstack or run hundreds of agents. It is that model capability compounds only when the engineering environment compounds with it.

pstack is a compelling public encoding of that idea: failures become skills, skills are evaluated, tools are shaped around model behavior, verification becomes infrastructure, and autonomy is earned through evidence. Its most valuable contribution is a change in what counts as AI engineering—from prompt cleverness to the design of reliable feedback systems.

Sources

• Lauren Tan, “How I Use Cursor”
• Cursor's pstack source tree
• Anthropic, “Building Effective AI Agents”
• Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering”
• Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning”
• Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models”
• OpenAI, “Inside OpenAI's in-house data agent”
• Yuan et al., “OSWorld 2.0”