A self-improving system should reduce specific errors on new tasks, not merely produce convincing reflection. This guide uses a fictional museum audio-guide reservation assistant to connect persistent memory, self-correction, and evaluation. The assistant must retain progress, recognize a misunderstood date, verify storage, and compare changes in an isolated test environment. This is not a six-week experiment report, and it does not adopt unverified quotations, attributions, or download figures.
Define the system outside the model
Here, harness means the prompts, tools, state, retrieval, validation, and execution around a model. Improving these can change behavior without updating weights, but does not guarantee automatic intelligence gains. Self-improving and self-evolving have overlapping informal uses. Specify what changes, who supplies feedback, and how it is judged rather than relying on terminology.
The pilot requires “next Wednesday” to be interpreted against an explicit reference date and timezone, with ambiguity surfaced. After a submission timeout, inspect actual reservation state before resubmitting. Evaluate parsing, request outcome, confirmed storage, and user notification separately. A reminder to “check carefully” cannot substitute for time handling, state, and tool contracts.
We searched GitHub for reusable open-source approaches and consulted LangGraph's checkpoint and persistence documentation. It distinguishes thread state from cross-thread stored information; an in-memory saver does not survive a process restart. Assess maintenance, license, version, database, credentials, and operating cost before adopting a framework. This article specifies implementation and acceptance rather than installing or connecting services.
Scenario one: separate progress, facts, and lessons
Current reservation state needs a task identifier, confirmed date, equipment quantity, request identifier, and persistence result. Cross-session lessons need provenance, scope, time, and rule version. Conversation history, execution checkpoints, and durable preferences are different objects. Recovery needs exact progress; learning needs reviewable experience. Old preferences must not override a new explicit instruction.
Isolate projects and users through namespaces and authorization. Similarity search alone does not establish complete protection from contamination. Keep visitor names, contact details, and identity information out of global lessons, retaining redacted patterns and necessary evidence references. Define retention, deletion propagation, and backup recovery. Removing a vector entry also requires considering related copies and indexes.
Verify three layers in a new session: state exists, the relevant rule is recalled, and execution applies it. Restart the process and repeat so an object held only in RAM is not mistaken for durable memory. Recheck versions and serialization after model, device, or storage changes. Do not assume migration is lossless.
Memory writes can fail. Record completed and unresolved writes instead of replying “remembered” without persistence evidence. Use versions or transactions for concurrent updates, retaining conflicts rather than accepting last-write wins blindly. External model extraction may transmit content even when the database is local. Confirm actual data and destination instead of calling the entire workflow offline.
Scenario two: correct the task interpretation and the external state
When a user corrects a date, revisit the original request, timezone, reference date, and actions already taken. Distinguish a changed preference from an interpretation error. Replacing a number in the previous answer is insufficient. If submission has not occurred, update and check the draft. If a reservation exists, query it and handle any authorized change explicitly; a verbal correction cannot alter stored state.
Diagnose input parsing, tool parameters, networking, authentication, service rules, and presentation separately. Capture requests, responses, state, and evidence before selecting a minimal falsifiable cause. A timeout may mean no write or a lost response after a write. Query by request identifier before retrying. Bound retries and duration, use an appropriate idempotency policy, and stop persistent permission failures.
Limit automatic repair. Editing a local test parser is different from changing production reservation access. Security, accounts, payments, and publication retain their existing boundaries. Preserve a diff and rollback before repair, then run relevant regression checks. Report observations, hypotheses, actions, results, and open questions. An agent's assertion that it fixed itself is not acceptance evidence.
A bounded request is: “Reproduce the relative-date error on fictional reservations, retaining reference time and timezone. Propose the smallest parser change and test normal dates, week boundaries, ambiguity, and rescheduling. Do not operate real reservations.” Deliver reproduction, code or prompt changes, results, and recovery instructions together.
Scenario three: evaluate the whole system
Start with product questions: correct dates, no duplicate reservations, honest unknown states, and respected access boundaries. Include ordinary requests, ambiguity, timezone differences, timeouts, duplicate attempts, unavailable storage, and unrelated-project memory. Use fictional records and isolated resources so a benchmark cannot consume real equipment availability.
Fix model, prompt, tools, data versions, and settings while changing one main factor such as date verification. Record pass, fail, transport or grading error, executed count, cost, and time. A grader failure is not the same as an assistant assertion failure; zero executed cases cannot mean all passed. Use deterministic checks for state, call counts, and fields, with clear language rubrics and human sampling where needed.
A single high score does not establish stability. Retain held-out samples and use budgeted repetitions for variable behavior, including failures and neutral results. Check whether improvements add unnecessary confirmations, calls, or regressions elsewhere. An improvement loop needs iteration, time, and cost bounds, plus an exit that keeps the existing method when no candidate is better.
For memory, compare absent, relevant, and stale or unrelated records to reveal contamination. For recovery, deliberately restart and verify that side effects are not duplicated. Concurrent benchmarks can compete for resources; isolate their state and budgets. Report the tested environment and remaining gaps rather than compress every capability into one score.
Inspect compatibility and interaction between Skills
Memory, correction, and evaluation are not inherently conflict-free. They may write the same rules, alter prompts, trigger hooks, update state, or start model calls. Compare triggers, paths, dependencies, network behavior, versions, and authorization before combining them. Pilot individually, then test the combination. Hosts differ in discovery, activation, and execution; extracting an archive does not establish that functionality is active.
The current SkillPI self-improving search has no matching record, so this guide does not invent verified links for the source topic's memory and recursive-improvement names. For new candidates, check public detail, pinned source, license, and actual function. Identify storage, correction, or evaluation specifically. Ordinary browser testing is not a complete agent benchmark, and install popularity is not maturity evidence.
State a capability gap concretely before searching: missing date context, status query, persistent storage, or acceptance assertion? Reuse existing tools where they suffice instead of installing duplicate suites. A free package can still require model calls, databases, runtime resources, and maintenance. There is no blanket zero-cost promise here.
FAQ and release acceptance
Memory before evaluation? Define expected outcomes and a baseline first, then test the smallest memory approach. Can an agent repair itself? It can propose, apply, and verify authorized reversible changes within a defined scope; external side effects and access remain constrained. Is longer reflection better? Without evidence and independent checks, more reflection can simply preserve an incorrect explanation.
How is a better harness demonstrated? Compare correctness, side effects, recovery, cost, and latency on comparable tasks and new samples, retaining versions. How is this adapted to Cursor, Codex, or another host? Verify current discovery and execution behavior rather than assuming identical directories and hooks.
Before release, deliver state definitions, isolation policy, reproduction, repair diff, evaluation cases and results, restart evidence, and rollback. Label plans separately from executed checks. Framework capabilities cannot be presented as capabilities already verified in the reader's system.
Sources
This original example covers the bilingual source topics' persistent-memory, correction, and performance-evaluation scenarios. Persistence boundaries refer to the LangGraph checkpoint explanation and official persistence documentation source. Refresh Skill associations by verified purpose and pinned provenance as the catalog grows, retesting interactions rather than assuming compatibility.