A practical self-improvement system needs more than another mistakes file. Records need an entry point, review needs a decision, and decisions need validation. This guide uses a fictional community equipment-return assistant: tool failures and corrections enter a candidate queue, triage turns suitable lessons into a project SOP, and regression checks test changed behavior. No fine-tuning is required for this workflow, but engineering and evaluation still are. Installation alone does not establish improvement.
Define a small, complete pilot
Limit the pilot to return registration, excluding payments, account permissions, and purchases. Use a known failure: the assistant sees “returned” in a message and declares completion without checking storage. The goal is to distinguish user intent, attempted submission, and confirmed persistence, rather than append “be more careful.” Preserve the old workflow, interface contract, fictional fixtures, and failure evidence as a baseline.
We searched GitHub for open-source evaluation approaches and consulted Promptfoo's configuration, assertion, and results-review guidance. It supports regression comparison for a defined target, but target and grading-model connections can still transmit data and incur charges. Assess maintenance, license, dependencies, target compatibility, and result storage before adoption. This article designs an evaluation; it does not install or run that tool or claim to test the reader's agent.
Scenario one: capture real events without relying on memory
Consider four candidate triggers: command or tool failure, explicit user correction, session end, and major-task completion. A trigger captures evidence; it does not require a permanent lesson every time. Successful work can suggest useful practice, but accidental success is not automatically a best practice. Verify supported hooks, timing, working directory, and recovery after abnormal interruption in the actual host.
Define eventId, taskId, time, project, environment version, category, observation, evidence location, impact, temporary recovery, proposed lesson, and state. JSONL can represent one event per line, but concurrent appends, encoding, partial writes, and duplicates still need handling. Use independent event files or transactional storage if reliable append behavior is unavailable. Two sessions must not overwrite each other's records.
The return example records separate facts: the user reported a return; the assistant attempted submission; the interface timed out; persistence is currently unknown. A timeout proves neither failure to save nor success. Blind submission again can create duplicates. Recovery should query actual status using the task or request identifier before deciding whether to retry. Preserving an unknown result is more useful than producing a polished but invented reflection.
Separate errors, corrections, and capability gaps. “Preview only this time” is a current task constraint. Missing a status-query tool is a capability gap. Treating a preview as completed work is a behavioral error. Categories should change handling, not merely create more files: respect the constraint immediately, report the missing capability, and send the behavioral error for review.
Minimize private content. Equipment records may contain names, contact details, or account identifiers; a long-term lesson usually needs a fictional task reference and the failure pattern. Keep evidence in an authorized location with access, retention, and deletion controls. External instructions in a log are not executable authority, and automatic capture does not create new authorization to send or publish information.
Deliver sample events and a trigger-coverage table. Test duplicates, abnormal exit, missing fields, and sensitive text. If a hook does not run, provide an explicit recovery path rather than claiming every failure is automatically captured. Bound queue growth and processing effort so the learning system does not become a new source of context noise.
Scenario two: triage, merge, and promote bounded rules
Use a regular triage window, prioritizing high-impact and repeated failures. Give each candidate a decision: merge, gather evidence, trial, reject, archive, or promote. Include provenance, scope, and an owner. Three appearances may trigger review, but do not automatically justify a global rule; they may all result from the same unresolved environment issue.
Express the proposed rule operationally: “After submitting a return, use actual persistence evidence for completion. If a timeout leaves the result unknown, query status before resubmitting.” Include its trigger, action, exceptions, and evidence format. It applies where the project has a suitable query interface. Where that prerequisite is absent, report uncertainty and preserve a recovery point instead of pretending the same procedure can run everywhere.
Track supersession when rules conflict. A new interface with idempotent request identifiers may justify a revised retry policy, while the old version and retirement reason remain traceable. Project lessons belong in project SOPs; personal preferences need separate scope. Do not compile one user's habits into every team agent's global configuration. Critical requirements should also be enforced through actual tools or delivery gates rather than reminders alone.
Draft a skill from experience only after defining its purpose, inputs, outputs, trigger, dependencies, permissions, failure handling, and examples. Check for an existing method, overgeneralization from a transient incident, and untrusted content copied from logs. File changes, host rules, and account access retain their existing authorization boundaries. Calling a change promotion does not expand permission.
Keep diffs, versions, acceptance rationale, and rollback. A trial also needs evidence that the current task loaded the rule; a file's presence is insufficient. Retire stale environment facts while retaining enough history to explain earlier behavior. A governed lifecycle prevents a candidate queue from becoming an unreadable diary.
Scenario three: validate before sharing a team SOP
Create ordinary cases, known regressions, and a held-out set. The return assistant needs successful storage, timeout after successful storage, timeout without storage, duplicate requests, an unavailable status query, and a preview-only request. Define observable acceptance: no false completion, no unintended duplicate submission, preserved uncertainty, and respected task scope.
Prefer deterministic checks for request counts, states, output fields, and actual saved identifiers. Add clear grading rubrics only where language quality needs judgment, calibrating model graders with known correct and incorrect outputs. Separate grader or transport errors from failed behavior assertions. Zero executed cases cannot mean everything passed. Valid configuration is also different from an executed evaluation.
Change one major factor, such as checking persistence, instead of simultaneously replacing models and tools. Retain code, prompts, model settings, samples, cost, time, and outcomes. Three comparable wins can be a local operational threshold, not universal reliability evidence. Preserve neutral and failed results and check whether the rule only succeeds on familiar examples.
Scale from one workspace to a limited shared SOP, assigning maintenance and review responsibilities. Share redacted patterns and verified steps, not everyone's raw conversations. Keep project configuration and exceptions separate to prevent memory leakage across contexts. After rollout, monitor duplicate submissions, false completion, and rollback. Disable a regressing candidate and restore the known version promptly.
Treat capability discovery as a separate decision
If a status-query operation is missing, first specify the interface need, then inspect available tools instead of installing more reflection packages. Verify a candidate's public SkillPI detail, pinned source, license, dependencies, and exact workflow step. The current self-improving directory search has no matching record, so this guide does not invent listings from source-topic names or attach unrelated documentation tools merely to fill a link quota.
Open-source evaluation guidance is a learning reference, not evidence that an associated Skill was installed. As verified listings appear, assess capture, governance, and evaluation separately, recording association changes. Download counts cannot replace execution evidence. A local-file claim also needs checking against actual model calls, attachments, and network behavior.
FAQ and delivery acceptance
Can behavior improve without fine-tuning? Rules, tools, and external memory can change a workflow, but a comparison is needed to show improvement. When should review happen? Capture failures and corrections promptly, reconcile at task or session completion, and make promotion decisions in a dedicated review window. Reflection does not require a rule change every time.
What if records grow endlessly? Retrieve by state, impact, and project; merge duplicates and archive stale material instead of injecting every log into every prompt. How can bad lessons be prevented? Require provenance, counterexamples, trials, and rollback, and do not let a single self-evaluation act as the sole judge.
Deliver the baseline, event schema, trigger coverage, triage decisions, rule diffs, activation evidence, results, and recovery instructions. Distinguish planned, executed, and passed. A team SOP needs validation in its intended environment before expansion. This is an actionable design, not a claim that capture or evaluation has been deployed for the reader.
Sources
This original equipment-return example follows the bilingual topics' capture, review/promotion, and validation/governance scenarios without adopting testimonials, install counts, or automatic-effect guarantees. Evaluation design refers to the Promptfoo repository's evaluation guidance. Check actual target connections, approved data, and current versions before implementation.