Choose a Skill by whether it improves a real task, not whether it completes a list of essential packages. This guide uses a fictional community plant-guide project: approved observation records must become traceable entries, unknown species must remain unknown, and image provenance must be checked. Pre-install verification, capability discovery, and post-install comparison answer different questions. This is an evaluation method, not the source topic's claimed user field test.
Turn selection into an acceptance question
Define a small task: ten fictional observations and a source manifest become consistent fields, missing-information notes, and citations without guessing uncertain species, overwriting originals, or publishing images. Specify correctness, coverage, activation, action boundaries, time, and cost. Ten is an illustrative fixture size, not a sufficient statistical guarantee.
Keep a baseline without the candidate. If the existing agent already completes the work, a Skill may add format consistency or fewer omissions rather than new capability. Choosing not to install is valid. There is no universal best number: neither a small fixed count nor a large collection establishes better results without task evidence.
We searched GitHub for mature evaluation approaches and consulted Promptfoo's cases, assertions, and grader-calibration workflow. Assess maintenance, license, dependencies, target connections, and output storage before adoption. Valid configuration is not an executed evaluation, and model-based grading still calls actual providers. No evaluation tool or candidate was installed or run for this article.
Scenario one: verify structure, provenance, and runtime boundaries
Check the public listing, source repository, pinned revision, license, and maintenance evidence. A name and description state intended purpose, not proven behavior. Read the entry document and relevant scripts, resources, and dependencies, identifying file access, network destinations, credentials, writes, cost, and failures. Record static review separately from runtime testing. Reading instructions is not a security scan.
Structural checks cover metadata, triggers, inputs, outputs, prerequisites, resolvable references, and failure reporting. Run behavioral checks only in an isolated approved environment with fictional data, bounded directories, and a budget. Real photos, membership lists, and production services are not default fixtures. Dependencies and external code follow host authorization requirements; valid packaging does not grant execution permission.
For the plant-guide candidate, include ordinary records, missing fields, duplicate sources, unknown species, and inaccessible images. Check uncertainty, invented provenance, silent omissions, and original-file integrity. An attractive output is insufficient: reconcile entries and citations against each input with expected results, observations, and evidence.
A quick screen can consider task fit, correctness, clear prerequisites, action boundaries, and maintenance cost. Do not let an average conceal a hard failure. Private-asset disclosure or unconditional destructive writes cannot be offset by a high layout score. Define rejection criteria before seeing outputs so enthusiasm for a tool cannot rewrite the test.
Scenario two: discover the capability gap before searching
Replace “install useful skills” with a precise gap: inconsistent field extraction, unverifiable image references, or a host unable to write files? Methods may improve the first two; the last needs actual host capability and authorization. Instructions cannot create a missing tool. Reuse existing functionality where it suffices.
Keep query, directory, date, candidates, exclusions, and unresolved evidence in a search record. Discovery expands a candidate set; selection compares against a task. Products may overlap in those roles, so different names do not establish complementarity. Report an unmatched gap instead of inventing a verified installation link from a similar name.
Compare environment, connected applications, supported language, image handling, outbound data, and cost. A portable Skill format does not make scripts, hooks, and dependencies identical across hosts. Check current discovery behavior, then observe activation on a minimal task. Separate missed and mistaken triggers. Mentioning a Skill name is not evidence that the host loaded and applied it.
The current SkillPI self-improving search has no matching record. Earlier prompt-related candidates also cannot become general quality evaluators merely because their descriptions mention prompts. This guide does not present skill-tester, skillscope, or other source-page names as verified on-platform listings. Refresh associations when purpose-matching public records, pinned sources, and licenses can be checked.
Scenario three: compare activation, execution, and outcomes
Evaluate three layers. Activation asks whether relevant tasks load the method and unrelated tasks avoid it. Execution checks correct inputs, bounded tool use, and honest failure handling. Outcomes check completeness, source traceability, and the actual deliverable. Style scoring alone neither proves activation nor reveals intermediate side effects.
Run baseline and candidate on the same tasks with fixed model, input, environment, and budget. Retain held-out samples and budgeted repetitions when outputs vary. Record passes, failures, transport or grader errors, executed coverage, cost, time, and human rework. Passing a subset is not full-suite success. A model update must not be credited to a Skill merely because tests ran at different times.
Prefer deterministic checks for fields, record counts, source identifiers, and file scope. For semantic criteria, give clear rubrics and separate source evidence from candidate output. Calibrate against known-good and deliberately wrong responses. If wrong answers pass, repair the grader before trusting scores. Grader failures remain separate; a mock judge verifies wiring, not real quality.
An unknown plant should stay unknown with requested evidence. An invented species fails even when the name is formatted correctly. An inaccessible image should be marked unverified rather than replaced with a fabricated link. Retain item-level evidence so another maintainer can review decisions without relying on a total-score chart.
Test self-improvement claims explicitly
Introduce one controlled error, supply a clear correction, and test a new task. Separate saving the lesson, retrieving it, applying it, and producing the correct result. Appending to a corrections file is insufficient. Include an unrelated project and a stale rule to reveal contamination or inappropriate persistence.
Compare absent, relevant, and incorrect memory, including retrieval effort and side effects. Reflection is not independent factual evidence. Framework behavior, feedback quality, and execution gates may all affect outcomes. Do not cite an unverified researcher attribution as proof that a particular package works, or claim a memory approach always produces faster gains than a framework.
Decide, retain, and maintain versions
Give each candidate an accepted, scoped-trial, evidence-needed, or rejected decision with supporting scope and results. Packages can conflict by editing rules or triggering work simultaneously; compare them individually before testing combinations. Preserve a pinned version and rollback. Rerun relevant regressions after model, dependency, or Skill changes.
Disable unused candidates through supported host mechanisms or narrow their scope, backing up configuration and locations first. Do not bulk-delete directories without authorization. Periodically review active purpose, provenance, and licensing. New SkillPI records justify association changes only after verification, not automatic replacement by popularity.
FAQ and the acceptance report
Are evaluation Skills themselves trustworthy? They also need calibration, provenance, and known counterexamples. Are all agents compatible? Portable format does not establish identical tools, dependencies, and permissions. Does easy uninstall make a wrong choice harmless? No: external changes and transmitted data are not automatically reversed by uninstalling.
How many should remain? Keep the smallest combination with demonstrated value for actual work; there is no universal count. Report task, pinned candidate, host, data, criteria, executed scope, failures, cost, decision, and open questions. This guide does not certify runtime behavior of any candidate.
Sources
This original plant-guide example reorganizes the bilingual topics' verification, discovery, and comparison scenarios without adopting catalog counts, install figures, testimonials, or unverified performance claims. Evaluation and grader boundaries refer to Promptfoo's official evaluation guidance. A reference is not installation authorization, and disabling result sharing does not stop configured target and grader calls from transmitting approved data.