llm-evaluation-guide

本条目面向文献与学术工具场景,提供“llm-evaluation-guide”技能的目录入口。公开标签:learning、大模型、评估指标、基准测试。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。

仓库
brycewang-stanford/Auto-Empirical-Research-Skills
仓库最近推送
Stars
5,023
Forks
578
Issues / PRs
0

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "llm-evaluation-guide":https://skillpi.com/zh/skills/deepskill-731c640833c76bf234c6dd00e634919588493143a0c0ca5ce848dcfcfb3cf076。安装前核对上游来源和版本:https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/domains/ai-ml/llm-evaluation-guide/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/tree/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/domains/ai-ml/llm-evaluation-guide --skill llm-evaluation-guide
许可协议: —

来源与配置

查看 GitHub 源码

brycewang-stanford/Auto-Empirical-Research-Skills固定提交 bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目面向文献与学术工具场景,提供“llm-evaluation-guide”技能的目录入口。公开标签:learning、大模型、评估指标、基准测试。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: llm-evaluation-guide
description: "Evaluate and benchmark large language models for research applications"
metadata:
  openclaw:
    emoji: "🧠"
    category: "domains"
    subcategory: "ai-ml"
    keywords: ["LLM evaluation", "benchmarking", "language models", "model evaluation", "NLP metrics", "BLEU", "perplexity"]
    source: "wentor-research-plugins"
---

# LLM Evaluation Guide

A skill for evaluating and benchmarking large language models (LLMs) in research settings. Covers automatic metrics, human evaluation protocols, benchmark suites, evaluation pitfalls, and best practices for reporting LLM performance.

## Evaluation Taxonomy

### Types of Evaluation

```
1. Intrinsic evaluation:
   Measures model quality on its own terms
   - Perplexity, likelihood, calibration
   - Useful for comparing architectures and training procedures

2. Extrinsic evaluation:
   Measures model quality on downstream tasks
   - Task-specific benchmarks (QA, summarization, classification)
   - Closer to real-world usefulness

3. Human evaluation:
   Human judges rate model outputs
   - Fluency, correctness, helpfulness, safety
   - Gold standard but expensive and slow
```

## Automatic Metrics

### Common Metrics by Task

| Task | Metric | Description |
|------|--------|-------------|
| Language modeling | Perplexity | Lower is better; measures prediction quality |
| Machine translation | BLEU, COMET | N-gram overlap; learned quality estimation |
| Summarization | ROUGE-1/2/L | Recall of n-grams against reference |
| Question answering | Exact Match, F1 | Token-level match against reference answer |
| Classification | Accuracy, F1 | Standard classification metrics |
| Generation quality | BERTScore | Semantic similarity via embeddings |
| Factuality | FActScore | Proportion of atomic facts supported by evidence |

### Computing Key Metrics

```python
from collections import Counter
import math


def compute_bleu(reference: list[str], hypothesis: list[str],
                 max_n: int = 4) -> float:
    """
    Compute corpus-level BLEU score (simplified).

    Args:
        reference: List of reference token sequences
        hypothesis: List of hypothesis token sequences
        max_n: Maximum n-gram order
    """
    precisions = []

    for n in range(1, max_n + 1):
        num = 0
        den = 0
        for ref_tokens, hyp_tokens in zip(reference, hypothesis):
            ref_ngrams = Counter(
                tuple(ref_tokens[i:i+n]) for i in range(len(ref_tokens) - n + 1)
            )
            hyp_ngrams = Counter(
                tuple(hyp_tokens[i:i+n]) for i in range(len(hyp_tokens) - n + 1)
            )
            clipped = {ng: min(c, ref_ngrams.get(ng, 0))
                       for ng, c in hyp_ngrams.items()}
            num += sum(clipped.values())
            den += max(sum(hyp_ngrams.values()), 1)

        precisions.append(num / max(den, 1))

    # Brevity penalty
    ref_len = sum(len(r) for r in reference)
    hyp_len = sum(len(h) for h in hypothesis)
    bp = math.exp(1 - ref_len / max(hyp_len, 1)) if hyp_len < ref_len else 1.0

    # Geometric mean of precisions
    log_avg = sum(math.log(max(p, 1e-10)) for p in precisions) / max_n
    return bp * math.exp(log_avg)
```

## Benchmark Suites

### Major LLM Benchmarks

```
General knowledge and reasoning:
  - MMLU (Massive Multitask Language Understanding): 57 subjects, MCQ
  - HellaSwag: Commonsense sentence completion
  - ARC (AI2 Reasoning Challenge): Science questions
  - WinoGrande: Coreference resolution / commonsense

Coding:
  - HumanEval: Python function completion (pass@k)
  - MBPP: Mostly basic Python problems
  - SWE-bench: Real-world software engineering tasks

Math:
  - GSM8K: Grade school math word problems
  - MATH: Competition-level mathematics

Safety and alignment:
  - TruthfulQA: Resistance to common misconceptions
  - BBQ (Bias Benchmark for QA): Social bias in QA
  - RealToxicityPrompts: Tendency to generate toxic text

Instruction following:
  - MT-Bench: Multi-turn conversation quality (LLM-as-judge)
  - AlpacaEval: Instruction-following quality
  - Chatbot Arena: ELO-based human preference ranking
```

## Human Evaluation

### Designing a Human Evaluation Protocol

```python
def design_human_eval(task: str, n_annotators: int = 3,
                      n_examples: int = 200) -> dict:
    """
    Design a human evaluation protocol for LLM outputs.

    Args:
        task: The task being evaluated
        n_annotators: Number of independent annotators per example
        n_examples: Number of examples to evaluate
    """
    return {
        "task": task,
        "n_annotators": n_annotators,
        "n_examples": n_examples,
        "criteria": [
            {"name": "Fluency", "scale": "1-5",
             "description": "Is the text grammatically correct and natural?"},
            {"name": "Relevance", "scale": "1-5",
             "description": "Does the output address the input/question?"},
            {"name": "Correctness", "scale": "1-5",
             "description": "Is the factual content accurate?"},
            {"name": "Helpfulness", "scale": "1-5",
             "description": "Would a user find this response useful?"}
        ],
        "agreement_metric": "Krippendorff's alpha (ordinal)",
        "presentation": "Randomize model order; blind annotators to model identity",
        "calibration": "Have all annotators rate 20 shared examples first",
        "cost_estimate": f"~{n_examples * n_annotators * 0.50:.0f} USD at typical rates"
    }
```

## Evaluation Pitfalls

### Common Mistakes

```
1. Data contamination:
   Test data may appear in the LLM's training set.
   Mitigation: Use held-out datasets, check for contamination,
   create new test sets.

2. Metric gaming:
   High BLEU does not mean high quality; ROUGE rewards verbosity.
   Mitigation: Use multiple metrics and human evaluation.

3. Cherry-picking examples:
   Showing only best-case outputs misrepresents model capabilities.
   Mitigation: Report aggregate metrics over full test sets.

4. Ignoring variance:
   LLM outputs vary with temperature and random seeds.
   Mitigation: Report mean and standard deviation over multiple runs.

5. Unfair comparisons:
   Comparing models with different prompt formats or few-shot counts.
   Mitigation: Standardize prompts and report all hyperparameters.
```

## Reporting Standards

When publishing LLM evaluation results, report: model name and version, parameter count and architecture, evaluation dataset with version number, exact prompts used (include in appendix), number of few-shot examples, decoding parameters (temperature, top-p, max tokens), multiple metrics (not just one), confidence intervals or significance tests, and hardware and inference cost where relevant.

版本历史

  1. v2026.07 GitHub ↗
    原版更新说明

    ## Highlights - **First tagged release.** Every gate is green: `make check` (validate + 183 unit tests + eval-harness + numeric benchmarks), validate-catalog, quality-evals, and OpenSSF Scorecard on `main`. - **Rigor double loop at 13 method families.** Each family closes taxonomy tag → eval scenario (does the agent avoid the method's known traps?) → deterministic numeric benchmark (does it recover the true effect?). New this cycle: heterogeneous treatment effects (CATE) and quantile treatment effects (QTE). - **Paper-WorkFlow flagship at 29/29 executable rigor gates**, now merged to its `main` with a drift-gated README badge — the credibility artifact is checked by CI, so it can never silently go stale ([RIGOR.md](https://github.com/brycewang-stanford/Paper-WorkFlow/blob/main/RIGOR.md)). - **1,150 vendored skills / 69 collections**, all license-audited, security-scanned, tracked in a machine-readable catalog with CI-enforced freshness, and importable as a single whole-repo router skill. - **Hardened weekly upstream syncs**: synced trees must pass the full gate before merge; this cycle both sync PRs were verified, patched (StatsPAI's SkillOpt execution-gate card restored), and merged. - **First-party plugin marketplace** (Python / R / Stata empirical-analysis plugins + guide) shipped in the previous cycle, now under a tagged version. ## Known Follow-Ups - Ecosystem benchmark comparison (AERS vs econometrics agents on the shared benchmark suite) — [PLAN-2026-07 week 3](https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/main/docs/PLAN-2026-07.md). - Decomposition (Oaxaca–Blinder) and shift-share (Bartik) method families (weeks 3–4). - GitHub Pages catalog site, README demo screenshots, five-language README consistency script (week 4). ## Catalog - Top-level collections: **69** - `SKILL.md` files: **1150** - Cataloged agent tools: **335** ([`tools/CATALOG.md`](../tools/CATALOG.md)) ## Methodological rigor - Method families in the coverage map: **13** ([`RIGOR_COVERAGE.md`](RIGOR_COVERAGE.md)) - Eval scenarios: **26** (138 rubric items, 122 auto-checkable) ([`../eval-harness/`](../eval-harness/README.md)) - Benchmark tasks: **13** (70 gold checks, 58 required) ([`../benchmark/`](../benchmark/README.md)) ### Method families (skills tagged) - `iv`: 27 - `rdd`: 25 - `panel-fe`: 19 - `staggered-did`: 17 - `bayesian`: 13 - `did`: 10 - `event-study`: 10 - `matching`: 10 - `synthetic-control`: 10 - `cate`: 8 - `dml`: 8 - `survival`: 3 - `quantile`: 2 ### Eval scenarios by category - `causal-identification`: 14 - `citation-hygiene`: 1 - `reproducibility`: 2 - `research-integrity`: 1 - `runtime-safety`: 1 - `statistical-validity`: 2 - `writing-compliance`: 3 - `writing-style`: 2 ## License buckets - AGPL-3.0: 1 - Apache-2.0: 1 - CC-BY-NC-4.0 (non-commercial): 1 - CC-BY-SA-4.0: 1 - CC-BY-SA-4.0 (repository default): 4 - GPL-3.0: 1 - MIT: 35 - MIT Non-Commercial: 1 - Mixed (per-skill; see folder): 2 - UNKNOWN - check upstream: 22 ## Quality gates - `make check` — catalog/provenance/audit/eval freshness, repo hygiene, Paper-WorkFlow gates, stdlib unit tests, eval-harness lint + graded smoke, and the strict numeric benchmark (reference-gold drift fails the build). - External links and OpenSSF Scorecard run on their scheduled workflows.

当前收录版本

0.0.1

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值85/100
评分依据

提供LLM评估与基准测试的完整方法论体系

触发精准95/100
评分依据

触发词精准完整,覆盖LLM评估相关多种场景

结构规范90/100
评分依据

文档结构清晰,章节划分合理

执行可靠88/100
评分依据

提供的Python代码示例逻辑完整,参数明确

证据可验 (前四项均值)90/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。