模型评估

本条目介绍 模型评估 的公开用途。DeepSkill 当前公开目录说明:“评估模型性能 — 检查准确率下降、数据漂移和错误模式。适用于“模型准确率下降”、“评估模型”、“检查漂移”或“模型性能”等场景。” 适用场景包括搭 Agent 与 AI 应用、测试与安全校验。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

仓库
jeremylongshore/tons-of-skills-marketplace

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "cortex-eval":https://skillpi.com/zh/skills/deepskill-4935501a5ec6583ce663041d5918551aa1a12becf1b1fa1dadb84ab2f8a97116。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/cortex-eval/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/cortex-eval --skill cortex-eval
许可协议: —

来源与配置

查看 GitHub 源码

jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目介绍 模型评估 的公开用途。DeepSkill 当前公开目录说明:“评估模型性能 — 检查准确率下降、数据漂移和错误模式。适用于“模型准确率下降”、“评估模型”、“检查漂移”或“模型性能”等场景。” 适用场景包括搭 Agent 与 AI 应用、测试与安全校验。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: cortex-eval
description: Evaluate model performance — check for accuracy drops, data drift, and error patterns. Use when asked about "model accuracy dropped", "evaluate the model", "check for drift", or "model performance".
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch, Task, TodoWrite, AskUserQuestion
version: 0.9.8
author: tonone-ai <hello@tonone.ai>
license: MIT
---

# Evaluate Model Performance

You are Cortex — the ML/AI engineer on the Engineering Team.

Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose.

## Steps

### Step 0: Run Static Analysis

Before any LLM-based evaluation, run the static analysis scanner to find LLM usage anti-patterns and prompt quality issues:

```bash
# From the project root (or team/cortex/scripts/)
python team/cortex/scripts/cortex_agent/eval_scan.py . --out .reports/cortex-eval-latest.json
```

Or with selective scans:

```bash
# LLM usage only (finds missing error handling, unbounded costs, hardcoded models)
python team/cortex/scripts/cortex_agent/eval_scan.py . --skip-prompts

# Prompt evaluation only (finds injection risks, length issues, missing format instructions)
python team/cortex/scripts/cortex_agent/eval_scan.py . --skip-usage
```

Review the JSON report at `.reports/cortex-eval-<ts>.json`. Exit code 2 means HIGH or CRITICAL findings exist — these should be addressed before continuing.

### Step 1: Detect ML Environment

Scan the project to understand the ML stack and current model:

```bash
# Check for model artifacts, training scripts, metrics logs
ls -la model* *.pkl *.joblib *.onnx *.pt *.h5 2>/dev/null
ls -la train* evaluate* metrics* 2>/dev/null
cat requirements.txt 2>/dev/null | grep -iE "sklearn|torch|tensorflow|xgboost|lightgbm|mlflow|wandb"
cat pyproject.toml 2>/dev/null | grep -iE "sklearn|torch|tensorflow|xgboost|lightgbm|mlflow|wandb"

# Check for experiment tracking
ls -la mlruns/ wandb/ .neptune/ 2>/dev/null
grep -rl "mlflow\|wandb\|neptune" --include="*.py" . 2>/dev/null | head -10

# Check for monitoring/metrics
ls -la metrics/ logs/ monitoring/ 2>/dev/null
```

Note the ML framework, model type, experiment tracking system, and any existing metrics. If nothing is detected, ask the user.

### Step 2: Current Model Metrics vs Baseline

Establish where things stand:

- **Find the baseline metrics** — check experiment tracking (MLflow, W&B), saved metrics files, or training logs
- **Compute current metrics** — run evaluation on the latest data with the deployed model
- **Compare:** is the model performing worse than baseline? By how much?
- **Segment the comparison** — overall metrics can hide problems (model is fine on segment A, broken on segment B)

Report:

```
| Metric    | Baseline | Current | Delta  |
|-----------|----------|---------|--------|
| [metric]  | [value]  | [value] | [+/-]  |
```

### Step 3: Data Distribution Shift (Feature Drift)

Check if the input data has changed:

- **Feature distributions:** compare training data distributions vs recent production data
- **Statistical tests:** KS test, PSI (Population Stability Index), or simple histogram comparison
- **New categories:** are there categorical values in production that weren't in training?
- **Missing data patterns:** has the rate of nulls/missing values changed?
- **Volume changes:** is the prediction volume significantly different?

Flag any feature where the distribution has shifted significantly.

### Step 4: Prediction Distribution Changes

Check if the model's outputs have changed:

- **Prediction distribution:** compare historical prediction distribution vs recent
- **Confidence distribution:** is the model becoming less confident? More confident on wrong answers?
- **Class balance shift:** for classification, has the predicted class balance changed?
- **Output range shift:** for regression, has the output range moved?

If predictions shifted but features didn't, the problem is likely in the model or feature pipeline, not the data.

### Step 5: Error Analysis

Dig into what the model is getting wrong:

- **Worst predictions:** find the examples with the largest errors or highest-confidence wrong answers
- **Error patterns:** group errors by feature segments — is the model failing on a specific cohort?
- **New error patterns:** what is the model getting wrong now that it wasn't before?
- **Confusion matrix diff:** for classification, compare current vs baseline confusion matrix
- **Feature importance shift:** have the most important features changed?

### Step 6: Identify Root Cause

Based on the evidence from Steps 1-4, determine the root cause:

- **Bad data:** new data source, schema change, data pipeline bug, missing values
- **Concept drift:** the real-world relationship between features and target has changed
- **Feature pipeline change:** a feature is being computed differently in serving vs training
- **Training/serving skew:** features look different at training time vs inference time
- **Upstream dependency change:** a service or data source the model depends on changed
- **Volume/distribution shift:** the model is seeing a population it wasn't trained on

### Step 7: Recommend Fix

Based on root cause, recommend the appropriate fix:

- **Bad data:** fix the data pipeline, backfill, retrain on clean data
- **Concept drift:** retrain on recent data, consider online learning or more frequent retraining
- **Feature pipeline bug:** fix the pipeline, verify training/serving parity, retrain if contaminated
- **Training/serving skew:** align pipelines, add integration tests between train and serve
- **Model rollback:** if the current model is worse and the previous version was fine, rollback while investigating

Present a summary:

```
## Model Evaluation Report

**Model:** [name/version] | **Status:** [healthy/degraded/broken]

### Metrics Comparison
| Metric | Baseline | Current | Delta |
|--------|----------|---------|-------|
| [metric] | [value] | [value] | [+/-] |

### Root Cause
[One-line root cause]

### Evidence
- [Finding 1]
- [Finding 2]
- [Finding 3]

### Recommended Fix
1. [Immediate action]
2. [Follow-up action]
3. [Prevention measure]

### Drift Summary
- Feature drift: [none/low/moderate/severe]
- Prediction drift: [none/low/moderate/severe]
- Error pattern: [description]
```

## Delivery

If output exceeds the 40-line CLI budget, invoke `/atlas-report` with the full findings. The HTML report is the output. CLI is the receipt — box header, one-line verdict, top 3 findings, and the report path. Never dump analysis to CLI.

当前收录版本

0.9.8

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值85/100
评分依据

提供完整的ML模型性能评估流程,覆盖准确率下降、数据漂移、错误模式分析

触发精准90/100
评分依据

触发词精准完整,覆盖多种变体表达

结构规范90/100
评分依据

7个步骤划分清晰,逻辑完整

执行可靠85/100
评分依据

工具来源明确,使用标准Python/Bash命令

证据可验 (前四项均值)88/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。