statistical-software-qa
安装方式
先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。
用 AI 助手安装
复制提示词,让助手先核对固定版本和依赖。
查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "statistical-software-qa":https://skillpi.com/zh/skills/deepskill-78e041ec0822ce07d82293babe788dba857656dcda5192ba3a44d5c70f727be6。安装前核对上游来源和版本:https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/26-Data-Wise-scholar/skills/implementation/statistical-software-qa/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。
此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。
npx skills add https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/tree/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/26-Data-Wise-scholar/skills/implementation/statistical-software-qa --skill statistical-software-qa来源与配置
查看 GitHub 源码brycewang-stanford/Auto-Empirical-Research-Skills固定提交 bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba
配置方式待核实
请查看原始来源中的适用工具、依赖和权限要求。
安装实测待补充
本条目面向测试与安全校验场景,提供“statistical-software-qa”技能的目录入口。公开标签:learning、R包测试、统计验证、数值精度。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。
文件浏览
---
name: statistical-software-qa
description: Quality assurance and testing protocols for statistical software
---
# Statistical Software QA
**Quality assurance patterns and testing strategies for statistical R packages**
Use this skill when working on: R package testing, numerical accuracy validation, reference implementation comparison, edge case identification, statistical correctness verification, or software quality assurance for methodology packages.
---
## Testing Philosophy
### Statistical Software is Different
Unlike typical software where "correct output" is clear, statistical software must:
1. **Produce statistically correct results** - Not just bug-free
2. **Handle edge cases gracefully** - Small samples, boundary conditions
3. **Maintain numerical precision** - Floating-point considerations
4. **Match known results** - Verification against published examples
5. **Degrade gracefully** - Informative errors, not crashes
### Testing Pyramid for Statistical Packages
```
▲
/│\
/ │ \
/ │ \
/Manual│\ <- Human review of outputs
/ Tests │ \
/─────────────\
/ Integration \ <- Cross-function workflows
/ Tests \
/───────────────────\
/ Statistical Tests \ <- Correctness verification
/ \
/─────────────────────────\
/ Reference Tests \ <- Match known results
/ \
/───────────────────────────────\
/ Unit Tests \ <- Individual functions
/___________________________________\
```
---
## Unit Testing Patterns
### Pattern 1: Known Value Tests
Test against analytically derivable results:
```r
test_that("indirect effect equals a*b for simple mediation", {
# Create data where we know true values
set.seed(42)
n <- 10000
x <- rnorm(n)
m <- 0.5 * x + rnorm(n, sd = 0.1) # a = 0.5
y <- 0.3 * m + rnorm(n, sd = 0.1) # b = 0.3
result <- mediate(y ~ x + m, mediator = "m", data = data.frame(x, m, y))
# True indirect = 0.5 * 0.3 = 0.15
expect_equal(result$indirect, 0.15, tolerance = 0.02)
})
```
### Pattern 2: Boundary Condition Tests
```r
test_that("handles minimum sample size", {
# Minimum viable sample
small_data <- data.frame(
x = c(0, 0, 1, 1),
m = c(0, 1, 1, 2),
y = c(1, 2, 2, 3)
)
# Should work without error
expect_no_error(mediate(y ~ x + m, mediator = "m", data = small_data))
# Should warn about low power
expect_warning(
mediate(y ~ x + m, mediator = "m", data = small_data),
"sample size"
)
})
```
### Pattern 3: Equivalence Tests
```r
test_that("bootstrap CI contains delta method CI asymptotically", {
set.seed(123)
data <- simulate_mediation(n = 5000, a = 0.3, b = 0.4)
boot_result <- mediate(data, method = "bootstrap", R = 2000)
delta_result <- mediate(data, method = "delta")
# CIs should be similar for large n
expect_equal(boot_result$ci, delta_result$ci, tolerance = 0.05)
})
```
---
## Reference Implementation Testing
### Strategy: Compare Against Published Results
```r
test_that("matches Imai et al. (2010) JOBS II example",
# Load reference data from mediation package
data("jobs", package = "mediation")
# Our implementation
our_result <- our_mediate(
outcome = job_seek ~ treat + econ_hard + sex + age,
mediator = job_disc ~ treat + econ_hard + sex + age,
data = jobs
)
# Published results (from paper Table 2)
expected_acme <- 0.015
expected_acme_ci <- c(-0.004, 0.035)
expect_equal(our_result$acme, expected_acme, tolerance = 0.005)
expect_equal(our_result$acme_ci, expected_acme_ci, tolerance = 0.01)
})
```
### Cross-Package Validation
```r
test_that("matches lavaan for SEM-based mediation", {
data <- simulate_mediation(n = 1000)
# Our implementation
our_result <- our_mediate(data)
# lavaan implementation
library(lavaan)
model <- '
m ~ a*x
y ~ b*m + c*x
indirect := a*b
'
lavaan_fit <- sem(model, data = data)
lavaan_indirect <- parameterEstimates(lavaan_fit)[
parameterEstimates(lavaan_fit)$label == "indirect", "est"
]
expect_equal(our_result$indirect, lavaan_indirect, tolerance = 0.01)
})
```
---
## Statistical Correctness Tests
### Coverage Probability Tests
Verify confidence intervals achieve nominal coverage:
```r
test_that("95% CI achieves nominal coverage", {
set.seed(42)
n_sims <- 1000
true_indirect <- 0.15
coverage <- 0
for (i in 1:n_sims) {
data <- simulate_mediation(n = 200, a = 0.5, b = 0.3)
result <- mediate(data, conf.level = 0.95)
if (result$ci[1] <= true_indirect && true_indirect <= result$ci[2]) {
coverage <- coverage + 1
}
}
coverage_rate <- coverage / n_sims
# Coverage should be between 93% and 97% (accounting for MC error)
expect_gte(coverage_rate, 0.93)
expect_lte(coverage_rate, 0.97)
})
```
### Bias Tests
```r
test_that("estimator is approximately unbiased", {
set.seed(123)
n_sims <- 500
true_indirect <- 0.2
estimates <- numeric(n_sims)
for (i in 1:n_sims) {
data <- simulate_mediation(n = 500, a = 0.5, b = 0.4)
estimates[i] <- mediate(data)$indirect
}
# Mean should be close to true value
bias <- mean(estimates) - true_indirect
expect_lt(abs(bias), 0.02) # Less than 2% bias
})
```
### Type I Error Tests
```r
test_that("maintains nominal Type I error under null", {
set.seed(456)
n_sims <- 1000
rejections <- 0
for (i in 1:n_sims) {
# Null: no indirect effect (a = 0)
data <- simulate_mediation(n = 200, a = 0, b = 0.5)
result <- mediate(data, conf.level = 0.95)
# Reject if CI excludes 0
if (result$ci[1] > 0 || result$ci[2] < 0) {
rejections <- rejections + 1
}
}
type1_rate <- rejections / n_sims
# Should be close to 5%
expect_lt(type1_rate, 0.07) # Allow some MC error
})
```
---
## Numerical Precision Testing
### Stability Under Scaling
```r
test_that("results invariant to variable scaling", {
data <- simulate_mediation(n = 500)
# Original scale
result1 <- mediate(data)
# Scale variables by 1000
data_scaled <- data
data_scaled$y <- data$y * 1000
data_scaled$m <- data$m * 1000
result2 <- mediate(data_scaled)
# Standardized effects should match
expect_equal(
result1$indirect / (sd(data$y)),
result2$indirect / (sd(data_scaled$y)),
tolerance = 1e-10
)
})
```
### Extreme Values
```r
test_that("handles extreme correlations", {
set.seed(789)
# Nearly collinear
x <- rnorm(100)
m <- x + rnorm(100, sd = 0.001) # r ≈ 0.9999
y <- m + rnorm(100, sd = 0.1)
data <- data.frame(x, m, y)
# Should warn about collinearity
expect_warning(mediate(data), "collinear|singular")
})
test_that("handles near-zero variance", {
data <- data.frame(
x = c(rep(0, 99), 1), # Almost constant
m = rnorm(100),
y = rnorm(100)
)
# Should handle gracefully
expect_error(
mediate(data),
"variance|constant"
)
})
```
---
## Edge Case Identification
### Edge Case Checklist
- [ ] **Sample size**: n = 3, 4, 10, 30, 100, 1000, 10000
- [ ] **Effect sizes**: 0, very small (0.01), medium (0.3), large (0.8), 1.0
- [ ] **Correlations**: 0, near-zero, high (>0.9), perfect (1.0)
- [ ] **Missing data**: None, 1%, 10%, 50%
- [ ] **Outliers**: None, 1 extreme, 5%
- [ ] **Variable types**: Continuous, binary, categorical
- [ ] **Model fit**: Perfect, good, poor
### Systematic Edge Case Generation
```r
#' Generate Edge Case Test Suite
#'
#' @return List of edge case datasets
generate_edge_cases <- function() {
list(
# Minimal sample
minimal = data.frame(
x = c(0, 1, 0, 1),
m = c(0, 0.5, 0.5, 1),
y = c(0, 0.3, 0.3, 0.6)
),
# Zero effect
null_effect = {
set.seed(1)
n <- 100
data.frame(
x = rnorm(n),
m = rnorm(n), # No relationship with x
y = rnorm(n) # No relationship with m
)
},
# Perfect mediation
perfect = {
x <- c(0, 0, 1, 1, 2, 2)
m <- x # Perfect a path
y <- m # Perfect b path
data.frame(x, m, y)
},
# High collinearity
collinear = {
set.seed(2)
x <- rnorm(100)
m <- x + rnorm(100, sd = 0.01)
y <- m + rnorm(100)
data.frame(x, m, y)
},
# Outliers
with_outliers = {
set.seed(3)
n <- 100
x <- c(rnorm(n-2), 10, -10)
m <- 0.5 * x + c(rnorm(n-2), 20, -20)
y <- 0.3 * m + rnorm(n)
data.frame(x, m, y)
},
# Missing data
with_missing = {
set.seed(4)
n <- 100
x <- rnorm(n)
m <- 0.5 * x + rnorm(n)
y <- 0.3 * m + rnorm(n)
# Introduce 10% missing
m[sample(n, 10)] <- NA
y[sample(n, 10)] <- NA
data.frame(x, m, y)
}
)
}
# Run all edge cases
test_that("handles all edge cases", {
edge_cases <- generate_edge_cases()
for (name in names(edge_cases)) {
expect_no_error(
tryCatch(
mediate(edge_cases[[name]]),
warning = function(w) NULL # Warnings OK
),
info = paste("Edge case:", name)
)
}
})
```
---
## Performance Testing
### Benchmarking
```r
test_that("meets performance requirements", {
skip_on_cran() # Skip during CRAN checks
data <- simulate_mediation(n = 1000)
# Point estimate should be fast
time_point <- system.time({
mediate(data, method = "delta")
})["elapsed"]
expect_lt(time_point, 0.1) # < 100ms
# Bootstrap should be reasonable
time_boot <- system.time({
mediate(data, method = "bootstrap", R = 1000)
})["elapsed"]
expect_lt(time_boot, 10) # < 10s for 1000 bootstraps
})
```
### Memory Testing
```r
test_that("memory usage is reasonable", {
skip_on_cran()
# Large dataset
data <- simulate_mediation(n = 100000)
mem_before <- pryr::mem_used()
result <- mediate(data)
mem_after <- pryr::mem_used()
mem_increase <- as.numeric(mem_after - mem_before) / 1e6 # MB
expect_lt(mem_increase, 100) # Less than 100MB increase
})
```
---
## Test Organization
### Recommended Test Structure
```
tests/
├── testthat/
│ ├── test-mediate.R # Main function tests
│ ├── test-mediate-bootstrap.R # Bootstrap-specific
│ ├── test-mediate-delta.R # Delta method-specific
│ ├── test-numerical.R # Numerical precision
│ ├── test-edge-cases.R # Edge cases
│ ├── test-reference.R # Reference implementation
│ ├── test-coverage.R # Statistical coverage
│ └── helper-simulate.R # Test helpers
├── testthat.R
└── reference-results/ # Saved reference results
├── jobs-example.rds
└── known-values.rds
```
### Test Tagging
```r
test_that("coverage probability (slow)", {
skip_on_cran()
skip_if_not(Sys.getenv("RUN_SLOW_TESTS") == "true")
# ... slow coverage test ...
})
```
---
## Continuous Integration
### GitHub Actions Workflow
```yaml
# .github/workflows/R-CMD-check.yaml
name: R-CMD-check
on: [push, pull_request]
jobs:
R-CMD-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: r-lib/actions/setup-r@v2
- uses: r-lib/actions/setup-r-dependencies@v2
- uses: r-lib/actions/check-r-package@v2
coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: r-lib/actions/setup-r@v2
- uses: r-lib/actions/setup-r-dependencies@v2
- name: Test coverage
run: covr::codecov()
shell: Rscript {0}
slow-tests:
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- uses: r-lib/actions/setup-r@v2
- uses: r-lib/actions/setup-r-dependencies@v2
- name: Run slow tests
run: |
Sys.setenv(RUN_SLOW_TESTS = "true")
testthat::test_local()
shell: Rscript {0}
```
---
## QA Checklist
### Pre-Release Checklist
- [ ] All unit tests pass
- [ ] Reference tests match published results
- [ ] Coverage > 80%
- [ ] No R CMD check warnings
- [ ] Edge cases handled gracefully
- [ ] Performance benchmarks met
- [ ] Documentation examples run
- [ ] Vignette builds without error
- [ ] NEWS.md updated
- [ ] Version number incremented
### Statistical Validation Template
```markdown
## Statistical Validation Report
**Package**: [name] v[version]
**Date**: [date]
**Validator**: [name]
### Coverage Probability (95% CI)
| Scenario | N | True Effect | Observed Coverage | Pass |
|----------|---|-------------|-------------------|------|
| Small sample | 50 | 0.15 | 94.2% | Yes |
| Medium sample | 200 | 0.15 | 95.1% | Yes |
| Large sample | 1000 | 0.15 | 94.8% | Yes |
| Null effect | 200 | 0 | 95.3% | Yes |
### Bias Assessment
| Scenario | True | Mean Estimate | Bias | Pass |
|----------|------|---------------|------|------|
| a=0.5, b=0.3 | 0.15 | 0.151 | 0.001 | Yes |
| a=0.1, b=0.1 | 0.01 | 0.012 | 0.002 | Yes |
### Reference Comparison
| Reference | Our Result | Difference | Pass |
|-----------|------------|------------|------|
| Imai et al. Table 2 | Match | <0.001 | Yes |
| lavaan output | Match | <0.001 | Yes |
```
---
## References
### R Package Testing
- Wickham, H. (2011). testthat: Get started with testing. *The R Journal*
- Wickham, H., & Bryan, J. (2023). *R Packages* (Testing chapter)
### Statistical Software Validation
- FDA Guidance for Industry: Statistical Software Validation
- Altman, M., et al. (2004). Numerical Issues in Statistical Computing
### Quality Assurance
- McCullough, B. D. (1999). Assessing the reliability of statistical software
- Keeling, K. B., & Pavur, R. J. (2007). Statistical accuracy of spreadsheet software
---
**Version**: 1.0.0
**Created**: 2025-12-09
**Domain**: Quality assurance for statistical R packages
**Target**: MediationVerse ecosystem and similar methodology packages
版本历史
-
v2026.07
GitHub ↗
原版更新说明
## Highlights - **First tagged release.** Every gate is green: `make check` (validate + 183 unit tests + eval-harness + numeric benchmarks), validate-catalog, quality-evals, and OpenSSF Scorecard on `main`. - **Rigor double loop at 13 method families.** Each family closes taxonomy tag → eval scenario (does the agent avoid the method's known traps?) → deterministic numeric benchmark (does it recover the true effect?). New this cycle: heterogeneous treatment effects (CATE) and quantile treatment effects (QTE). - **Paper-WorkFlow flagship at 29/29 executable rigor gates**, now merged to its `main` with a drift-gated README badge — the credibility artifact is checked by CI, so it can never silently go stale ([RIGOR.md](https://github.com/brycewang-stanford/Paper-WorkFlow/blob/main/RIGOR.md)). - **1,150 vendored skills / 69 collections**, all license-audited, security-scanned, tracked in a machine-readable catalog with CI-enforced freshness, and importable as a single whole-repo router skill. - **Hardened weekly upstream syncs**: synced trees must pass the full gate before merge; this cycle both sync PRs were verified, patched (StatsPAI's SkillOpt execution-gate card restored), and merged. - **First-party plugin marketplace** (Python / R / Stata empirical-analysis plugins + guide) shipped in the previous cycle, now under a tagged version. ## Known Follow-Ups - Ecosystem benchmark comparison (AERS vs econometrics agents on the shared benchmark suite) — [PLAN-2026-07 week 3](https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/main/docs/PLAN-2026-07.md). - Decomposition (Oaxaca–Blinder) and shift-share (Bartik) method families (weeks 3–4). - GitHub Pages catalog site, README demo screenshots, five-language README consistency script (week 4). ## Catalog - Top-level collections: **69** - `SKILL.md` files: **1150** - Cataloged agent tools: **335** ([`tools/CATALOG.md`](../tools/CATALOG.md)) ## Methodological rigor - Method families in the coverage map: **13** ([`RIGOR_COVERAGE.md`](RIGOR_COVERAGE.md)) - Eval scenarios: **26** (138 rubric items, 122 auto-checkable) ([`../eval-harness/`](../eval-harness/README.md)) - Benchmark tasks: **13** (70 gold checks, 58 required) ([`../benchmark/`](../benchmark/README.md)) ### Method families (skills tagged) - `iv`: 27 - `rdd`: 25 - `panel-fe`: 19 - `staggered-did`: 17 - `bayesian`: 13 - `did`: 10 - `event-study`: 10 - `matching`: 10 - `synthetic-control`: 10 - `cate`: 8 - `dml`: 8 - `survival`: 3 - `quantile`: 2 ### Eval scenarios by category - `causal-identification`: 14 - `citation-hygiene`: 1 - `reproducibility`: 2 - `research-integrity`: 1 - `runtime-safety`: 1 - `statistical-validity`: 2 - `writing-compliance`: 3 - `writing-style`: 2 ## License buckets - AGPL-3.0: 1 - Apache-2.0: 1 - CC-BY-NC-4.0 (non-commercial): 1 - CC-BY-SA-4.0: 1 - CC-BY-SA-4.0 (repository default): 4 - GPL-3.0: 1 - MIT: 35 - MIT Non-Commercial: 1 - Mixed (per-skill; see folder): 2 - UNKNOWN - check upstream: 22 ## Quality gates - `make check` — catalog/provenance/audit/eval freshness, repo hygiene, Paper-WorkFlow gates, stdlib unit tests, eval-harness lint + graded smoke, and the strict numeric benchmark (reference-gold drift fails the build). - External links and OpenSSF Scorecard run on their scheduled workflows.
当前收录版本
0.0.1
质量评测
满分 100。前四项来自 DeepSkill 收录的质量报告。
评分依据
提供统计软件质量保证的完整方法论体系
评分依据
触发词精准覆盖 R package testing、numerical accuracy validation、statistical correctness verification 等核心场景
评分依据
Phase/步骤清晰,包含测试金字塔、多种测试模式、CI/CD 集成等完整流程
评分依据
基于标准 R 测试框架(testthat),工具来源明确
计算方式
取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。
权限与使用前检查
根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。
未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。
安全评级与访问范围
risk · 静态证据核验,不代表运行安全保证。
使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。
用户讨论
还没有已核实的点评。