数据处理

本条目介绍 数据处理 的公开用途。DeepSkill 当前公开目录说明:“用于处理结构化数据文件(CSV、JSON、YAML、TOML、Parquet)——包括查询、转换、过滤、聚合或在不同格式之间进行转换” 适用场景包括分析数据与指标归因、写 SQL 取数。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

仓库
jeremylongshore/tons-of-skills-marketplace

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "data-processing":https://skillpi.com/zh/skills/deepskill-857db2ce6711ee749ad9abb1a89b167edbb7dfb2e97dd511b8aeb1e8aadd9506。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/data-processing/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/data-processing --skill data-processing
许可协议: —

来源与配置

查看 GitHub 源码

jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目介绍 数据处理 的公开用途。DeepSkill 当前公开目录说明:“用于处理结构化数据文件(CSV、JSON、YAML、TOML、Parquet)——包括查询、转换、过滤、聚合或在不同格式之间进行转换” 适用场景包括分析数据与指标归因、写 SQL 取数。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: data-processing
description: "Use when working with structured data files (CSV, JSON, YAML, TOML, Parquet) — querying, transforming, filtering, aggregating, or converting between formats"
allowed-tools: [Bash(jq*), Bash(yq*), Bash(gron*), Bash(mlr*), Bash(xsv*), Bash(duckdb*), Read, Glob, Grep]
version: 1.0.0
author: ykotik
license: MIT
---

# Data Processing

## When to Use
- Querying, filtering, or transforming JSON files
- Reading or converting YAML, TOML, or XML config files
- Analyzing, aggregating, or joining CSV/TSV/Parquet files
- Running SQL queries against local data files without a database server
- Converting between data formats (JSON to YAML, CSV to JSON, etc.)
- Exploring deeply nested JSON structures

## Tools

| Tool | Purpose | Structured output |
|------|---------|-------------------|
| **jq** | Query and transform JSON | Native JSON |
| **yq** | Query and transform YAML, TOML, XML | `-o json` for JSON output |
| **gron** | Flatten JSON into greppable `path.to.key = value` lines | `--ungron` to reverse back to JSON |
| **miller (mlr)** | Transform CSV/JSON/TSV records with awk-like verbs | `--json` for JSON output |
| **xsv** | Fast CSV slicing, searching, joining, statistics | CSV native (pipe to `xsv table` for display) |
| **DuckDB** | SQL queries on CSV, JSON, Parquet files | `-json` flag for JSON output |

## Patterns

### JSON: Filter array elements by field value
```bash
jq '.items[] | select(.status == "active")' data.json
```

### JSON: Extract specific fields from array
```bash
jq '[.users[] | {name: .name, email: .email}]' data.json
```

### JSON: Count items grouped by field
```bash
jq '[.events[] | .type] | group_by(.) | map({type: .[0], count: length})' data.json
```

### YAML: Read a nested value
```bash
yq '.spec.containers[0].image' deployment.yaml
```

### YAML: Convert entire file to JSON
```bash
yq -o json config.yaml
```

### TOML: Read a value
```bash
yq -p toml '.database.host' config.toml
```

### JSON: Explore unknown structure by grepping paths
```bash
gron data.json | grep -i "error"
```

### CSV: Column statistics (min, max, mean, stddev)
```bash
xsv stats data.csv | xsv table
```

### CSV: Search rows matching a pattern in a column
```bash
xsv search -s status "active" data.csv
```

### CSV: Select specific columns
```bash
xsv select name,email,created_at users.csv
```

### CSV: Sort by column
```bash
xsv sort -s revenue -R sales.csv
```

### SQL: Query a CSV file
```bash
duckdb -c "SELECT department, COUNT(*) as cnt, AVG(salary) as avg_sal FROM 'employees.csv' GROUP BY department ORDER BY avg_sal DESC"
```

### SQL: Query a JSON file
```bash
duckdb -c "SELECT * FROM read_json_auto('events.json') WHERE type = 'error' LIMIT 20"
```

### SQL: Query Parquet files
```bash
duckdb -c "SELECT * FROM 'data/*.parquet' WHERE created_at > '2026-01-01'"
```

### CSV/JSON: Transform records with miller
```bash
mlr --csv filter '$revenue > 1000' then sort-by -nr revenue sales.csv
```

### Format conversion: CSV to JSON
```bash
mlr --icsv --ojson cat data.csv
```

## Pipelines

### YAML config → JSON → SQL query
```bash
yq -o json config.yaml | duckdb -c "SELECT key, value FROM read_json_auto('/dev/stdin') WHERE env = 'production'"
```
Each stage: yq converts YAML to JSON, DuckDB runs SQL on the JSON stream.

### Grep nested JSON paths → reconstruct matching subset
```bash
gron large.json | grep "\.errors\[" | gron --ungron
```
Each stage: gron flattens JSON to paths, grep filters, ungron reconstructs valid JSON from matches.

### CSV filter → aggregate with SQL
```bash
xsv search -s region "EU" sales.csv | duckdb -c "SELECT product, SUM(revenue) as total FROM read_csv_auto('/dev/stdin') GROUP BY product ORDER BY total DESC"
```
Each stage: xsv filters rows by region, DuckDB aggregates the filtered stream.

### Join two CSV files with SQL
```bash
duckdb -c "SELECT u.name, u.email, o.total, o.date FROM 'users.csv' u JOIN 'orders.csv' o ON u.id = o.user_id ORDER BY o.date DESC"
```

### Multi-format pipeline: JSON → CSV → stats
```bash
jq -r '.records[] | [.name, .score] | @csv' data.json | xsv stats
```
Each stage: jq extracts fields to CSV format, xsv computes statistics.

## Prefer Over
- Prefer **DuckDB** over Python/pandas for ad-hoc SQL queries on files — single command, no script, handles large files
- Prefer **jq** over Python `json` module for one-off JSON transforms — single pipeline vs. multi-line script
- Prefer **xsv** over `awk`/`cut` for CSV operations — correct CSV parsing, handles quoted fields and escapes
- Prefer **miller** over `awk` for format-aware record transformations — understands CSV/JSON headers natively
- Prefer **yq** over custom parsers for config file reads — handles YAML, TOML, XML with consistent syntax

## Do NOT Use When
- Data is already in a running database — query the database directly
- File is under 10 lines — just use the Read tool and process in-context
- Task requires complex multi-step logic with conditionals — write a Python script instead
- JSON is simple enough to read by eye — use the Read tool, don't over-engineer
- Working with binary or non-structured data formats

当前收录版本

1.0.0

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值95/100
评分依据

覆盖多种结构化数据格式(CSV、JSON、YAML、TOML、Parquet)的处理需求

触发精准95/100
评分依据

触发词精准完整,覆盖多种变体,边界清晰

结构规范95/100
评分依据

Phase/步骤清晰,有出口条件,含端到端示例

执行可靠95/100
评分依据

工具来源明确,参数正确,覆盖主要异常

证据可验 (前四项均值)95/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。