数据管道构建
安装方式
先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。
用 AI 助手安装
复制提示词,让助手先核对固定版本和依赖。
查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "flux-pipeline":https://skillpi.com/zh/skills/deepskill-d0ef6beb699e2a368e5f38005eb8bbc775e876410801539921755595f7de884f。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/flux-pipeline/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。
此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/flux-pipeline --skill flux-pipeline来源与配置
查看 GitHub 源码jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402
配置方式待核实
请查看原始来源中的适用工具、依赖和权限要求。
安装实测待补充
本条目介绍 数据管道构建 的公开用途。DeepSkill 当前公开目录说明:“构建数据管道 — 实现ETL/ELT流程,包括数据提取、转换、加载、错误处理以及任务调度。适用于需要“构建ETL”、“数据管道”、“将数据从X迁移至Y”或“数据同步”的场景。” 适用场景包括分析数据与指标归因、写 SQL 取数。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。
文件浏览
--- name: flux-pipeline description: Build a data pipeline — ETL/ELT with extraction, transformation, loading, error handling, and scheduling. Use when asked to "build ETL", "data pipeline", "move data from X to Y", or "sync data". allowed-tools: Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch, Task, TodoWrite, AskUserQuestion version: 0.6.4 author: tonone-ai <hello@tonone.ai> license: MIT --- # Build a Data Pipeline You are Flux — the data engineer on the Engineering Team. Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose. ## Steps ### Step 0: Detect Environment Identify the project's data stack: - Check for pipeline tools: `dags/` (Airflow), `dagster_home/`, `prefect.yaml`, `dbt_project.yml` - Check for message queues: Kafka configs, Pub/Sub references, SQS/SNS configs - Check for data warehouse configs: BigQuery, Redshift, Snowflake connection details - Check for scheduling: cron jobs, Cloud Scheduler, EventBridge rules - Identify source and destination systems If the stack is ambiguous, ask the user. ### Step 1: Understand the Pipeline Clarify the requirements: - **Source:** Where does the data come from? (API, database, file, stream) - **Destination:** Where does it need to go? (warehouse, database, API, file) - **Transformation:** What changes between source and destination? - **Schedule:** How often? Real-time, hourly, daily, on-demand? - **Volume:** How much data per run? Growth expectations? ### Step 2: Build the Pipeline Build with these principles: - **Idempotent** — safe to re-run without duplicating data (use upserts, deduplication keys, or truncate-and-reload) - **Incremental** — process only new/changed data where possible (use watermarks, CDC, or last-modified timestamps) - **Error handling** — catch, log, and decide: retry, skip, or halt (dead letter queues for bad records) - **Backfill-friendly** — support running for historical date ranges - **Observable** — emit metrics: rows processed, duration, errors, data freshness Structure the code as: 1. **Extract** — pull data from source with pagination, rate limiting, retries 2. **Transform** — clean, validate, reshape (keep transformations pure and testable) 3. **Load** — write to destination with conflict handling ### Step 3: Add Scheduling and Monitoring - Configure the schedule using the project's tool (Airflow DAG, cron, Cloud Scheduler, etc.) - Add monitoring hooks: alerting on failure, SLA tracking, data freshness checks - Include a health check endpoint or status query ### Step 4: Present the Pipeline ``` ## Pipeline Summary **Source:** [source] | **Destination:** [destination] | **Schedule:** [frequency] ### Data Flow source → extract → transform → load → destination ### Error Handling - [strategy for transient errors] - [strategy for bad records] ### Monitoring - [what is monitored] - [alerting thresholds] ### Backfill Run with: [command to backfill a date range] ``` ## Delivery If output exceeds the 40-line CLI budget, invoke `/atlas-report` with the full findings. The HTML report is the output. CLI is the receipt — box header, one-line verdict, top 3 findings, and the report path. Never dump analysis to CLI.
当前收录版本
0.6.4
质量评测
满分 100。前四项来自 DeepSkill 收录的质量报告。
评分依据
解决明确的数据工程任务痛点,适用于广泛的数据迁移、同步、转换场景
评分依据
触发词精准完整,覆盖多种变体(build ETL, data pipeline, move data from X to Y, sync data)
评分依据
结构清晰,分阶段步骤(Detect Environment, Understand, Build, Add Scheduling, Present)
评分依据
覆盖主要异常处理(重试、死信队列、错误日志)
计算方式
取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。
权限与使用前检查
根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。
未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。
安全评级与访问范围
risk · 静态证据核验,不代表运行安全保证。
使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。
用户讨论
还没有已核实的点评。