网页抓取

本条目介绍 网页抓取 的公开用途。DeepSkill 当前公开目录说明:“用于网页抓取、自动化浏览器交互、网站内容爬取,或从渲染后的 JavaScript 页面中提取数据” 适用场景包括浏览器自动化。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

仓库
jeremylongshore/tons-of-skills-marketplace

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "web-crawling":https://skillpi.com/zh/skills/deepskill-d3e5178ce0c21e80b99825566c7b0d15878f3e7ad9fb95c13189942156ba5d20。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/web-crawling/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/web-crawling --skill web-crawling
许可协议: —

来源与配置

查看 GitHub 源码

jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目介绍 网页抓取 的公开用途。DeepSkill 当前公开目录说明:“用于网页抓取、自动化浏览器交互、网站内容爬取,或从渲染后的 JavaScript 页面中提取数据” 适用场景包括浏览器自动化。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: web-crawling
description: "Use when scraping web pages, automating browser interactions, crawling sites for content, or extracting data from rendered JavaScript pages"
allowed-tools: [Bash(node*), Bash(python3*), Bash(scrapy*), Bash(katana*), Read, Write]
version: 1.0.0
author: ykotik
license: MIT
---

# Web Crawling

## When to Use
- Scraping content from web pages (especially JavaScript-rendered)
- Automating browser interactions (login, form submission, navigation)
- Crawling a site to discover and extract structured data
- Taking screenshots or generating PDFs of web pages
- Discovering URLs and endpoints on a target site
- Extracting data from sites with anti-bot protection

## Tools

| Tool | Purpose | When to choose |
|------|---------|---------------|
| **Playwright** | Cross-browser automation (Chromium, Firefox, WebKit) | JS-rendered pages, screenshots, complex interactions |
| **Puppeteer** | Chrome/Firefox automation via DevTools Protocol | Chrome-specific automation, familiar API |
| **Scrapy** | Production-grade Python crawling framework | Large-scale structured crawling with pipelines |
| **Crawlee** | Node.js crawling with anti-blocking built in | Sites with bot detection, proxy rotation needed |
| **Katana** | Fast Go-based URL discovery crawler | Endpoint discovery, reconnaissance, speed |

## Patterns

### Playwright: Scrape a page (inline Node.js script)
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  const title = await page.title();
  const text = await page.locator('body').innerText();
  console.log(JSON.stringify({ title, text }));
  await browser.close();
})();
"
```

### Playwright: Take a screenshot
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({ path: 'screenshot.png', fullPage: true });
  await browser.close();
})();
"
```

### Playwright: Extract structured data from a table
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/data');
  const rows = await page.locator('table tr').evaluateAll(trs =>
    trs.map(tr => Array.from(tr.querySelectorAll('td,th')).map(c => c.textContent.trim()))
  );
  console.log(JSON.stringify(rows));
  await browser.close();
})();
"
```

### Playwright: Wait for JS content to load
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/spa');
  await page.waitForSelector('.content-loaded');
  const data = await page.locator('.content-loaded').innerText();
  console.log(data);
  await browser.close();
})();
"
```

### Puppeteer: Generate PDF from a page
```bash
node -e "
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'networkidle0' });
  await page.pdf({ path: 'output.pdf', format: 'A4' });
  await browser.close();
})();
"
```

### Katana: Discover all URLs on a site
```bash
katana -u https://example.com -silent -d 3
```

### Katana: Discover URLs and output as JSON
```bash
katana -u https://example.com -silent -d 2 -jsonl
```

### Katana: Crawl with headless browser for JS-rendered links
```bash
katana -u https://example.com -headless -d 2 -silent
```

### Scrapy: Quick single-page scrape (no project needed)
```bash
scrapy fetch --nolog https://example.com | python3 -c "
import sys
from scrapy import Selector
html = sys.stdin.read()
sel = Selector(text=html)
for item in sel.css('h2::text').getall():
    print(item)
"
```

### Crawlee: Basic crawler with anti-blocking
```bash
node -e "
const { PlaywrightCrawler } = require('crawlee');
const crawler = new PlaywrightCrawler({
  async requestHandler({ page, request }) {
    const title = await page.title();
    console.log(JSON.stringify({ url: request.url, title }));
  },
  maxRequestsPerCrawl: 10,
});
crawler.run(['https://example.com']);
"
```

## Pipelines

### Discover URLs → filter interesting ones → scrape
```bash
katana -u https://example.com -silent -d 2 | grep '/blog/' | head -5 | while read url; do
  node -e "
    const { chromium } = require('playwright');
    (async () => {
      const browser = await chromium.launch();
      const page = await browser.newPage();
      await page.goto('$url');
      const title = await page.title();
      const text = await page.locator('article').innerText().catch(() => '');
      console.log(JSON.stringify({ url: '$url', title, text: text.slice(0, 500) }));
      await browser.close();
    })();
  "
done
```
Each stage: Katana discovers URLs, grep filters to blog posts, Playwright scrapes each page.

### Crawl → extract to JSON → query with DuckDB
```bash
katana -u https://example.com -silent -d 2 -jsonl > crawl.jsonl
duckdb -c "SELECT endpoint, status_code, COUNT(*) FROM read_json_auto('crawl.jsonl') GROUP BY endpoint, status_code ORDER BY COUNT(*) DESC"
```
Each stage: Katana crawls to JSONL, DuckDB runs SQL on the crawl results.

## Prefer Over
- Prefer **Playwright** over curl for JavaScript-rendered pages — curl only gets raw HTML, Playwright executes JS
- Prefer **Katana** over manual URL enumeration — fast, recursive, handles JS links
- Prefer **Scrapy** over ad-hoc scripts for structured multi-page crawling — built-in rate limiting, pipelines, persistence

## Do NOT Use When
- Page content is available via a REST API — use the API directly (api-testing skill)
- Extracting article text from a news URL — use newspaper4k (web-research skill), it's purpose-built
- Simple static HTML that curl can fetch — use `curl` + `jq`/`pup`
- The site explicitly prohibits scraping — respect robots.txt and ToS

当前收录版本

1.0.0

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值92/100
评分依据

提供网页抓取、自动化浏览器交互、JS渲染页面数据提取等高频刚需能力

触发精准95/100
评分依据

description 明确覆盖多种场景,触发词完整

结构规范93/100
评分依据

Tools、Patterns、Pipelines 分层明确,结构清晰

执行可靠91/100
评分依据

工具来源明确,参数使用正确,语法规范

证据可验 (前四项均值)93/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

  • 截图或屏幕采集

    只采集明确选择的窗口或区域,避开密码、客户资料等敏感信息,分享前再次检查。

    主文档提及 · 第 56 行
文档引用的域名或地址

这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。

  • example.com

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。