网页抓取
安装方式
先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。
用 AI 助手安装
复制提示词,让助手先核对固定版本和依赖。
查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "web-crawling":https://skillpi.com/zh/skills/deepskill-d3e5178ce0c21e80b99825566c7b0d15878f3e7ad9fb95c13189942156ba5d20。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/web-crawling/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。
此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/productivity/cli-power-skills/skills/web-crawling --skill web-crawling来源与配置
查看 GitHub 源码jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402
配置方式待核实
请查看原始来源中的适用工具、依赖和权限要求。
安装实测待补充
本条目介绍 网页抓取 的公开用途。DeepSkill 当前公开目录说明:“用于网页抓取、自动化浏览器交互、网站内容爬取,或从渲染后的 JavaScript 页面中提取数据” 适用场景包括浏览器自动化。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。
文件浏览
---
name: web-crawling
description: "Use when scraping web pages, automating browser interactions, crawling sites for content, or extracting data from rendered JavaScript pages"
allowed-tools: [Bash(node*), Bash(python3*), Bash(scrapy*), Bash(katana*), Read, Write]
version: 1.0.0
author: ykotik
license: MIT
---
# Web Crawling
## When to Use
- Scraping content from web pages (especially JavaScript-rendered)
- Automating browser interactions (login, form submission, navigation)
- Crawling a site to discover and extract structured data
- Taking screenshots or generating PDFs of web pages
- Discovering URLs and endpoints on a target site
- Extracting data from sites with anti-bot protection
## Tools
| Tool | Purpose | When to choose |
|------|---------|---------------|
| **Playwright** | Cross-browser automation (Chromium, Firefox, WebKit) | JS-rendered pages, screenshots, complex interactions |
| **Puppeteer** | Chrome/Firefox automation via DevTools Protocol | Chrome-specific automation, familiar API |
| **Scrapy** | Production-grade Python crawling framework | Large-scale structured crawling with pipelines |
| **Crawlee** | Node.js crawling with anti-blocking built in | Sites with bot detection, proxy rotation needed |
| **Katana** | Fast Go-based URL discovery crawler | Endpoint discovery, reconnaissance, speed |
## Patterns
### Playwright: Scrape a page (inline Node.js script)
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
const title = await page.title();
const text = await page.locator('body').innerText();
console.log(JSON.stringify({ title, text }));
await browser.close();
})();
"
```
### Playwright: Take a screenshot
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({ path: 'screenshot.png', fullPage: true });
await browser.close();
})();
"
```
### Playwright: Extract structured data from a table
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/data');
const rows = await page.locator('table tr').evaluateAll(trs =>
trs.map(tr => Array.from(tr.querySelectorAll('td,th')).map(c => c.textContent.trim()))
);
console.log(JSON.stringify(rows));
await browser.close();
})();
"
```
### Playwright: Wait for JS content to load
```bash
node -e "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/spa');
await page.waitForSelector('.content-loaded');
const data = await page.locator('.content-loaded').innerText();
console.log(data);
await browser.close();
})();
"
```
### Puppeteer: Generate PDF from a page
```bash
node -e "
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle0' });
await page.pdf({ path: 'output.pdf', format: 'A4' });
await browser.close();
})();
"
```
### Katana: Discover all URLs on a site
```bash
katana -u https://example.com -silent -d 3
```
### Katana: Discover URLs and output as JSON
```bash
katana -u https://example.com -silent -d 2 -jsonl
```
### Katana: Crawl with headless browser for JS-rendered links
```bash
katana -u https://example.com -headless -d 2 -silent
```
### Scrapy: Quick single-page scrape (no project needed)
```bash
scrapy fetch --nolog https://example.com | python3 -c "
import sys
from scrapy import Selector
html = sys.stdin.read()
sel = Selector(text=html)
for item in sel.css('h2::text').getall():
print(item)
"
```
### Crawlee: Basic crawler with anti-blocking
```bash
node -e "
const { PlaywrightCrawler } = require('crawlee');
const crawler = new PlaywrightCrawler({
async requestHandler({ page, request }) {
const title = await page.title();
console.log(JSON.stringify({ url: request.url, title }));
},
maxRequestsPerCrawl: 10,
});
crawler.run(['https://example.com']);
"
```
## Pipelines
### Discover URLs → filter interesting ones → scrape
```bash
katana -u https://example.com -silent -d 2 | grep '/blog/' | head -5 | while read url; do
node -e "
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('$url');
const title = await page.title();
const text = await page.locator('article').innerText().catch(() => '');
console.log(JSON.stringify({ url: '$url', title, text: text.slice(0, 500) }));
await browser.close();
})();
"
done
```
Each stage: Katana discovers URLs, grep filters to blog posts, Playwright scrapes each page.
### Crawl → extract to JSON → query with DuckDB
```bash
katana -u https://example.com -silent -d 2 -jsonl > crawl.jsonl
duckdb -c "SELECT endpoint, status_code, COUNT(*) FROM read_json_auto('crawl.jsonl') GROUP BY endpoint, status_code ORDER BY COUNT(*) DESC"
```
Each stage: Katana crawls to JSONL, DuckDB runs SQL on the crawl results.
## Prefer Over
- Prefer **Playwright** over curl for JavaScript-rendered pages — curl only gets raw HTML, Playwright executes JS
- Prefer **Katana** over manual URL enumeration — fast, recursive, handles JS links
- Prefer **Scrapy** over ad-hoc scripts for structured multi-page crawling — built-in rate limiting, pipelines, persistence
## Do NOT Use When
- Page content is available via a REST API — use the API directly (api-testing skill)
- Extracting article text from a news URL — use newspaper4k (web-research skill), it's purpose-built
- Simple static HTML that curl can fetch — use `curl` + `jq`/`pup`
- The site explicitly prohibits scraping — respect robots.txt and ToS
当前收录版本
1.0.0
质量评测
满分 100。前四项来自 DeepSkill 收录的质量报告。
评分依据
提供网页抓取、自动化浏览器交互、JS渲染页面数据提取等高频刚需能力
评分依据
description 明确覆盖多种场景,触发词完整
评分依据
Tools、Patterns、Pipelines 分层明确,结构清晰
评分依据
工具来源明确,参数使用正确,语法规范
计算方式
取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。
权限与使用前检查
根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。
-
截图或屏幕采集
只采集明确选择的窗口或区域,避开密码、客户资料等敏感信息,分享前再次检查。
主文档提及 · 第 56 行
文档引用的域名或地址
这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。
- example.com
安全评级与访问范围
risk · 静态证据核验,不代表运行安全保证。
使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。
用户讨论
还没有已核实的点评。