grobid-pdf-parsing
安装方式
先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。
用 AI 助手安装
复制提示词,让助手先核对固定版本和依赖。
查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "grobid-pdf-parsing":https://skillpi.com/zh/skills/deepskill-7d40ddea3537e7305193242cf4e6d3ab8b5acfcce73250781b459929ab764df1。安装前核对上游来源和版本:https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/tools/document/grobid-pdf-parsing/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。
此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。
npx skills add https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/tree/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/tools/document/grobid-pdf-parsing --skill grobid-pdf-parsing来源与配置
查看 GitHub 源码brycewang-stanford/Auto-Empirical-Research-Skills固定提交 bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba
配置方式待核实
请查看原始来源中的适用工具、依赖和权限要求。
安装实测待补充
本条目面向文献与学术工具场景,提供“grobid-pdf-parsing”技能的目录入口。公开标签:learning、学术PDF、GROBID、TEI XML。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。
文件浏览
---
name: grobid-pdf-parsing
description: "Extract structured text, metadata, and references from academic PDFs"
metadata:
openclaw:
emoji: "📄"
category: "tools"
subcategory: "document"
keywords: ["PDF parsing", "PDF extraction", "document chunking", "format conversion"]
source: "https://github.com/kermitt2/grobid"
---
# GROBID PDF Parsing Guide
## Overview
Academic PDFs are the primary format for distributing research, yet extracting structured data from them remains challenging. PDFs encode visual layout, not semantic structure -- headings, paragraphs, equations, tables, and citations are all just positioned text and graphics. GROBID (GeneRation Of BIbliographic Data) is the leading open-source tool for parsing academic PDFs into structured XML/TEI format, extracting metadata, body text, references, and figures with high accuracy.
GROBID is used by major academic platforms including CORE, ResearchGate, and others for large-scale document processing. It combines machine learning models (CRF and deep learning) with heuristic rules to handle the diverse formatting of academic papers across publishers and disciplines.
This guide covers installing and running GROBID, using its REST API for batch processing, extracting specific elements (metadata, references, body sections), and integrating GROBID output into downstream workflows such as knowledge bases, systematic reviews, and literature analysis pipelines.
## Installation
### Docker (Recommended)
```bash
# Pull the latest GROBID image
docker pull grobid/grobid:0.8.1
# Run GROBID server
docker run --rm --init \
--ulimit core=0 \
-p 8070:8070 \
grobid/grobid:0.8.1
# GROBID is now running at http://localhost:8070
# Web console: http://localhost:8070/console
```
### From Source
```bash
git clone https://github.com/kermitt2/grobid.git
cd grobid
./gradlew clean install
./gradlew run
```
## REST API Usage
### Process Full Document
```bash
# Process a single PDF and get TEI XML
curl -v --form input=@paper.pdf \
http://localhost:8070/api/processFulltextDocument \
-o paper.tei.xml
# With options
curl -v --form input=@paper.pdf \
--form consolidateHeader=1 \
--form consolidateCitations=1 \
--form includeRawCitations=1 \
http://localhost:8070/api/processFulltextDocument \
-o paper.tei.xml
```
### API Endpoints
| Endpoint | Purpose | Input | Output |
|----------|---------|-------|--------|
| `/api/processFulltextDocument` | Full paper parsing | PDF | TEI XML |
| `/api/processHeaderDocument` | Metadata only | PDF | TEI XML (header) |
| `/api/processReferences` | Reference parsing | PDF | TEI XML (refs) |
| `/api/processCitation` | Parse citation string | Text | TEI XML |
| `/api/processDate` | Parse date string | Text | Structured date |
### Python Client
```python
import requests
from pathlib import Path
class GrobidClient:
def __init__(self, base_url='http://localhost:8070'):
self.base_url = base_url
def process_fulltext(self, pdf_path, consolidate_header=True,
consolidate_citations=True):
"""Process a PDF and return TEI XML."""
url = f'{self.base_url}/api/processFulltextDocument'
files = {'input': open(pdf_path, 'rb')}
data = {
'consolidateHeader': '1' if consolidate_header else '0',
'consolidateCitations': '1' if consolidate_citations else '0',
}
response = requests.post(url, files=files, data=data)
response.raise_for_status()
return response.text
def process_header(self, pdf_path):
"""Extract only header metadata from PDF."""
url = f'{self.base_url}/api/processHeaderDocument'
files = {'input': open(pdf_path, 'rb')}
response = requests.post(url, files=files)
response.raise_for_status()
return response.text
def is_alive(self):
"""Check if GROBID server is running."""
try:
resp = requests.get(f'{self.base_url}/api/isalive')
return resp.status_code == 200
except requests.ConnectionError:
return False
# Usage
client = GrobidClient()
if client.is_alive():
tei_xml = client.process_fulltext('paper.pdf')
with open('paper.tei.xml', 'w') as f:
f.write(tei_xml)
```
## Parsing TEI XML Output
### Extracting Metadata
```python
from lxml import etree
def parse_tei_metadata(tei_xml):
"""Extract title, authors, abstract from TEI XML."""
ns = {'tei': 'http://www.tei-c.org/ns/1.0'}
root = etree.fromstring(tei_xml.encode('utf-8'))
# Title
title_el = root.find('.//tei:titleStmt/tei:title', ns)
title = title_el.text if title_el is not None else ''
# Authors
authors = []
for author in root.findall('.//tei:sourceDesc//tei:author', ns):
forename = author.findtext('.//tei:forename', '', ns)
surname = author.findtext('.//tei:surname', '', ns)
if surname:
authors.append(f'{forename} {surname}'.strip())
# Abstract
abstract_el = root.find('.//tei:profileDesc/tei:abstract', ns)
abstract = ''.join(abstract_el.itertext()).strip() if abstract_el is not None else ''
# DOI
doi_el = root.find('.//tei:idno[@type="DOI"]', ns)
doi = doi_el.text if doi_el is not None else ''
return {
'title': title,
'authors': authors,
'abstract': abstract,
'doi': doi,
}
```
### Extracting Body Sections
```python
def parse_tei_sections(tei_xml):
"""Extract structured sections from TEI XML body."""
ns = {'tei': 'http://www.tei-c.org/ns/1.0'}
root = etree.fromstring(tei_xml.encode('utf-8'))
sections = []
for div in root.findall('.//tei:body/tei:div', ns):
head = div.findtext('tei:head', '', ns).strip()
paragraphs = []
for p in div.findall('tei:p', ns):
text = ''.join(p.itertext()).strip()
if text:
paragraphs.append(text)
sections.append({
'heading': head,
'n': div.get('n', ''),
'paragraphs': paragraphs,
})
return sections
```
### Extracting References
```python
def parse_tei_references(tei_xml):
"""Extract structured references from TEI XML."""
ns = {'tei': 'http://www.tei-c.org/ns/1.0'}
root = etree.fromstring(tei_xml.encode('utf-8'))
refs = []
for bib in root.findall('.//tei:listBibl/tei:biblStruct', ns):
ref = {'id': bib.get('{http://www.w3.org/XML/1998/namespace}id', '')}
# Title
title_el = bib.find('.//tei:title[@level="a"]', ns)
if title_el is None:
title_el = bib.find('.//tei:title', ns)
ref['title'] = title_el.text if title_el is not None else ''
# Authors
ref['authors'] = []
for author in bib.findall('.//tei:author', ns):
name = f"{author.findtext('.//tei:forename', '', ns)} {author.findtext('.//tei:surname', '', ns)}".strip()
if name:
ref['authors'].append(name)
# Year
date_el = bib.find('.//tei:date[@type="published"]', ns)
ref['year'] = date_el.get('when', '') if date_el is not None else ''
# DOI
doi_el = bib.find('.//tei:idno[@type="DOI"]', ns)
ref['doi'] = doi_el.text if doi_el is not None else ''
refs.append(ref)
return refs
```
## Batch Processing
### Processing a Directory of PDFs
```python
from pathlib import Path
import json
from concurrent.futures import ThreadPoolExecutor
def batch_process(pdf_dir, output_dir, max_workers=4):
"""Process all PDFs in a directory using GROBID."""
client = GrobidClient()
pdf_dir = Path(pdf_dir)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
pdf_files = list(pdf_dir.glob('*.pdf'))
print(f"Processing {len(pdf_files)} PDFs...")
def process_one(pdf_path):
try:
tei = client.process_fulltext(str(pdf_path))
meta = parse_tei_metadata(tei)
refs = parse_tei_references(tei)
# Save TEI XML
tei_path = output_dir / f'{pdf_path.stem}.tei.xml'
tei_path.write_text(tei)
# Save structured JSON
json_path = output_dir / f'{pdf_path.stem}.json'
json_path.write_text(json.dumps({
'metadata': meta,
'references': refs,
'n_references': len(refs),
}, indent=2))
return pdf_path.name, 'success'
except Exception as e:
return pdf_path.name, f'error: {str(e)}'
with ThreadPoolExecutor(max_workers=max_workers) as executor:
results = list(executor.map(process_one, pdf_files))
for name, status in results:
print(f" {name}: {status}")
batch_process('papers/', 'parsed_output/')
```
## Best Practices
- **Use consolidation flags.** `consolidateHeader=1` and `consolidateCitations=1` cross-reference against Crossref for better metadata.
- **Handle errors gracefully.** Some PDFs are scanned images, corrupted, or have unusual layouts. Always wrap processing in try/except.
- **Limit concurrent requests.** GROBID is CPU-intensive. 4-8 concurrent requests is usually optimal.
- **Validate output.** Spot-check a sample of parsed documents against the original PDFs.
- **Use GROBID for structured extraction, not OCR.** For scanned documents, run OCR first (Tesseract) then GROBID.
- **Keep GROBID updated.** Each release improves parsing accuracy, especially for newer publisher formats.
## References
- [GROBID Documentation](https://grobid.readthedocs.io/) -- Official documentation
- [GROBID GitHub](https://github.com/kermitt2/grobid) -- Source code
- [TEI Guidelines](https://tei-c.org/release/doc/tei-p5-doc/en/html/) -- TEI XML standard
- [grobid-client-python](https://github.com/kermitt2/grobid_client_python) -- Official Python client
- [Science Parse](https://github.com/allenai/science-parse) -- Allen AI alternative parser
版本历史
-
v2026.07
GitHub ↗
原版更新说明
## Highlights - **First tagged release.** Every gate is green: `make check` (validate + 183 unit tests + eval-harness + numeric benchmarks), validate-catalog, quality-evals, and OpenSSF Scorecard on `main`. - **Rigor double loop at 13 method families.** Each family closes taxonomy tag → eval scenario (does the agent avoid the method's known traps?) → deterministic numeric benchmark (does it recover the true effect?). New this cycle: heterogeneous treatment effects (CATE) and quantile treatment effects (QTE). - **Paper-WorkFlow flagship at 29/29 executable rigor gates**, now merged to its `main` with a drift-gated README badge — the credibility artifact is checked by CI, so it can never silently go stale ([RIGOR.md](https://github.com/brycewang-stanford/Paper-WorkFlow/blob/main/RIGOR.md)). - **1,150 vendored skills / 69 collections**, all license-audited, security-scanned, tracked in a machine-readable catalog with CI-enforced freshness, and importable as a single whole-repo router skill. - **Hardened weekly upstream syncs**: synced trees must pass the full gate before merge; this cycle both sync PRs were verified, patched (StatsPAI's SkillOpt execution-gate card restored), and merged. - **First-party plugin marketplace** (Python / R / Stata empirical-analysis plugins + guide) shipped in the previous cycle, now under a tagged version. ## Known Follow-Ups - Ecosystem benchmark comparison (AERS vs econometrics agents on the shared benchmark suite) — [PLAN-2026-07 week 3](https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/main/docs/PLAN-2026-07.md). - Decomposition (Oaxaca–Blinder) and shift-share (Bartik) method families (weeks 3–4). - GitHub Pages catalog site, README demo screenshots, five-language README consistency script (week 4). ## Catalog - Top-level collections: **69** - `SKILL.md` files: **1150** - Cataloged agent tools: **335** ([`tools/CATALOG.md`](../tools/CATALOG.md)) ## Methodological rigor - Method families in the coverage map: **13** ([`RIGOR_COVERAGE.md`](RIGOR_COVERAGE.md)) - Eval scenarios: **26** (138 rubric items, 122 auto-checkable) ([`../eval-harness/`](../eval-harness/README.md)) - Benchmark tasks: **13** (70 gold checks, 58 required) ([`../benchmark/`](../benchmark/README.md)) ### Method families (skills tagged) - `iv`: 27 - `rdd`: 25 - `panel-fe`: 19 - `staggered-did`: 17 - `bayesian`: 13 - `did`: 10 - `event-study`: 10 - `matching`: 10 - `synthetic-control`: 10 - `cate`: 8 - `dml`: 8 - `survival`: 3 - `quantile`: 2 ### Eval scenarios by category - `causal-identification`: 14 - `citation-hygiene`: 1 - `reproducibility`: 2 - `research-integrity`: 1 - `runtime-safety`: 1 - `statistical-validity`: 2 - `writing-compliance`: 3 - `writing-style`: 2 ## License buckets - AGPL-3.0: 1 - Apache-2.0: 1 - CC-BY-NC-4.0 (non-commercial): 1 - CC-BY-SA-4.0: 1 - CC-BY-SA-4.0 (repository default): 4 - GPL-3.0: 1 - MIT: 35 - MIT Non-Commercial: 1 - Mixed (per-skill; see folder): 2 - UNKNOWN - check upstream: 22 ## Quality gates - `make check` — catalog/provenance/audit/eval freshness, repo hygiene, Paper-WorkFlow gates, stdlib unit tests, eval-harness lint + graded smoke, and the strict numeric benchmark (reference-gold drift fails the build). - External links and OpenSSF Scorecard run on their scheduled workflows.
当前收录版本
0.0.1
质量评测
满分 100。前四项来自 DeepSkill 收录的质量报告。
评分依据
解决学术 PDF 结构化提取的高频刚需场景
评分依据
功能描述清晰明确,关键词覆盖合理
评分依据
文档结构完整,逻辑清晰,涵盖安装、API、解析、批处理全流程
评分依据
提供 Docker 和源码两种可靠安装方式
计算方式
取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。
权限与使用前检查
根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。
-
读取本地文件
限定到你选择的文件或项目目录;避免读取无关的个人资料、密钥和配置文件。
主文档提及 · 第 93, 105 行 -
保存或修改文件
先确认输出目录和同名文件是否会被覆盖;重要文件保留备份,优先写入新的输出目录。
主文档提及 · 第 122, 256, 260 行
文档引用的域名或地址
这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。
- github.com
- grobid.readthedocs.io
- localhost
- tei-c.org
- www.tei-c.org
- www.w3.org
安全评级与访问范围
risk · 静态证据核验,不代表运行安全保证。
使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。
用户讨论
还没有已核实的点评。