deep-learning-papers-guide

本条目面向文献与学术工具场景,提供“deep-learning-papers-guide”技能的目录入口。公开标签:learning、深度学习、PyTorch、Transformer。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。

仓库
brycewang-stanford/Auto-Empirical-Research-Skills
仓库最近推送
Stars
5,023
Forks
578
Issues / PRs
0

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "deep-learning-papers-guide":https://skillpi.com/zh/skills/deepskill-dfd61b8d33a285fb8e430f7497f7599efc922e84f78591ce887d3decbe520c45。安装前核对上游来源和版本:https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/domains/ai-ml/deep-learning-papers-guide/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/tree/bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba/skills/43-wentorai-research-plugins/skills/domains/ai-ml/deep-learning-papers-guide --skill deep-learning-papers-guide
许可协议: —

来源与配置

查看 GitHub 源码

brycewang-stanford/Auto-Empirical-Research-Skills固定提交 bfa0a6e5a843babbf45a1ef1663fd5dc4f07c0ba

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目面向文献与学术工具场景,提供“deep-learning-papers-guide”技能的目录入口。公开标签:learning、深度学习、PyTorch、Transformer。请通过固定上游版本核对实际功能、依赖与权限;此页为 Skillπ 独立编写的用途介绍和原站链接,不代表运行安全保证。原文与附件依各自许可开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: deep-learning-papers-guide
description: "Annotated deep learning paper implementations with code walkthroughs"
metadata:
  openclaw:
    emoji: "🧠"
    category: "domains"
    subcategory: "ai-ml"
    keywords: ["deep learning", "neural network", "Transformer", "CNN", "NLP", "computer vision"]
    source: "https://github.com/labmlai/annotated_deep_learning_paper_implementations"
---

# Deep Learning Papers Guide

## Overview

Understanding deep learning architectures requires more than reading papers -- it requires reading and writing code. The annotated_deep_learning_paper_implementations repository (65,800+ stars) provides line-by-line annotated implementations of seminal deep learning papers in PyTorch, making it one of the most valuable learning resources in the field.

This guide organizes the key architectures by category, provides implementation patterns for the most important building blocks, and offers strategies for going from paper to working code. Whether you are implementing a Transformer variant for your research, understanding a GAN architecture for your experiments, or teaching a deep learning course, these patterns accelerate the process.

The focus is on practical understanding: what each component does, why it is designed that way, and how to implement it correctly in PyTorch.

## Core Architecture Families

### Transformer Architectures

The Transformer (Vaswani et al., 2017) is the foundation of modern NLP and increasingly of computer vision.

#### Multi-Head Self-Attention

```python
import torch
import torch.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, n_heads: int):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_k = d_model // n_heads

        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, query, key, value, mask=None):
        batch_size = query.size(0)

        # Linear projections and reshape to (batch, heads, seq, d_k)
        Q = self.W_q(query).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)
        K = self.W_k(key).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)
        V = self.W_v(value).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)

        # Scaled dot-product attention
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        attn = torch.softmax(scores, dim=-1)
        context = torch.matmul(attn, V)

        # Concatenate heads and project
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)
        return self.W_o(context)
```

#### Transformer Encoder Block

```python
class TransformerBlock(nn.Module):
    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.1):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, n_heads)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout)
        )
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Pre-norm variant (used in GPT-2, ViT, modern architectures)
        attn_out = self.attention(self.norm1(x), self.norm1(x), self.norm1(x), mask)
        x = x + self.dropout(attn_out)
        x = x + self.ffn(self.norm2(x))
        return x
```

### Convolutional Neural Networks

#### ResNet Bottleneck Block

```python
class BottleneckBlock(nn.Module):
    expansion = 4

    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3,
                               stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.conv3 = nn.Conv2d(out_channels, out_channels * self.expansion, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_channels * self.expansion)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        return self.relu(out)
```

## Key Architecture Comparison

| Architecture | Year | Parameters | Key Innovation | Primary Domain |
|-------------|------|------------|----------------|---------------|
| ResNet | 2015 | 25M (ResNet-50) | Skip connections | Vision |
| Transformer | 2017 | Varies | Self-attention | NLP |
| BERT | 2018 | 340M (Large) | Masked language modeling | NLP |
| GPT-2 | 2019 | 1.5B | Autoregressive generation | NLP |
| ViT | 2020 | 86M (Base) | Patch-based image tokenization | Vision |
| Diffusion | 2020 | Varies | Iterative denoising | Generation |
| LLaMA | 2023 | 7B-70B | Efficient open LLM | NLP |

## Training Patterns

### Standard Training Loop

```python
def train_epoch(model, dataloader, optimizer, criterion, device):
    model.train()
    total_loss = 0
    for batch_idx, (data, targets) in enumerate(dataloader):
        data, targets = data.to(device), targets.to(device)

        optimizer.zero_grad()
        outputs = model(data)
        loss = criterion(outputs, targets)
        loss.backward()

        # Gradient clipping (crucial for Transformers)
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        optimizer.step()
        total_loss += loss.item()

    return total_loss / len(dataloader)
```

### Learning Rate Scheduling

```python
# Cosine annealing with warmup (standard for Transformers)
from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR, SequentialLR

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
warmup = LinearLR(optimizer, start_factor=0.01, total_iters=1000)
cosine = CosineAnnealingLR(optimizer, T_max=50000)
scheduler = SequentialLR(optimizer, schedulers=[warmup, cosine], milestones=[1000])
```

## From Paper to Code: A Methodology

1. **Read the paper twice.** First pass for high-level understanding; second pass for implementation details.
2. **Identify the core algorithm.** Usually in Section 3 or 4 of the paper.
3. **List all hyperparameters.** Create a config dict before writing any code.
4. **Implement bottom-up.** Start with the smallest building blocks (attention, normalization), then compose.
5. **Test each component in isolation.** Verify tensor shapes and gradients at each level.
6. **Reproduce a known result first.** Match the paper's numbers on a small dataset before scaling.
7. **Use the official implementation as reference.** But write your own code for understanding.

## Best Practices

- **Always verify tensor shapes.** Add assert statements for dimensions during development.
- **Use mixed precision training.** `torch.cuda.amp` provides 2x speedup with minimal accuracy loss.
- **Log everything.** Use Weights & Biases or TensorBoard for experiment tracking.
- **Start small.** Debug on a tiny dataset before running on the full one.
- **Read the appendix.** Critical details (learning rates, initialization, data augmentation) are often in the supplementary material.
- **Join the community.** Papers With Code, Reddit r/MachineLearning, and Twitter/X are where implementation details are discussed.

## References

- [annotated_deep_learning_paper_implementations](https://github.com/labmlai/annotated_deep_learning_paper_implementations) -- Line-by-line annotated implementations (65,800+ stars)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) -- Original Transformer paper
- [Deep Residual Learning](https://arxiv.org/abs/1512.03385) -- ResNet paper
- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) -- Jay Alammar's visual guide
- [Papers With Code](https://paperswithcode.com/) -- Paper-implementation pairs with benchmarks

版本历史

  1. v2026.07 GitHub ↗
    原版更新说明

    ## Highlights - **First tagged release.** Every gate is green: `make check` (validate + 183 unit tests + eval-harness + numeric benchmarks), validate-catalog, quality-evals, and OpenSSF Scorecard on `main`. - **Rigor double loop at 13 method families.** Each family closes taxonomy tag → eval scenario (does the agent avoid the method's known traps?) → deterministic numeric benchmark (does it recover the true effect?). New this cycle: heterogeneous treatment effects (CATE) and quantile treatment effects (QTE). - **Paper-WorkFlow flagship at 29/29 executable rigor gates**, now merged to its `main` with a drift-gated README badge — the credibility artifact is checked by CI, so it can never silently go stale ([RIGOR.md](https://github.com/brycewang-stanford/Paper-WorkFlow/blob/main/RIGOR.md)). - **1,150 vendored skills / 69 collections**, all license-audited, security-scanned, tracked in a machine-readable catalog with CI-enforced freshness, and importable as a single whole-repo router skill. - **Hardened weekly upstream syncs**: synced trees must pass the full gate before merge; this cycle both sync PRs were verified, patched (StatsPAI's SkillOpt execution-gate card restored), and merged. - **First-party plugin marketplace** (Python / R / Stata empirical-analysis plugins + guide) shipped in the previous cycle, now under a tagged version. ## Known Follow-Ups - Ecosystem benchmark comparison (AERS vs econometrics agents on the shared benchmark suite) — [PLAN-2026-07 week 3](https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills/blob/main/docs/PLAN-2026-07.md). - Decomposition (Oaxaca–Blinder) and shift-share (Bartik) method families (weeks 3–4). - GitHub Pages catalog site, README demo screenshots, five-language README consistency script (week 4). ## Catalog - Top-level collections: **69** - `SKILL.md` files: **1150** - Cataloged agent tools: **335** ([`tools/CATALOG.md`](../tools/CATALOG.md)) ## Methodological rigor - Method families in the coverage map: **13** ([`RIGOR_COVERAGE.md`](RIGOR_COVERAGE.md)) - Eval scenarios: **26** (138 rubric items, 122 auto-checkable) ([`../eval-harness/`](../eval-harness/README.md)) - Benchmark tasks: **13** (70 gold checks, 58 required) ([`../benchmark/`](../benchmark/README.md)) ### Method families (skills tagged) - `iv`: 27 - `rdd`: 25 - `panel-fe`: 19 - `staggered-did`: 17 - `bayesian`: 13 - `did`: 10 - `event-study`: 10 - `matching`: 10 - `synthetic-control`: 10 - `cate`: 8 - `dml`: 8 - `survival`: 3 - `quantile`: 2 ### Eval scenarios by category - `causal-identification`: 14 - `citation-hygiene`: 1 - `reproducibility`: 2 - `research-integrity`: 1 - `runtime-safety`: 1 - `statistical-validity`: 2 - `writing-compliance`: 3 - `writing-style`: 2 ## License buckets - AGPL-3.0: 1 - Apache-2.0: 1 - CC-BY-NC-4.0 (non-commercial): 1 - CC-BY-SA-4.0: 1 - CC-BY-SA-4.0 (repository default): 4 - GPL-3.0: 1 - MIT: 35 - MIT Non-Commercial: 1 - Mixed (per-skill; see folder): 2 - UNKNOWN - check upstream: 22 ## Quality gates - `make check` — catalog/provenance/audit/eval freshness, repo hygiene, Paper-WorkFlow gates, stdlib unit tests, eval-harness lint + graded smoke, and the strict numeric benchmark (reference-gold drift fails the build). - External links and OpenSSF Scorecard run on their scheduled workflows.

当前收录版本

0.0.1

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值95/100
评分依据

提供深度学习论文的代码走读与实现指南,面向 AI/ML 学习者、研究者和开发者,具有明确的高频刚需价值

触发精准95/100
评分依据

触发词精准完整,覆盖 'deep learning', 'neural network', 'Transformer', 'CNN', 'NLP', 'computer vision' 等多种变体

结构规范95/100
评分依据

文档结构清晰,包含多个逻辑章节和端到端代码示例

执行可靠95/100
评分依据

所有代码示例均为标准 PyTorch 实现,工具来源明确

证据可验 (前四项均值)95/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。

文档引用的域名或地址

这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。

  • arxiv.org
  • github.com
  • jalammar.github.io
  • paperswithcode.com

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。