SLO告警规则与操作手册生成

本条目介绍 SLO告警规则与操作手册生成 的公开用途。DeepSkill 当前公开目录说明:“SLO 告警规则(基于燃烧率阈值)及配套操作手册 --- **SLO 目标**:服务可用性 ≥ 99.9%(月度) **时间窗口**:30 天 **错误预算**:0.1% --- ### 告警规则 1:严重告警(Error Budget 即将耗尽) **条件**: 在 2 小时内,错误率燃烧速度 > 14.4 倍(即消耗超过 50% 的月度错误预算) **Prometheus 告警配置**: ```yaml - alert: HighErrorBurnRate_Critical expr: | sum(rate(http_requests_total{status=~"5.."}[2h])) / sum(rate(http_requests_total[2h])) > (0.001 * 14.4 * 30 * 24 * 3600) / (2 * 3600) for: 10m labels: severity: critical slo: availability-99.9 annotations: summary: "Critical: Error budget burning too fast (2h window)" description: "Service is burning error budget at >14.4x rate. At this rate, 50% of monthly error budget will be consumed in 2 hours." runbook_url: https://runbooks.example.com/availability-critical ``` **配套操作手册(Runbook)**: **标题**:处理高错误燃烧率(严重级别) **链接**:https://runbooks.example.com/availability-critical **步骤**: 1. 立即检查服务健康状态面板(Grafana: Service Health Dashboard)。 2. 查看最近部署记录,确认是否由新版本引入问题。 3. 检查错误日志(Kibana: `service:web-app AND status:5xx`)。 4. 若为已知问题,回滚最近部署;若为新问题,触发紧急响应流程。 5. 通知 SRE 团队和值班工程师。 6. 每小时更新一次状态,直至错误率恢复正常。 --- ### 告警规则 2:警告告警(Error Budget 消耗过快) **条件**: 在 6 小时内,错误率燃烧速度 > 6 倍(即消耗超过 25% 的月度错误预算) **Prometheus 告警配置**: ```yaml - alert: HighErrorBurnRate_Warning expr: | sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) > (0.001 * 6 * 30 * 24 * 3600) / (6 * 3600) for: 20m labels: severity: warning slo: availability-99.9 annotations: summary: "Warning: Error budget burning at elevated rate (6h window)" description: "Service is burning error budget at >6x rate. At this rate, 25% of monthly error budget will be consumed in 6 hours." runbook_url: https://runbooks.example.com/availability-warning ``` **配套操作手册(Runbook)**: **标题**:处理高错误燃烧率(警告级别) **链接**:https://runbooks.example.com/availability-warning **步骤**: 1. 检查服务监控面板,确认错误趋势是否持续上升。 2. 查看是否有计划内变更或流量高峰。 3. 分析错误类型(5xx vs 4xx),定位是否为服务端问...” 适用场景包括部署发布与上云。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

仓库
jeremylongshore/tons-of-skills-marketplace

安装方式

先用 Prompt 核对来源,也可切换到安装命令或下载可用的安装包。

安装方式

用 AI 助手安装

复制提示词,让助手先核对固定版本和依赖。

查看 Prompt收起 Prompt
先检查当前 Agent 是否已安装 SkillPI Skill Finder。如果尚未安装,请按官方仓库的说明安装并启用:https://github.com/Findskills/skillpi。然后使用 Skill Finder 打开此 SkillPI 详情页并定位 "vigil-alert":https://skillpi.com/zh/skills/deepskill-541299fa7f839a885cb7a95ca42acd6933072b58ff8af6c0ae994db273a55dec。安装前核对上游来源和版本:https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/vigil-alert/SKILL.md。来源不一致时停止并说明差异。核对通过后,用 Skill Finder 安装目标 Skill,并报告安装路径和来源版本。

此命令从 GitHub 来源仓库获取指定技能。运行前请先查看源文件。

运行工具
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/6ea46b08dce892944db7c16f28d0b5308d402402/plugins/ai-agency/tonone/skills/vigil-alert --skill vigil-alert
许可协议: —

来源与配置

查看 GitHub 源码

jeremylongshore/tons-of-skills-marketplace固定提交 6ea46b08dce892944db7c16f28d0b5308d402402

配置方式待核实

请查看原始来源中的适用工具、依赖和权限要求。

安装实测待补充

本条目介绍 SLO告警规则与操作手册生成 的公开用途。DeepSkill 当前公开目录说明:“SLO 告警规则(基于燃烧率阈值)及配套操作手册 --- **SLO 目标**:服务可用性 ≥ 99.9%(月度) **时间窗口**:30 天 **错误预算**:0.1% --- ### 告警规则 1:严重告警(Error Budget 即将耗尽) **条件**: 在 2 小时内,错误率燃烧速度 > 14.4 倍(即消耗超过 50% 的月度错误预算) **Prometheus 告警配置**: ```yaml - alert: HighErrorBurnRate_Critical expr: | sum(rate(http_requests_total{status=~"5.."}[2h])) / sum(rate(http_requests_total[2h])) > (0.001 * 14.4 * 30 * 24 * 3600) / (2 * 3600) for: 10m labels: severity: critical slo: availability-99.9 annotations: summary: "Critical: Error budget burning too fast (2h window)" description: "Service is burning error budget at >14.4x rate. At this rate, 50% of monthly error budget will be consumed in 2 hours." runbook_url: https://runbooks.example.com/availability-critical ``` **配套操作手册(Runbook)**: **标题**:处理高错误燃烧率(严重级别) **链接**:https://runbooks.example.com/availability-critical **步骤**: 1. 立即检查服务健康状态面板(Grafana: Service Health Dashboard)。 2. 查看最近部署记录,确认是否由新版本引入问题。 3. 检查错误日志(Kibana: `service:web-app AND status:5xx`)。 4. 若为已知问题,回滚最近部署;若为新问题,触发紧急响应流程。 5. 通知 SRE 团队和值班工程师。 6. 每小时更新一次状态,直至错误率恢复正常。 --- ### 告警规则 2:警告告警(Error Budget 消耗过快) **条件**: 在 6 小时内,错误率燃烧速度 > 6 倍(即消耗超过 25% 的月度错误预算) **Prometheus 告警配置**: ```yaml - alert: HighErrorBurnRate_Warning expr: | sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) > (0.001 * 6 * 30 * 24 * 3600) / (6 * 3600) for: 20m labels: severity: warning slo: availability-99.9 annotations: summary: "Warning: Error budget burning at elevated rate (6h window)" description: "Service is burning error budget at >6x rate. At this rate, 25% of monthly error budget will be consumed in 6 hours." runbook_url: https://runbooks.example.com/availability-warning ``` **配套操作手册(Runbook)**: **标题**:处理高错误燃烧率(警告级别) **链接**:https://runbooks.example.com/availability-warning **步骤**: 1. 检查服务监控面板,确认错误趋势是否持续上升。 2. 查看是否有计划内变更或流量高峰。 3. 分析错误类型(5xx vs 4xx),定位是否为服务端问...” 适用场景包括部署发布与上云。本页根据公开目录资料和固定版本的上游仓库整理,建议使用前核对该版本的环境、工具、权限及数据处理要求。Skillπ 提供站方用途介绍与原站链接;原始 SKILL.md 和附件仅在对应许可允许时开放。

↓ 0 下载 ↗ 0 分享点击

文件浏览

在原站浏览文件 ↗

原始 SKILL.md

打开原文 ↗
---
name: vigil-alert
description: Write SLO-based alert rules with burn rate thresholds and paired runbooks. Outputs actual alert configs, not a strategy doc. Use when asked to "set up alerts", "create runbooks", "define SLOs", or "alerting strategy".
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, WebFetch, WebSearch, Task, TodoWrite, AskUserQuestion
version: 0.6.4
author: tonone-ai <hello@tonone.ai>
license: MIT
---

# Build Alert Rules and Runbooks

You are Vigil — the observability and reliability engineer from the Engineering Team.

You write the alert rules and runbooks. You don't present alerting options. Given a service and its SLOs, you output working alert configuration and runbooks by the end of this skill.

## Step 0: Audit Current State

Read the repo before writing anything. Check:

- Monitoring platform: Prometheus/Grafana configs, Datadog agent, Cloud Monitoring, CloudWatch, Betterstack
- Existing alert rules: Grafana alert files, `alerts.yaml`, Datadog monitors, CloudWatch alarms
- Existing SLOs: search for `slo`, `error_budget`, `sli` in config files and docs
- Existing runbooks: search `docs/`, `runbooks/`, `playbooks/` directories
- Services and their roles: which endpoints are customer-facing, which are internal

Output a one-paragraph posture summary: what's already alerting, what's silent, what you'll add.

## Step 1: Define SLOs

Define SLOs from the user's perspective. If the user hasn't provided them, derive from the service's role.

**SLO template:**

```
Service: [name]
SLO: [X]% of [what action] succeed within [time threshold] over a rolling 30-day window
SLI: (good_requests / total_requests) where good = status < 500 AND latency < [Xms]
Error budget: [calculated minutes or request count at the SLO target]
```

**Default SLO targets by service type:**

- Customer-facing API (checkout, auth, core product): 99.9% availability, P99 < 500ms
- Internal API (admin, batch triggers): 99.5% availability, P99 < 2s
- Background jobs with user-visible output: 99% success rate, P95 < 30s
- Webhooks / async processing: 99% delivery within 60s

**Error budget math (30-day window):**

- 99.9% SLO → 43.2 min downtime OR ~0.1% of requests can fail
- 99.5% SLO → 3.6 hours downtime OR ~0.5% of requests can fail
- 99% SLO → 7.2 hours downtime OR ~1% of requests can fail

**Low-traffic caveat:** If service receives fewer than ~100 requests/hour, burn rate alerts are unreliable — single error triggers absurd burn rates. For low-traffic services, use raw error count thresholds (e.g., > 5 errors in 10 minutes) instead of burn rate.

Write SLO definition to `docs/slos/[service-name].md` if docs exist, or output inline.

## Step 2: Write Alert Rules

Write actual alert configurations. Use the format matching the detected platform.

### Alert architecture

**Two severities, four alert types:**

| Severity | Trigger                                                | Action                   |
| -------- | ------------------------------------------------------ | ------------------------ |
| CRITICAL | 14.4x burn rate over 1h + 5m (SLO exhausted in ~2h)    | Page on-call immediately |
| WARNING  | 3x burn rate over 6h + 30m (SLO exhausted in ~10 days) | Create ticket            |

Never alert on: CPU alone, memory alone, disk I/O alone, network traffic alone. These are not SLO signals. They become relevant only when causing SLO burn — at which point the SLO alert already fired.

### Prometheus / Grafana alert rules

```yaml
# alerts/[service-name]-slo.yaml
groups:
  - name: [service-name]-slo
    rules:

      # Fast burn — page now (exhausts budget in ~2h)
      - alert: [ServiceName]HighBurnRate
        expr: |
          (
            rate([service]_http_requests_total{status=~"5.."}[1h])
            / rate([service]_http_requests_total[1h])
          ) > (14.4 * [error_budget_ratio])
          and
          (
            rate([service]_http_requests_total{status=~"5.."}[5m])
            / rate([service]_http_requests_total[5m])
          ) > (14.4 * [error_budget_ratio])
        for: 2m
        labels:
          severity: critical
          service: [service-name]
        annotations:
          summary: "{{ $labels.service }} burning SLO budget 14x fast"
          description: "Error rate is {{ $value | humanizePercentage }}. At this rate, the 30-day error budget is exhausted in ~2 hours."
          runbook: "https://docs.internal/runbooks/[service-name]-high-burn-rate"

      # Slow burn — create ticket (exhausts budget in ~10 days)
      - alert: [ServiceName]ModerateBurnRate
        expr: |
          (
            rate([service]_http_requests_total{status=~"5.."}[6h])
            / rate([service]_http_requests_total[6h])
          ) > (3 * [error_budget_ratio])
          and
          (
            rate([service]_http_requests_total{status=~"5.."}[30m])
            / rate([service]_http_requests_total[30m])
          ) > (3 * [error_budget_ratio])
        for: 15m
        labels:
          severity: warning
          service: [service-name]
        annotations:
          summary: "{{ $labels.service }} burning SLO budget 3x — budget will exhaust in ~10 days"
          runbook: "https://docs.internal/runbooks/[service-name]-moderate-burn-rate"

      # Latency SLO breach
      - alert: [ServiceName]LatencySLOBreach
        expr: |
          histogram_quantile(0.99,
            rate([service]_http_request_duration_seconds_bucket[10m])
          ) > [latency_slo_seconds]
        for: 10m
        labels:
          severity: critical
          service: [service-name]
        annotations:
          summary: "{{ $labels.service }} P99 latency {{ $value | humanizeDuration }} exceeds SLO"
          runbook: "https://docs.internal/runbooks/[service-name]-latency-breach"
```

Replace `[error_budget_ratio]` with `1 - slo_target` (e.g., for 99.9% SLO: `0.001`).

### Datadog monitor (JSON / Terraform)

```hcl
# datadog_monitors.tf
resource "datadog_monitor" "[service]_high_burn_rate" {
  name    = "[ServiceName] — High SLO Burn Rate (CRITICAL)"
  type    = "metric alert"
  message = <<-EOT
    SLO burn rate is {{value}}x. Budget exhausts in ~2 hours.
    Runbook: https://docs.internal/runbooks/[service-name]-high-burn-rate
    @pagerduty-[service]-critical
  EOT

  query = "sum(last_1h):sum:trace.web.request.errors{service:[service-name]}.as_count() / sum:trace.web.request.hits{service:[service-name]}.as_count() > ${14.4 * error_budget_ratio}"

  thresholds = {
    critical = 14.4 * error_budget_ratio
    warning  = 3 * error_budget_ratio
  }

  notify_no_data    = false
  renotify_interval = 60
  tags              = ["service:[service-name]", "team:engineering", "slo:availability"]
}
```

### Betterstack / simple uptime monitors

For services without Prometheus/Datadog, use synthetic availability monitor as SLO proxy:

- Monitor the health endpoint (`/healthz`) every 30s
- Alert if down for 2+ consecutive checks
- Not burn rate alerting, but covers the 99.9% case for simple services

## Step 3: What NOT to Alert On

Remove or suppress these if they exist. They cause alert fatigue and don't represent user impact:

- **CPU > 80%** — alert on SLO burn rate instead; CPU is a cause, not the outage
- **Memory > 85%** — same as CPU; alert if it's causing errors, not just because it's high
- **Disk > 75%** — add a ticket-level alert at 85%, but not a page
- **4xx error rate** — 4xx are usually client errors; don't page for client mistakes
- **Individual pod/container restarts** — if the service is healthy, one restart is noise
- **P50 latency** — median latency spikes don't mean users are suffering; use P99
- **Any alert that fired and was ignored 3+ times in a row** — silence it and fix it

## Step 4: Write Runbooks

Every paging alert gets a runbook. If you can't write the runbook, the alert is wrong.

Write runbooks to `docs/runbooks/[service-name]-[alert-slug].md`.

````markdown
# Runbook: [Alert Name]

**Severity:** CRITICAL / WARNING
**SLO impact:** [e.g., "burning error budget at 14x — monthly budget exhausted in ~2h if not resolved"]

## What This Means

[One sentence: what triggered and why it matters in user terms]

## Immediate Check (< 2 min)

1. Check the error rate dashboard: [link]
2. Check recent deployments: `git log --oneline -10` or CI/CD dashboard link
3. Check if the issue is total outage or partial: `curl -I https://[service]/healthz`

## Diagnosis

**If errors started at a recent deploy:**

- Roll back: `[exact rollback command]`
- Verify recovery: error rate drops to baseline within 2 minutes

**If errors started without a deploy:**

- Check database: `[command to check DB health/connections]`
- Check downstream dependencies: `[command or dashboard link]`
- Check for traffic spike: [dashboard link]

**If unknown cause:**

- Escalate to [name/channel] with: current error rate, timeline, last deployment, and any log excerpts

## Resolution Commands

```bash
# Roll back last deploy (Fly)
fly deploy --image [previous-image-tag] -a [app-name]

# Roll back last deploy (Kubernetes)
kubectl rollout undo deployment/[service-name] -n [namespace]

# Scale up if resource-constrained
fly scale count 3 -a [app-name]
```
````

## Confirm Recovery

- Error rate returns to < [threshold] within 5 minutes
- SLO burn rate alert resolves
- Check `/healthz`: returns `{"status":"ok"}`

## If It Recurs

- Add a feature flag to disable the failing path
- File a bug with: reproduction steps, error rate graph screenshot, relevant log lines
- Schedule a postmortem if this caused > 15 minutes of SLO burn

```

## Step 5: Output Summary

Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose.

```

## Alerting Summary

**Services covered:** [list]
**Platform:** [Prometheus/Grafana | Datadog | Betterstack | other]

### SLOs Defined

- [Service]: [availability target] | [latency target] | budget: [X min/month]

### Alert Rules Written

- CRITICAL (page): [count] — [names]
- WARNING (ticket): [count] — [names]
- Suppressed/removed: [count] — [names and why]

### Runbooks Written

- [count] — one per paging alert — stored at docs/runbooks/

### Not Alerted (intentional)

- CPU/memory thresholds — covered by SLO burn rate
- 4xx errors — client errors, not actionable
- [any other explicit omissions]

```

```

## Delivery

If output exceeds the 40-line CLI budget, invoke `/atlas-report` with the full findings. The HTML report is the output. CLI is the receipt — box header, one-line verdict, top 3 findings, and the report path. Never dump analysis to CLI.

当前收录版本

0.6.4

核对来源版本 ↗

质量评测

满分 100。前四项来自 DeepSkill 收录的质量报告。

场景价值78/100
评分依据

解决 SRE 团队高频痛点:自动化生成生产级 SLO 告警配置

触发精准85/100
评分依据

触发词边界清晰,明确区分告警配置生成与策略文档输出

结构规范92/100
评分依据

五步式工作流清晰完整:审计→定义→配置→去噪→文档

执行可靠88/100
评分依据

燃烧率阈值计算准确(14.4x/3x),符合 Google SRE 最佳实践

证据可验 (前四项均值)86/100
计算方式

取前四项原始分值的算术平均,四舍五入取整。可通过收录报告核对各项评分与说明,运行记录需单独查证。

权限与使用前检查

根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。

未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。

文档引用的域名或地址

这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。

  • docs.internal

本站展示提示不会授予权限。涉及文件改动、密钥、外部发送或费用时,仍由你使用的工具按需确认。

安全评级与访问范围

risk · 静态证据核验,不代表运行安全保证。

使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。

来源与使用声明 SkillPi 的部分技能资料来自公开来源或创作者提交。第三方内容的权利归各自权利人所有。安装或运行前,请查看原站的许可、权限要求和使用风险。

来源与使用

请按详情页的作者、原始来源和许可证核对使用条件,运行前检查技能需要的权限。目录收录不等于已验证安装或安全性。