【系统架构设计百科】故障排查方法论：从告警到根因的系统化路径

# Prometheus 告警规则示例：监控 MTTD
groups:
  - name: incident_metrics
    rules:
      - record: sre:incident:mttd_seconds
        expr: |
          avg(
            incident_detected_timestamp - incident_started_timestamp
          ) by (service, severity)
      - alert: HighMTTD
        expr: sre:incident:mttd_seconds{severity="P1"} > 300
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "P1 事故平均检测时间超过 5 分钟"
          description: "服务 {{ $labels.service }} 的 MTTD 为 {{ $value }}s"

# 事故升级规则示例
escalation_rules:
  - from: P2
    to: P1
    conditions:
      - "影响用户数超过总用户数的 10%"
      - "故障持续时间超过 30 分钟且无缓解方案"
      - "涉及数据丢失或数据不一致"
  - from: P1
    to: P0
    conditions:
      - "所有核心功能不可用"
      - "故障持续时间超过 15 分钟且无缓解方案"
      - "涉及用户数据泄露"

#!/bin/bash
# incident-check.sh：事故排查第一步——检查近期变更
# 用法：./incident-check.sh <service-name> <hours-lookback>

SERVICE=$1
HOURS=${2:-6}

echo "=== 近 ${HOURS} 小时的变更记录 ==="

echo ""
echo "--- 代码部署 ---"
kubectl rollout history deployment/${SERVICE} -n production | tail -10

echo ""
echo "--- 配置变更 ---"
# 假设使用 Apollo 配置中心
curl -s "http://apollo-portal/api/changes?app=${SERVICE}&hours=${HOURS}" | jq '.[] | {time: .changeTime, key: .key, oldValue: .oldValue, newValue: .newValue}'

echo ""
echo "--- 基础设施变更 ---"
kubectl get events -n production --field-selector involvedObject.name=${SERVICE} --sort-by='.lastTimestamp' | tail -20

echo ""
echo "--- HPA 事件 ---"
kubectl describe hpa ${SERVICE} -n production | grep -A 5 "Events:"

echo ""
echo "--- 最近的 CronJob 执行 ---"
kubectl get jobs -n production -l app=${SERVICE} --sort-by='.status.startTime' | tail -5

## 事故排查记录 - INC-2024-0312

### 03:17 告警触发
- 现象：订单服务 P99 延迟 > 2s
- 影响：约 30% 下单请求超时

### 03:25 假设 1：数据库慢查询
- 验证方法：查看 RDS 慢查询日志
- 结果：无慢查询（排除）

### 03:35 假设 2：近期代码部署
- 验证方法：检查 CI/CD 部署记录
- 结果：最近一次部署在 48 小时前（排除）

### 03:50 假设 3：上游服务异常
- 验证方法：检查上游服务的健康状态和延迟
- 结果：上游服务正常（排除）

### 04:10 假设 4：配置变更
- 验证方法：检查配置中心变更记录
- 结果：昨天 15:30 有人将下游超时从 500ms 改为 5000ms
- 关联分析：超时改大导致线程池被慢请求占满
- 结论：**根因确认**

### 04:15 缓解措施
- 操作：将超时参数回滚为 500ms
- 结果：延迟在 2 分钟内恢复正常

# Runbook：订单服务延迟飙升

## 元信息
- 服务：order-service
- 所有者：交易团队
- 最后更新：2024-03-12
- 关联告警：OrderServiceHighLatency、OrderServiceP99Above2s

## 症状
- 订单服务 P99 延迟超过 SLO（200ms）
- 可能伴随下单成功率下降

## 影响评估
- 直接影响：用户下单体验降级
- 间接影响：交易额下降
- 影响范围评估命令：

  ```bash
  # 查看受影响请求占比
  curl -s 'http://prometheus:9090/api/v1/query?query=
    sum(rate(http_request_duration_seconds_count{service="order-service",le="0.2"}[5m]))
    /
    sum(rate(http_request_duration_seconds_count{service="order-service"}[5m]))
  ' | jq '.data.result[0].value[1]'

kubectl rollout undo deployment/order-service -n production

# 通过 Apollo API 回滚配置
curl -X PUT "http://apollo-portal/api/rollback?app=order-service&env=PRO&cluster=default&namespace=application"

# 通过 Sentinel 控制台设置限流规则
curl -X POST "http://sentinel-dashboard/api/flow" \
  -H "Content-Type: application/json" \
  -d '{"resource": "/api/order/create", "limitApp": "default", "grade": 1, "count": 500}'

# Postmortem：订单服务延迟飙升事故

## 元信息
- 事故编号：INC-2024-0312
- 事故级别：P1
- 事故日期：2024-03-12
- 持续时间：2 小时 3 分钟（03:17 - 05:20）
- 作者：张三
- 状态：初稿 / 评审中 / 已完成
- 评审人：李四、王五

## 摘要
2024 年 3 月 12 日 03:17，订单服务 P99 延迟从正常的 80ms 飙升至 2s 以上，
约 30% 的下单请求超时失败。事故持续 2 小时 3 分钟，影响约 12,000 笔订单。
根因是配置中心的超时参数被从 500ms 误改为 5000ms，导致线程池被慢请求占满。

## 影响
- 用户影响：约 30% 的下单请求失败，预估影响订单 12,000 笔
- 收入影响：预估损失约 ¥180 万
- 客户投诉：收到 47 条客户投诉工单
- SLA 影响：月度 SLA 从 99.95% 降至 99.87%

## 时间线
| 时间（UTC+8） | 事件 |
|----------------|------|
| 03:17 | PagerDuty 触发告警：订单服务 P99 > 2s |
| 03:19 | oncall 工程师张三确认告警 |
| 03:25 | 检查数据库慢查询日志，无异常 |
| 03:35 | 检查 CI/CD 部署记录，最近部署在 48 小时前 |
| 03:50 | 检查上游服务健康状态，均正常 |
| 04:10 | 检查配置中心变更记录，发现超时参数异常变更 |
| 04:15 | 回滚超时参数为 500ms |
| 04:17 | 延迟开始恢复 |
| 04:25 | 延迟完全恢复正常，确认事故缓解 |
| 05:20 | 完成受影响订单的补偿处理，事故关闭 |

## 根因分析
3 月 11 日 15:30，工程师 A 在修改另一个配置项时，误将订单服务对支付服务
的超时参数从 500ms 改为 5000ms。配置在 16:00 灰度发布到 10% 的实例，
未触发告警（因为延迟仅轻微上升）。22:00 配置全量发布后，叠加夜间批处理
任务对支付服务的压力，部分请求开始命中 5 秒超时，线程池逐渐被占满。

核心问题不在于工程师误改了配置，而在于：
1. 配置中心的变更没有强制关联到变更原因和审批流程
2. 配置的灰度发布没有与业务指标联动——灰度期间延迟上升未触发告警
3. 超时参数缺乏合理范围校验——5000ms 明显不合理但系统没有拦截

## 经验教训
### 做得好的
- oncall 工程师在 2 分钟内确认了告警
- 排查过程有结构化记录，便于事后分析
- 回滚操作执行迅速，缓解时间仅 2 分钟

### 需要改进的
- 排查方向的优先级不对：应该先查变更再查数据库
- 没有第一时间建立 War Room，前期排查依赖单人
- 配置变更的审批和校验机制缺失

## 改进措施
| 编号 | 措施 | 优先级 | 负责人 | 截止日期 | 状态 |
|------|------|--------|--------|----------|------|
| A1 | 配置变更增加强制 Review 和审批流程 | P0 | 李四 | 2024-03-26 | 进行中 |
| A2 | 为关键配置项增加合理范围校验 | P0 | 王五 | 2024-03-26 | 待开始 |
| A3 | 配置灰度发布与业务指标告警联动 | P1 | 赵六 | 2024-04-09 | 待开始 |
| A4 | 更新 Runbook：将"检查变更"调整为排查第一步 | P1 | 张三 | 2024-03-19 | 待开始 |
| A5 | 季度演练覆盖"配置变更导致故障"场景 | P2 | 张三 | 2024-06-30 | 待开始 |

#!/usr/bin/env python3
"""
postmortem_tracker.py：从 Postmortem 文档中提取 Action Items 并同步到 Jira
"""

import re
import json
from pathlib import Path
from datetime import datetime, timedelta

def parse_action_items(postmortem_path: str) -> list[dict]:
    """从 Postmortem Markdown 文件中解析 Action Items 表格"""
    content = Path(postmortem_path).read_text(encoding="utf-8")
    
    # 匹配 "改进措施" 部分的表格行
    table_pattern = re.compile(
        r'\|\s*(A\d+)\s*\|\s*(.+?)\s*\|\s*(P\d)\s*\|\s*(.+?)\s*\|\s*(\d{4}-\d{2}-\d{2})\s*\|\s*(.+?)\s*\|'
    )
    
    items = []
    for match in table_pattern.finditer(content):
        items.append({
            "id": match.group(1),
            "description": match.group(2).strip(),
            "priority": match.group(3),
            "assignee": match.group(4).strip(),
            "due_date": match.group(5),
            "status": match.group(6).strip(),
        })
    return items


def check_overdue(items: list[dict]) -> list[dict]:
    """检查逾期未完成的 Action Items"""
    today = datetime.now().date()
    overdue = []
    for item in items:
        due = datetime.strptime(item["due_date"], "%Y-%m-%d").date()
        if due < today and item["status"] not in ("已完成", "done"):
            item["overdue_days"] = (today - due).days
            overdue.append(item)
    return overdue


def sync_to_jira(items: list[dict], project_key: str):
    """将 Action Items 同步到 Jira（示意代码）"""
    for item in items:
        jira_payload = {
            "fields": {
                "project": {"key": project_key},
                "summary": f"[Postmortem] {item['description']}",
                "issuetype": {"name": "Task"},
                "priority": {"name": _map_priority(item["priority"])},
                "assignee": {"name": item["assignee"]},
                "duedate": item["due_date"],
                "labels": ["postmortem", "incident-followup"],
            }
        }
        print(f"  创建 Jira Ticket: {json.dumps(jira_payload, ensure_ascii=False, indent=2)}")


def _map_priority(p: str) -> str:
    return {"P0": "Highest", "P1": "High", "P2": "Medium", "P3": "Low"}.get(p, "Medium")


if __name__ == "__main__":
    import sys
    if len(sys.argv) < 2:
        print("用法: python postmortem_tracker.py <postmortem.md>")
        sys.exit(1)
    
    items = parse_action_items(sys.argv[1])
    print(f"发现 {len(items)} 个 Action Items")
    
    overdue = check_overdue(items)
    if overdue:
        print(f"\n逾期未完成的 Action Items ({len(overdue)} 个):")
        for item in overdue:
            print(f"  - {item['id']}: {item['description']} (逾期 {item['overdue_days']} 天)")

-- Postmortem 聚合分析查询示例（假设数据存储在关系数据库中）

-- 按根因类别统计故障分布
SELECT
    root_cause_category,
    COUNT(*) AS incident_count,
    ROUND(AVG(time_to_detect_minutes), 1) AS avg_mttd,
    ROUND(AVG(time_to_mitigate_minutes), 1) AS avg_ttm,
    ROUND(AVG(time_to_resolve_minutes), 1) AS avg_mttr
FROM postmortems
WHERE created_at >= DATE_SUB(NOW(), INTERVAL 12 MONTH)
GROUP BY root_cause_category
ORDER BY incident_count DESC;

-- 改进措施完成率
SELECT
    YEAR(due_date) AS year,
    MONTH(due_date) AS month,
    COUNT(*) AS total_actions,
    SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS completed,
    ROUND(
        SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) * 100.0 / COUNT(*),
        1
    ) AS completion_rate_pct
FROM postmortem_actions
GROUP BY YEAR(due_date), MONTH(due_date)
ORDER BY year, month;

# oncall 轮值配置示例（PagerDuty 风格）
schedule:
  name: "order-service-oncall"
  timezone: "Asia/Shanghai"
  layers:
    - name: "primary"
      rotation_type: "weekly"
      start: "2024-01-01T10:00:00+08:00"
      users:
        - "engineer-a"
        - "engineer-b"
        - "engineer-c"
        - "engineer-d"
      restrictions:
        - type: "daily"
          start_time: "00:00"
          end_time: "24:00"
    - name: "secondary"
      rotation_type: "weekly"
      start: "2024-01-08T10:00:00+08:00"
      users:
        - "engineer-b"
        - "engineer-c"
        - "engineer-d"
        - "engineer-a"

  escalation_policy:
    name: "order-service-escalation"
    rules:
      - targets:
          - type: "schedule"
            id: "primary"
        escalation_delay_minutes: 5
      - targets:
          - type: "schedule"
            id: "secondary"
        escalation_delay_minutes: 10
      - targets:
          - type: "user"
            id: "team-lead"
        escalation_delay_minutes: 15

# oncall 健康度 Dashboard 配置（Grafana 风格）
dashboard:
  title: "Oncall Health Dashboard"
  panels:
    - title: "每周告警数量趋势"
      type: timeseries
      query: |
        sum(increase(alertmanager_alerts_received_total[7d])) by (team)
      description: "告警数量应当随着系统改进逐步减少"

    - title: "夜间告警比例"
      type: gauge
      query: |
        sum(alertmanager_alerts_received_total{hour=~"22|23|0|1|2|3|4|5|6"})
        /
        sum(alertmanager_alerts_received_total)
      thresholds:
        - value: 0.2
          color: green
        - value: 0.4
          color: yellow
        - value: 0.6
          color: red
      description: "夜间告警比例过高说明告警策略需要优化"

    - title: "告警到确认的时间分布"
      type: histogram
      query: |
        histogram_quantile(0.5, sum(rate(incident_ack_duration_seconds_bucket[30d])) by (le))

    - title: "每人每周被呼叫次数"
      type: table
      query: |
        sum(increase(pagerduty_notifications_total[7d])) by (user)
      description: "每人每周不应超过 2 次夜间呼叫"

    - title: "Oncall 期间工单处理量"
      type: bargauge
      query: |
        sum(increase(oncall_tickets_resolved_total[7d])) by (oncall_user)

# Statuspage.io 事故更新模板
incident_templates:
  - name: "服务降级"
    status: "investigating"
    body: |
      我们正在调查 {{service_name}} 的性能问题。
      部分用户可能会遇到 {{user_impact}}。
      我们已经组建了应急团队，正在积极排查中。
      下次更新时间：{{next_update_time}}

  - name: "已识别问题"
    status: "identified"
    body: |
      我们已经定位到 {{service_name}} 问题的原因。
      根因是 {{root_cause_summary}}。
      我们正在实施修复方案，预计恢复时间为 {{eta}}。
      下次更新时间：{{next_update_time}}

  - name: "已实施修复"
    status: "monitoring"
    body: |
      我们已经实施了修复方案，{{service_name}} 正在恢复中。
      目前正在监控系统状态以确认问题已完全解决。
      下次更新时间：{{next_update_time}}

  - name: "事故关闭"
    status: "resolved"
    body: |
      {{service_name}} 已完全恢复正常。
      此次故障从 {{start_time}} 持续到 {{end_time}}，共计 {{duration}}。
      影响范围：{{impact_summary}}。
      我们将进行事后复盘以避免类似问题再次发生。
      对于此次故障给您带来的不便，我们深表歉意。

"""
incident_bot.py：Slack Incident Bot 核心逻辑（示意代码）
功能：
  - /incident create <title> <severity> - 创建事故
  - /incident update <status> - 更新事故状态
  - /incident resolve - 关闭事故
  - /incident timeline - 查看事故时间线
"""

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum


class Severity(Enum):
    P0 = "P0"
    P1 = "P1"
    P2 = "P2"
    P3 = "P3"
    P4 = "P4"


class IncidentStatus(Enum):
    INVESTIGATING = "investigating"
    IDENTIFIED = "identified"
    MITIGATING = "mitigating"
    MONITORING = "monitoring"
    RESOLVED = "resolved"


@dataclass
class TimelineEntry:
    timestamp: datetime
    author: str
    action: str
    detail: str


@dataclass
class Incident:
    id: str
    title: str
    severity: Severity
    status: IncidentStatus = IncidentStatus.INVESTIGATING
    created_at: datetime = field(default_factory=datetime.now)
    ic: str = ""
    communication_lead: str = ""
    operations_lead: str = ""
    timeline: list[TimelineEntry] = field(default_factory=list)
    war_room_channel: str = ""

    def create_war_room(self, slack_client) -> str:
        """创建专用的 War Room 频道"""
        channel_name = f"war-room-{self.id}"
        channel = slack_client.conversations_create(name=channel_name)
        self.war_room_channel = channel["channel"]["id"]

        # 发送初始消息
        slack_client.chat_postMessage(
            channel=self.war_room_channel,
            text=self._format_initial_message(),
        )
        return channel_name

    def add_timeline_entry(self, author: str, action: str, detail: str):
        entry = TimelineEntry(
            timestamp=datetime.now(),
            author=author,
            action=action,
            detail=detail,
        )
        self.timeline.append(entry)

    def update_status(self, new_status: IncidentStatus, updater: str):
        old_status = self.status
        self.status = new_status
        self.add_timeline_entry(
            author=updater,
            action="status_change",
            detail=f"{old_status.value} -> {new_status.value}",
        )

    def _format_initial_message(self) -> str:
        return (
            f"*事故 {self.id}*\n"
            f"标题：{self.title}\n"
            f"级别：{self.severity.value}\n"
            f"状态：{self.status.value}\n"
            f"创建时间：{self.created_at.strftime('%Y-%m-%d %H:%M:%S')}\n"
            f"---\n"
            f"IC：{self.ic or '待指定'}\n"
            f"通信负责人：{self.communication_lead or '待指定'}\n"
            f"操作负责人：{self.operations_lead or '待指定'}\n"
        )

    def format_timeline(self) -> str:
        lines = [f"*事故 {self.id} 时间线*\n"]
        for entry in self.timeline:
            lines.append(
                f"[{entry.timestamp.strftime('%H:%M:%S')}] "
                f"{entry.author}: {entry.action} - {entry.detail}"
            )
        return "\n".join(lines)

# Game Day 计划模板
game_day:
  date: "2024-04-15"
  time: "14:00-17:00"
  participants:
    - team: "交易团队"
      role: "被测试团队"
    - team: "SRE 团队"
      role: "故障注入方"
    - team: "管理层"
      role: "观察者"

  scenario:
    name: "支付服务级联故障"
    description: |
      模拟支付服务的第三方支付网关超时，
      验证订单服务的降级策略和团队的响应能力。
    
    injection_plan:
      - time: "T+0"
        action: "对支付网关的出站请求注入 5s 延迟"
        tool: "Chaos Mesh"
        config:
          kind: NetworkChaos
          spec:
            action: delay
            mode: all
            selector:
              labelSelectors:
                app: payment-service
            delay:
              latency: "5s"
              jitter: "1s"
            duration: "30m"

      - time: "T+15m"
        action: "如果团队未发现，升级为支付网关完全不可达"
        tool: "Chaos Mesh"
        config:
          kind: NetworkChaos
          spec:
            action: loss
            mode: all
            selector:
              labelSelectors:
                app: payment-service
            loss:
              loss: "100"
            duration: "15m"

  success_criteria:
    - "告警在 5 分钟内触发"
    - "团队在 10 分钟内确认并开始响应"
    - "降级策略在 15 分钟内生效（如：异步支付、排队处理）"
    - "客户在 20 分钟内收到状态通知"
    - "事故在 30 分钟内缓解"

  debrief:
    time: "17:00-18:00"
    agenda:
      - "回顾 Game Day 时间线"
      - "对比预期行为与实际行为"
      - "记录发现的问题和改进项"
      - "更新 Runbook 和告警规则"

# Chaos Mesh 实验示例：模拟数据库连接超时
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: db-connection-timeout
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      app: order-service
  delay:
    latency: "10s"
    correlation: "100"
    jitter: "0ms"
  direction: to
  target:
    selector:
      namespaces:
        - production
      labelSelectors:
        app: mysql
    mode: all
  duration: "5m"
  scheduler:
    cron: "@every 720h"

# Alertmanager 告警路由配置
route:
  receiver: default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # P0/P1 告警：立即通过 PagerDuty 呼叫
    - match_re:
        severity: P0|P1
      receiver: pagerduty-oncall
      group_wait: 0s
      repeat_interval: 15m

    # P2 告警：工作时间 Slack 通知
    - match:
        severity: P2
      receiver: slack-alerts
      active_time_intervals:
        - work-hours

    # P3/P4 告警：创建 Jira 工单
    - match_re:
        severity: P3|P4
      receiver: jira-ticket

receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: '<pagerduty-service-key>'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          service: '{{ .CommonLabels.service }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'

  - name: slack-alerts
    slack_configs:
      - api_url: '<slack-webhook-url>'
        channel: '#alerts'
        title: '[{{ .CommonLabels.severity }}] {{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

  - name: jira-ticket
    webhook_configs:
      - url: 'http://alertmanager-jira-bridge/api/create'

time_intervals:
  - name: work-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'

{
  "dashboard": {
    "title": "Order Service - Incident Dashboard",
    "panels": [
      {
        "title": "请求速率 (Rate)",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service='order-service'}[5m])) by (status_code)",
            "legendFormat": "{{status_code}}"
          }
        ]
      },
      {
        "title": "错误率 (Errors)",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service='order-service',status_code=~'5..'}[5m])) / sum(rate(http_requests_total{service='order-service'}[5m]))"
          }
        ],
        "thresholds": [
          {"value": 0.001, "color": "green"},
          {"value": 0.01, "color": "yellow"},
          {"value": 0.05, "color": "red"}
        ]
      },
      {
        "title": "延迟分布 (Duration)",
        "type": "timeseries",
        "targets": [
          {
            "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service='order-service'}[5m])) by (le))",
            "legendFormat": "P50"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service='order-service'}[5m])) by (le))",
            "legendFormat": "P99"
          },
          {
            "expr": "histogram_quantile(0.999, sum(rate(http_request_duration_seconds_bucket{service='order-service'}[5m])) by (le))",
            "legendFormat": "P999"
          }
        ]
      },
      {
        "title": "饱和度 - 线程池使用率",
        "type": "gauge",
        "targets": [
          {
            "expr": "jvm_threads_current{service='order-service'} / jvm_threads_peak{service='order-service'}"
          }
        ]
      },
      {
        "title": "最近变更事件",
        "type": "annotations",
        "datasource": "deployment-events",
        "description": "展示最近的代码部署、配置变更和基础设施变更"
      }
    ]
  }
}

# 基于 SLO 的告警规则（多窗口多燃烧率）
groups:
  - name: slo_alerts
    rules:
      # 快速燃烧告警：过去 1 小时内，错误预算消耗速度是正常的 14.4 倍
      # 如果这个速度持续，SLO 将在 1 天内耗尽
      - alert: OrderServiceHighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{service="order-service",status_code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{service="order-service"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{service="order-service",status_code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="order-service"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: P1
        annotations:
          summary: "订单服务错误预算快速消耗"
          description: "当前错误率是 SLO 允许值的 14.4 倍，按此速度 SLO 将在 1 天内耗尽"
          runbook_url: "https://wiki.example.com/runbooks/order-service-high-error-rate"

      # 慢速燃烧告警：过去 6 小时内，错误预算消耗速度是正常的 6 倍
      - alert: OrderServiceSlowErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{service="order-service",status_code=~"5.."}[6h]))
            /
            sum(rate(http_requests_total{service="order-service"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{service="order-service",status_code=~"5.."}[30m]))
            /
            sum(rate(http_requests_total{service="order-service"}[30m]))
          ) > (6 * 0.001)
        for: 5m
        labels:
          severity: P2
        annotations:
          summary: "订单服务错误预算缓慢消耗"
          description: "当前错误率是 SLO 允许值的 6 倍，按此速度 SLO 将在 4 天内耗尽"

## Oncall 交接模板

### 本周值班概况
- 告警总数：17 次
- 需要人工介入的告警：3 次
- 事故数量：0

### 待关注事项
1. 支付服务在周三出现过一次短暂的延迟飙升，根因未确认，
   可能与第三方支付网关的维护有关。已在 Jira 中跟踪：PAY-1234
2. 周五的批处理任务执行时间比正常慢了 30%，可能是数据量增长导致。
   如果下周继续恶化，需要联系数据团队。

### 近期变更
- 周二部署了订单服务 v2.3.4（新增了优惠券叠加逻辑）
- 周四配置中心调整了缓存过期时间（从 5 分钟改为 10 分钟）

### Runbook 更新
- 更新了 "支付超时" Runbook，增加了第三方网关状态检查步骤

"""
toil_tracker.py：Toil 工作量跟踪工具
每周 oncall 交接时填写，用于跟踪和分析 Toil 趋势
"""

from dataclasses import dataclass
from datetime import datetime
from enum import Enum


class ToilCategory(Enum):
    ALERT_RESPONSE = "告警响应"
    MANUAL_OPERATION = "手工操作"
    TICKET_HANDLING = "工单处理"
    DATA_FIX = "数据修复"
    CAPACITY_MANAGEMENT = "容量管理"
    RELEASE_MANAGEMENT = "发布管理"
    OTHER = "其他"


@dataclass
class ToilEntry:
    date: datetime
    category: ToilCategory
    description: str
    duration_minutes: int
    is_automatable: bool
    automation_effort_hours: float = 0.0

    @property
    def roi_ratio(self) -> float:
        """如果自动化，多久能回本（假设每周出现一次）"""
        if not self.is_automatable or self.automation_effort_hours == 0:
            return 0.0
        weekly_saving_hours = self.duration_minutes / 60.0
        return self.automation_effort_hours / weekly_saving_hours


def calculate_toil_budget(entries: list[ToilEntry], total_hours: float = 40.0) -> dict:
    """计算 Toil 占总工作时间的比例"""
    total_toil_minutes = sum(e.duration_minutes for e in entries)
    total_toil_hours = total_toil_minutes / 60.0
    toil_percentage = (total_toil_hours / total_hours) * 100

    by_category = {}
    for entry in entries:
        cat = entry.category.value
        by_category[cat] = by_category.get(cat, 0) + entry.duration_minutes

    automatable_minutes = sum(
        e.duration_minutes for e in entries if e.is_automatable
    )

    return {
        "total_toil_hours": round(total_toil_hours, 1),
        "toil_percentage": round(toil_percentage, 1),
        "by_category": by_category,
        "automatable_percentage": round(
            automatable_minutes / total_toil_minutes * 100, 1
        ) if total_toil_minutes > 0 else 0,
        "status": "healthy" if toil_percentage < 50 else "warning" if toil_percentage < 75 else "critical",
    }

## 事故响应检查清单

### 确认阶段（前 5 分钟）
- [ ] 确认告警内容和影响范围
- [ ] 确定事故级别（P0/P1/P2/P3/P4）
- [ ] 指定 IC（如果自己不是）
- [ ] 创建事故频道/War Room
- [ ] 发送第一条状态更新

### 分诊阶段（5-15 分钟）
- [ ] 检查近期变更（代码/配置/基础设施）
- [ ] 检查是否有关联的已知问题
- [ ] 查看 Runbook 是否覆盖此场景
- [ ] 分配调查任务

### 缓解阶段
- [ ] 确定缓解方案（回滚/降级/限流）
- [ ] 执行缓解操作
- [ ] 验证缓解效果
- [ ] 更新状态页面

### 解决阶段
- [ ] 确认根因
- [ ] 实施永久修复
- [ ] 验证修复效果
- [ ] 确认所有受影响的用户已恢复

### 收尾阶段
- [ ] 发送事故关闭通知
- [ ] 创建 Postmortem 文档
- [ ] 安排 Postmortem 评审会议
- [ ] 创建改进措施的 Jira Tickets