12 · 监控 & 可观测性

Prometheus 文档：https://prometheus.io/docs/ Grafana 文档：https://grafana.com/docs/ Loki 文档：https://grafana.com/docs/loki/latest/ Google SRE Book：https://sre.google/sre-book/table-of-contents/

---

1. Prometheus + Grafana 部署

# docker-compose 快速启动（本地/测试）
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    ports: ["3000:3000"]

  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - ./promtail-config.yml:/etc/promtail/config.yml

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['10.0.10.1:9100', '10.0.10.2:9100']

  - job_name: 'app-metrics'
    static_configs:
      - targets: ['api-server:8080']
    metrics_path: '/metrics'

---

2. 关键告警规则（iGaming）

# alerts/igaming.yml
groups:
  - name: igaming-critical
    rules:
      # API 可用性
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "5xx 错误率超过 1%"
          description: "当前错误率: {{ $value | humanizePercentage }}"

      # 赔率更新停止
      - alert: OddsUpdateStopped
        expr: rate(odds_updates_total[5m]) < 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "赔率更新已停止！"

      # 数据库连接池
      - alert: DBConnectionPoolExhausted
        expr: db_pool_available_connections < 5
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "数据库连接池即将耗尽"

      # Redis 内存
      - alert: RedisMemoryHigh
        expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Redis 内存使用超过 85%"

---

3. 关键 PromQL 查询

# API 错误率（过去5分钟）
sum(rate(http_requests_total{status=~"5.."}[5m])) 
/ sum(rate(http_requests_total[5m])) * 100

# p95/p99 响应时间
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 每秒请求数（RPS）
sum(rate(http_requests_total[1m]))

# WebSocket 当前连接数
websocket_connections_total

# Redis 命令执行速率
rate(redis_commands_total[1m])

---

官方文档

• Prometheus 查询语言：https://prometheus.io/docs/prometheus/latest/querying/basics/
• Grafana Dashboard 市场：https://grafana.com/grafana/dashboards/
• Node Exporter（服务器监控）：https://github.com/prometheus/node_exporter
• AlertManager 配置：https://prometheus.io/docs/alerting/latest/alertmanager/
• Grafana Loki 日志：https://grafana.com/docs/loki/latest/
• Uptime Kuma（自部署 Uptime）：https://github.com/louislam/uptime-kuma

---

最后更新：2025-04