Investigação de Consumo de Orçamento de Erro

# 1. Error budget remaining over time (burn-down chart)
1 - (
  sum(increase(http_requests_total{service="checkout", status=~"5.."}[30d]))
  /
  (sum(increase(http_requests_total{service="checkout"}[30d])) * 0.001)
)

# 2. Error rate by endpoint -- find the hot spot
sum by (endpoint) (
  rate(http_requests_total{service="checkout", status=~"5.."}[1h])
)
/ sum by (endpoint) (
  rate(http_requests_total{service="checkout"}[1h])
)

# 3. Error rate by status code -- understand the failure mode
sum by (status) (
  increase(http_requests_total{service="checkout", status=~"5.."}[2d])
)

# 4. Dependency error rates -- is it us or a downstream?
sum by (dependency) (
  rate(dependency_requests_total{service="checkout", status="error"}[2d])
)

# Check what was deployed over the weekend
kubectl rollout history deployment/checkout-api -n production
# REVISION  CHANGE-CAUSE
# 14        deploy: v2.3.1 (Thu Apr 9 17:42:00)
# 15        deploy: v2.3.2 (Fri Apr 10 16:58:00)  <-- Friday afternoon deploy
# 16        rollback to v2.3.1 (Mon Apr 13 09:15:00)

# Check the diff between the Friday deploy and the previous version
git log --oneline v2.3.1..v2.3.2
# a1b2c3d Update payment retry logic
# d4e5f6g Remove connection pool size limit  <-- suspicious

# Correlate with error spike timing
kubectl logs deployment/checkout-api -n production --since=48h \
  | grep -c 'connection refused'
# 14,832

postmortem:
  title: "80% error budget burn - checkout API (Apr 10-12)"
  date: 2025-04-13
  authors: ["oncall-engineer"]
  status: action-items-pending

  summary: |
    Friday deploy v2.3.2 removed connection pool limits,
    causing DB connection exhaustion under weekend batch load.
    80% of monthly error budget consumed over 48 hours.

  timeline:
    - time: "Fri 16:58" 
      event: "v2.3.2 deployed to production"
    - time: "Fri 23:00"
      event: "Weekend batch jobs begin, connection pool grows unbounded"
    - time: "Sat 02:15"
      event: "Slow-burn alert fires (10x burn rate)"
    - time: "Sat 02:15"
      event: "Alert acknowledged but not investigated (oncall assumed transient)"
    - time: "Mon 09:00"
      event: "Budget burn discovered during morning review"
    - time: "Mon 09:15"
      event: "Rollback to v2.3.1"

  root_cause: |
    Connection pool size limit was removed in v2.3.2.
    Under normal weekday traffic this was fine, but weekend
    batch jobs created enough concurrent connections to
    exhaust the database connection limit.

  action_items:
    - owner: sre-team
      action: "Add connection pool limits to deploy-time config validation"
      priority: P1
    - owner: oncall-lead
      action: "Update runbook: slow-burn alerts require investigation, not just ack"
      priority: P1
    - owner: platform-team
      action: "Add canary analysis gate for Friday deploys"
      priority: P2