← All one-liners·#048·Troubleshooting·llm cli·expert

Analyze CrashLoopBackOff Pod Logs with Gemini

Automatically finds all Kubernetes pods in a CrashLoopBackOff state, extracts their recent logs, and uses Gemini to diagnose the root cause.

Setup
  • → kubectl configured with cluster access
  • → jq installed
  • → gemini CLI installed and authenticated with an API key
Cost per run
Gemini API usage costs based on log token volume
The one-liner
$ kubectl get pods -A -o json | \
  jq -r '.items[] | select(any(.status.containerStatuses[]?; .state.waiting.reason == "CrashLoopBackOff")) | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -r -n2 sh -c 'echo "\n--- Logs for $0/$1 ---"; kubectl logs -n "$0" "$1" --tail=50' | \
  gemini -p "Analyze these logs from crashing Kubernetes pods and determine the root cause."
What each stage does
  1. [01] kubectlkubectl get pods -A -o json
    Fetches the status of all pods across all namespaces in JSON format for reliable structured parsing.
  2. [02] jqjq -r '.items[] | select(any(.status.containerStatuses[]?; .state.waiting.reason…
    Filters the JSON to find pods with at least one container in CrashLoopBackOff, extracting the namespace and pod name as raw text.
  3. [03] xargsxargs -r -n2 sh -c 'echo "\n--- Logs for $0/$1 ---"; kubectl logs -n "$0" "$1" -…
    Reads the namespace and pod name pairs (-n2), safely passing them to a subshell to fetch the last 50 lines of logs for each pod.
  4. [04] geminigemini -p "Analyze these logs..."
    Pipes the aggregated logs directly into the Gemini LLM CLI to generate a human-readable root cause analysis.
Expected output (sample)
**Root Cause Analysis:**
1. `frontend/web-app-7b89f`: The application is failing to start due to a missing environment variable `DATABASE_URL`.
2. `backend/api-worker-2c44a`: The container is running out of memory (OOMKilled) during the initial cache hydration phase.

**Recommendation:** 
Update the ConfigMap for `web-app` and increase memory limits for `api-worker`.
Caveats & tips
  • Cost/Permission: Requires a configured Gemini API key, which may incur token costs for large log payloads.
  • Footgun: If many pods are crashing simultaneously, the combined logs might exceed the LLM's context window.
  • Security: Ensure your pod logs do not contain sensitive PII or secrets before sending them to an external LLM API.