☸ KubeOwl — Agentic RAG for Kubernetes Operations

KubeOwl is an AI agent that diagnoses operational problems in Kubernetes clusters and recommends fixes. It combines a multi-agent ReAct architecture with retrieval-augmented generation over your own runbooks.

What It Does

• Diagnoses cluster problems: CrashLoopBackOff, OOMKilled, ImagePullBackOff, Node NotReady, disk pressure, PVC/StorageClass issues, Service/DNS failures
• Searches your runbooks with hybrid retrieval (BM25 keyword + vector semantic search, reciprocal-rank fusion)
• Inspects the live cluster with read-only kubectl commands (get, describe, logs, top) — locally via your kubeconfig or remotely over SSH
• Recommends concrete fixes backed by runbook evidence, with the exact commands to run

Architecture

flowchart TD
    UI[Web UI - chat, runbooks, settings] --> API[FastAPI backend]
    API --> SUP[Supervisor ReActAgent]
    SUP --> C[Compute specialist]
    SUP --> N[Network specialist]
    SUP --> S[Storage specialist]
    SUP --> RB[search_runbooks tool]
    C --> K8S[kubectl_exec - read-only allowlist]
    N --> K8S
    S --> K8S
    SUP --> K8S
    RB --> RAG[(ChromaDB + BM25 hybrid retrieval)]
    K8S --> CLUSTER[Kubernetes cluster - local kubeconfig or SSH]

The supervisor agent routes each question to the right domain specialist (compute / network / storage). Every agent can search the runbook knowledge base and run allowlisted, read-only kubectl commands. Constructed agents are cached per provider.

LLM Providers

Provider	LLM	Embeddings	Notes
ollama (default)	any local Ollama model	nomic-embed-text	fully local, no data leaves your machine
ollama-cloud	remote Ollama instance	via remote instance	set OLLAMA_CLOUD_URL
gemini	Gemini (google-genai)	text-embedding-004	requires GEMINI_API_KEY
claude	Anthropic Claude	falls back to local Ollama	requires ANTHROPIC_API_KEY
openai	OpenAI GPT	falls back to local Ollama	requires OPENAI_API_KEY

Embedding caveat: documents are embedded with the provider active at upload time. Querying the same collection with a different embedding model degrades retrieval; KubeOwl logs a warning when it detects a mismatch. Re-index your documents after switching embedding providers.

Quickstart

Local (recommended for development)

# 1. Python 3.11 environment
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Configuration
cp .env.example .env   # then edit: API keys, provider, flags

# 3. Local models (for the default Ollama provider)
ollama pull qwen2.5:7b
ollama pull nomic-embed-text

# 4. Run
./start.sh             # or: uvicorn backend.main:app --reload

Open http://localhost:8000, upload a runbook or two from runbooks/, and ask something like "pods in the production namespace keep restarting, what could be wrong?".

Docker

cp .env.example .env   # configure first — compose reads it at runtime
docker compose up --build

The image is a multi-stage build that runs as a non-root user and contains no secrets; configuration is injected at runtime from your .env via env_file. The data/ directory (ChromaDB, uploads, SSH sessions) is a host volume.

Configuration

All settings live in .env (see .env.example). Highlights:

Variable	Default	Purpose
DEFAULT_PROVIDER	ollama	active LLM provider
KUBECONFIG_PATH	~/.kube/config	kubeconfig used for local kubectl
ENABLE_TERMINAL	false	opt-in web terminal (kubectl allowlist enforced)
API_KEY	(empty)	when set, /api/* requires the X-API-Key header
ALLOWED_ORIGINS	["http://localhost:8000"]	CORS origins
SSH_AUTO_ADD_HOST_KEYS	false	auto-trust unknown SSH host keys (lab only)

Without a kubeconfig or an active SSH session, kubectl queries return an honest error — there is no fake data mode.

Security Model

• kubectl is restricted to a read-only allowlist (get, describe, logs, top); write verbs and shell metacharacters are rejected
• kubectl output is sanitized (tokens, secrets, and base64 blobs are redacted) before reaching the LLM
• Uploaded filenames are sanitized and path-contained; upload size is enforced while streaming
• The web terminal is disabled by default and, even when enabled, only accepts allowlist-validated kubectl commands
• SSH host keys are verified by default (RejectPolicy); saved sessions are written with mode 0600
• Agent answers are rendered through DOMPurify to block markdown-borne XSS

See SECURITY.md for the full threat model and accepted risks.

Development

pip install -e ".[dev]"      # dev tools: ruff, mypy, bandit, pytest, pip-audit
pre-commit install           # fast checks on every commit

ruff check . && ruff format --check .   # lint + formatting
mypy                                     # type check
bandit -c pyproject.toml -r backend      # security lint
pytest                                   # 200+ hermetic tests, 80% coverage gate
pip-audit -r requirements.txt --disable-pip --no-deps   # dependency CVE scan

The test suite is fully offline: LLM and embedding calls use LlamaIndex mocks, ChromaDB runs in a temp directory, and subprocess/SSH calls are stubbed.

Dependencies are declared as ranges in pyproject.toml and locked in requirements.txt:

uv pip compile pyproject.toml -o requirements.txt   # refresh the lock

Project Structure

backend/
├── main.py                 # FastAPI app, CORS, API-key middleware
├── config.py               # pydantic-settings configuration
├── api/                    # routers: agent, documents, system
├── core/
│   ├── agent.py            # supervisor agent + caching
│   ├── specialist_agents/  # compute / network / storage experts
│   ├── indexer.py          # ChromaDB + BM25 hybrid indexing
│   ├── memory.py           # per-session conversation memory
│   ├── prompts.py          # system prompts
│   └── providers.py        # LLM/embedding factories
├── tools/                  # kubectl_exec + runbook search tools
└── utils/                  # security validator, file handling, SSH
frontend/                   # vanilla JS + Bulma single-page UI
runbooks/                   # starter Kubernetes troubleshooting runbooks
tests/                      # hermetic pytest suite

License

MIT © 2026 Çağatay ÜRESİN