Mission 04
On-Prem AI for Operations
Local AI, private RAG, and operational assistants running inside your infrastructure.
Some operational data should stay inside your network: logs, incidents, runbooks, internal docs, and platform context.
This mission builds local AI or RAG systems with the same operating rules as the rest of your platform: RBAC, audit logs, observability, backups, and clear ownership.
When this is NOT a fit
- Your data can safely use a managed cloud AI service and latency, cost, or compliance do not justify local operation.
- The runbooks, incident notes, and service documentation are too stale or fragmented to support useful retrieval.
- There is no team ready to own model updates, access reviews, prompt changes, and GPU capacity after the initial deployment.
- The expected use case requires guaranteed answers or autonomous remediation without human review.
Common failure modes we’ve seen
- Teams deploy a model endpoint first, then discover nobody defined access control, audit logs, retention, or incident handling for AI usage.
- RAG quality looks good in demos but fails during incidents because documents are duplicated, outdated, or missing service ownership metadata.
- GPU sizing is based on model fit only, ignoring concurrent users, context length, batch behavior, and acceptable response latency.
- Operational agents get broad credentials instead of scoped, read-only access, turning a convenience tool into a production risk.
What's included
- Model serving: vLLM or Ollama deployment on your GPU-equipped nodes, with OpenAI-compatible API
- Hardware assessment: GPU sizing, memory requirements, inference latency targets, quantization strategy (GGUF/AWQ)
- RAG pipeline: document ingestion, vector store setup (PostgreSQL/pgvector), embedding pipeline, retrieval-optimized prompting
- SRE agent integration: Graphia-style agent that connects to your observability stack (Grafana queries, log search, incident context) with RBAC
- Security hardening: access control, audit logging, prompt injection defenses, network isolation
- Cost analysis: inference cost per query vs. cloud API pricing, break-even on GPU investment
Deliverables
- Local LLM inference endpoint with OpenAI-compatible API, running on your infra
- Private RAG pipeline indexed on your documentation and runbooks
- Grafana/observability integration proof-of-concept with RBAC
- Deployment manifests (Proxmox VM or Kubernetes) with GPU passthrough configured
- Security and access-control documentation
- Model evaluation report: quality, latency, and cost per query vs. cloud alternatives
Tech stack
vLLMOllamaRAGGPUProxmoxPythonMCP
Want to scope this?
Think this fits your needs? Let's scope it together.