nimmersky/oghma-proxy/TECHNICAL-SPEC.md

# Oghma RAG Proxy — Technical Specification

**Project:** SkyrimNet Lore Enhancement via RAG Proxy
**Status:** Design Phase
**Author:** Chrysalis + dafit
**Created:** 2026-03-30

---

## 1. Problem Statement

SkyrimNet currently relies on:
1. LLM's baked-in Skyrim knowledge (incomplete, potentially wrong)
2. Dynamic memories (what the NPC witnessed)

**Missing:** Authoritative lore retrieval filtered by what each NPC *should* know.

**Result:**
- Knowledge bleedover (guard knows Telvanni secrets)
- Lore inaccuracies (mixing up timelines, factions)
- No grounding in canon

---

## 2. Solution: Oghma RAG Proxy

A transparent proxy that sits between SkyrimNet and the LLM inference endpoint.

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           OGHMA RAG PROXY                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌──────────────┐         ┌──────────────────┐         ┌──────────────┐   │
│   │  SkyrimNet   │         │   Oghma Proxy    │         │  OpenRouter  │   │
│   │  SKSE Plugin │────────▶│   (FastAPI)      │────────▶│  / vLLM      │   │
│   │              │         │                  │         │              │   │
│   │  Port: N/A   │         │  Port: 8100      │         │  Upstream    │   │
│   └──────────────┘         └────────┬─────────┘         └──────────────┘   │
│                                     │                                       │
│                                     │ Query                                 │
│                                     ▼                                       │
│                            ┌──────────────────┐                            │
│                            │   iris-dev       │                            │
│                            │   ChromaDB       │                            │
│                            │   Port: 35000    │                            │
│                            │                  │                            │
│                            │   Collections:   │                            │
│                            │   - oghma_lore   │                            │
│                            │   - oghma_basic  │                            │
│                            └──────────────────┘                            │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## 3. Core Principles

1. **Zero SkyrimNet Changes** — Only change the endpoint URL in config
2. **Transparent Passthrough** — Unknown requests forward unchanged
3. **NPC-Aware Filtering** — Lore filtered by extracted NPC profile
4. **Two-Tier Content** — Scholars get deep lore, commoners get basics
5. **Nimmerverse Native** — Runs on existing infrastructure (iris-dev)

---

## 4. Architecture Components

### 4.1 Oghma Proxy Service

**Location:** VM on nimmerverse (could run on phoebe-dev or dedicated)
**Tech Stack:** Python 3.11+, FastAPI, httpx, chromadb-client
**Port:** 8100 (configurable)

**Responsibilities:**
- Intercept OpenRouter-compatible API requests
- Parse prompts to extract NPC context
- Query ChromaDB for relevant lore
- Inject lore into system prompt
- Forward to upstream LLM
- Stream response back to SkyrimNet

### 4.2 Oghma ChromaDB Collection

**Location:** iris-dev.eachpath.local:35000
**Collections:**

| Collection | Content | Use Case |
|------------|---------|----------|
| `oghma_lore` | Full `topic_desc` entries | Scholars, mages, priests |
| `oghma_basic` | Simplified `topic_desc_basic` | Commoners, guards, peasants |

**Metadata Schema:**
```json
{
  "topic": "Akatosh",
  "category": "Figures/Gods",
  "knowledge_classes": ["priest", "scholar", "dragon", "snowelf"],
  "knowledge_classes_basic": ["nord", "imperial", "breton"],
  "tags": ["divine", "time", "dragon-god"],
  "source_sheet": "Figures/Gods"
}
```

### 4.3 NPC Profile Extractor

Parses SkyrimNet prompts to extract:
- NPC name
- Race
- Profession/class (from bio or context)
- Factions
- Location (hold)
- Special traits

**Extraction Patterns:**
```python
# From character bio section
r"## (?P<name>\w+) Bio\n- Gender: (?P<gender>\w+)\n- Race: (?P<race>\w+)"

# From system context
r"You are (?P<name>[^,]+), a (?P<race>\w+) (?P<profession>\w+)"

# From faction mentions
r"member of (?:the )?(?P<faction>[\w\s]+)"
```

---

## 5. Data Flow

### 5.1 Request Interception

```
1. SkyrimNet sends POST /chat/completions to proxy
2. Proxy extracts NPC profile from messages
3. Proxy extracts conversation context/topic
4. Proxy queries ChromaDB:
   - Collection: oghma_lore or oghma_basic (based on NPC education)
   - Filter: knowledge_classes intersects NPC's classes
   - Query: conversation context (semantic search)
   - Limit: 3-5 most relevant entries
5. Proxy injects lore block into system message
6. Proxy forwards enriched request to upstream
7. Proxy streams response back to SkyrimNet
```

### 5.2 Lore Injection Format

Injected after character bio, before conversation:

```
## Relevant Lore Knowledge

Based on your background as a Nord priest in Whiterun, you would know:

- **Talos**: [Condensed lore about Talos worship, appropriate to character]
- **Whiterun**: [Local knowledge about the hold]
- **Companions**: [If character has connection]

Remember: This is knowledge your character possesses. Reference it naturally in conversation, don't recite it.
```

---

## 6. NPC Knowledge Classification

### 6.1 Profile → Knowledge Classes Mapping

```python
RACE_CLASSES = {
    "Nord": ["nord"],
    "Dunmer": ["darkelf", "dunmer"],
    "Altmer": ["highelf", "altmer"],
    "Bosmer": ["woodelf", "bosmer"],
    "Argonian": ["argonian"],
    "Khajiit": ["khajiit"],
    "Breton": ["breton"],
    "Redguard": ["redguard"],
    "Orsimer": ["orc", "orsimer"],
    "Imperial": ["imperial"],
}

PROFESSION_CLASSES = {
    "priest": ["priest"],
    "mage": ["mage", "scholar"],
    "scholar": ["scholar"],
    "blacksmith": ["blacksmith"],
    "guard": ["guard", "warrior"],
    "thief": ["thief"],
    "merchant": ["merchant"],
    "innkeeper": ["innkeeper"],
    "hunter": ["hunter"],
    "farmer": ["peasant"],
    "noble": ["noble"],
    "bard": ["bard"],
}

LOCATION_CLASSES = {
    "Whiterun": ["whiterun"],
    "Windhelm": ["eastmarch"],
    "Solitude": ["haafingar"],
    "Riften": ["rift"],
    "Markarth": ["reach"],
    "Morthal": ["hjaalmarch"],
    "Dawnstar": ["pale"],
    "Winterhold": ["winterhold"],
    "Falkreath": ["falkreath"],
    "Solstheim": ["solstheim"],
}

FACTION_CLASSES = {
    "Companions": ["companions"],
    "College of Winterhold": ["college", "mage"],
    "Thieves Guild": ["thieves"],
    "Dark Brotherhood": ["darkbrotherhood"],
    "Stormcloaks": ["stormcloak"],
    "Imperial Legion": ["imperial"],
    "Thalmor": ["thalmor"],
    "Dawnguard": ["dawnguard"],
    "Volkihar": ["vampire", "volkihar"],
}
```

### 6.2 Education Level Detection

```python
def get_education_level(npc_profile: NPCProfile) -> str:
    """Determine if NPC gets full lore or basic summaries."""

    educated_professions = {"mage", "scholar", "priest", "noble", "bard"}
    educated_factions = {"College of Winterhold", "Thalmor"}

    if npc_profile.profession in educated_professions:
        return "scholar"
    if any(f in educated_factions for f in npc_profile.factions):
        return "scholar"

    return "commoner"
```

---

## 7. API Specification

### 7.1 Proxy Endpoints

**POST /v1/chat/completions** (OpenRouter compatible)
- Intercepts, enriches, forwards
- Supports streaming

**POST /v1/completions** (Legacy)
- Same enrichment logic

**GET /health**
- Returns proxy + ChromaDB status

**GET /stats**
- Lore injection statistics
- Cache hit rates
- Average latency added

### 7.2 Configuration

```yaml
# oghma-proxy.yaml
proxy:
  host: 0.0.0.0
  port: 8100

upstream:
  # OpenRouter
  url: https://openrouter.ai/api/v1
  api_key: ${OPENROUTER_API_KEY}
  # Or local vLLM
  # url: http://localhost:8000/v1

chromadb:
  host: iris-dev.eachpath.local
  port: 35000
  collection_lore: oghma_lore
  collection_basic: oghma_basic

retrieval:
  max_results: 5
  min_score: 0.6

injection:
  enabled: true
  position: after_bio  # after_bio | before_conversation | system_suffix

logging:
  level: INFO
  log_injections: true
  log_to_phoebe: true  # Log to nimmerverse decision table
```

---

## 8. Deployment Architecture

### 8.1 Host Options

**Option A: Dedicated VM (Recommended)**
```
VM ID: 109
Hostname: oghma.eachpath.local
IP: 10.0.40.109
Resources: 2 vCPU, 4GB RAM
OS: Rocky Linux 10
```

**Option B: Co-locate on phoebe-dev**
```
Hostname: phoebe-dev.eachpath.local
Port: 8100 (alongside PostgreSQL 35432)
Pros: No new VM, shared resources
Cons: Resource contention
```

### 8.2 Service Configuration

```ini
# /etc/systemd/system/oghma-proxy.service
[Unit]
Description=Oghma RAG Proxy for SkyrimNet
After=network.target

[Service]
Type=simple
User=chrysalis
WorkingDirectory=/opt/oghma-proxy
ExecStart=/opt/oghma-proxy/venv/bin/python -m uvicorn main:app --host 0.0.0.0 --port 8100
Restart=always
RestartSec=5
Environment=OPENROUTER_API_KEY=your-key-here

[Install]
WantedBy=multi-user.target
```

### 8.3 SkyrimNet Configuration Change

In `config/OpenRouter.yaml`:
```yaml
# Before
openrouter:
  base_url: https://openrouter.ai/api/v1

# After
openrouter:
  base_url: http://oghma.eachpath.local:8100/v1
  # Or if running locally:
  # base_url: http://localhost:8100/v1
```

---

## 9. Data Pipeline: Oghma Ingestion

### 9.1 Google Sheets → ChromaDB Pipeline

```python
# ingest_oghma.py

import pandas as pd
import chromadb
from chromadb.config import Settings

SHEET_ID = "1dcfctU-iOqprwy2BOc7___4Awteczgdlv8886KalPsQ"
SHEETS = [
    ("Figures/Gods", 0),
    ("Artifacts", 1),
    ("Locations - Whiterun", 2),
    # ... etc
]

def ingest_sheet(sheet_name: str, gid: int, chroma_client):
    url = f"https://docs.google.com/spreadsheets/d/{SHEET_ID}/export?format=csv&gid={gid}"
    df = pd.read_csv(url)

    collection_lore = chroma_client.get_or_create_collection("oghma_lore")
    collection_basic = chroma_client.get_or_create_collection("oghma_basic")

    for _, row in df.iterrows():
        topic = row['topic']

        # Full lore for educated NPCs
        if pd.notna(row.get('topic_desc')):
            collection_lore.add(
                documents=[row['topic_desc']],
                ids=[f"{sheet_name}:{topic}"],
                metadatas=[{
                    "topic": topic,
                    "category": sheet_name,
                    "knowledge_classes": row.get('knowledge_class', ''),
                    "tags": row.get('tags', ''),
                }]
            )

        # Basic lore for commoners
        if pd.notna(row.get('topic_desc_basic')):
            collection_basic.add(
                documents=[row['topic_desc_basic']],
                ids=[f"{sheet_name}:{topic}:basic"],
                metadatas=[{
                    "topic": topic,
                    "category": sheet_name,
                    "knowledge_classes": row.get('knowledge_class_basic', ''),
                    "tags": row.get('tags', ''),
                }]
            )
```

### 9.2 Embedding Model

Use same embedding model as SkyrimNet memories for consistency:
- **Model:** `all-MiniLM-L6-v2` (384 dimensions)
- **Or:** Match whatever SkyrimNet uses in Memory.yaml

---

## 10. Implementation Phases

### Phase 1: Foundation (Week 1)
- [ ] Set up oghma-proxy repository
- [ ] Implement basic FastAPI proxy (passthrough mode)
- [ ] Test with SkyrimNet (verify transparent forwarding)
- [ ] Deploy on phoebe-dev for initial testing

### Phase 2: Oghma Ingestion (Week 1-2)
- [ ] Write Google Sheets ingestion script
- [ ] Ingest all Oghma sheets into iris-dev ChromaDB
- [ ] Verify embeddings and metadata
- [ ] Test semantic queries manually

### Phase 3: NPC Profile Extraction (Week 2)
- [ ] Implement prompt parser for NPC context
- [ ] Build knowledge class mapper
- [ ] Test with various NPC types
- [ ] Handle edge cases (unnamed NPCs, generic guards)

### Phase 4: RAG Integration (Week 2-3)
- [ ] Implement ChromaDB query logic
- [ ] Build lore injection formatter
- [ ] Add education-level routing (scholar vs commoner)
- [ ] Test end-to-end with SkyrimNet

### Phase 5: Polish & Deploy (Week 3)
- [ ] Add streaming support
- [ ] Implement caching (avoid re-querying same context)
- [ ] Add metrics/logging to phoebe
- [ ] Create dedicated VM or finalize deployment
- [ ] Write operational runbook

### Phase 6: Iteration (Ongoing)
- [ ] Tune retrieval parameters based on gameplay
- [ ] Add more knowledge sources beyond Oghma
- [ ] Consider contributing upstream to SkyrimNet

---

## 11. Success Metrics

| Metric | Target | How to Measure |
|--------|--------|----------------|
| Lore accuracy | NPCs reference correct lore | Manual spot-checks |
| Knowledge scoping | Guards don't know mage secrets | Test with various NPC types |
| Latency overhead | < 200ms added | Proxy metrics |
| Stability | 99.9% uptime | Service monitoring |
| User experience | Immersion improved | Subjective gameplay testing |

---

## 12. Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| Latency too high | Breaks dialogue flow | Cache aggressively, async prefetch |
| Wrong lore injected | Immersion broken | Strict knowledge filtering, fallback to no injection |
| ChromaDB down | No lore enrichment | Graceful degradation (passthrough mode) |
| Prompt parsing fails | NPC profile unknown | Default to basic/generic lore |
| Oghma data incomplete | Missing topics | Supplement with UESP scraping |

---

## 13. Future Enhancements

1. **Player profile tracking** — Remember what player has learned, NPCs can reference shared knowledge
2. **Gossip propagation** — Lore spreads through NPC network with degradation
3. **Dynamic lore updates** — Events in-game add to lore corpus
4. **Multi-source RAG** — Combine Oghma + UESP + custom worldbuilding
5. **Upstream contribution** — Propose RAG API to SkyrimNet author

---

**Version:** 1.0 | **Created:** 2026-03-30 | **Updated:** 2026-03-30