feat: Add Oghma RAG Proxy for SkyrimNet lore injection
RAG proxy that intercepts SkyrimNet LLM requests and enriches them with relevant Tamrielic lore from CHIM's Oghma Infinium database. Features: - FastAPI proxy compatible with OpenAI API - ChromaDB semantic search for lore retrieval - NPC profile extraction from SkyrimNet prompts - Google Sheets ingestion for CHIM's Oghma data - Kubernetes deployment manifests - Debug endpoint for RAG operation monitoring Collections ingested to iris-dev ChromaDB: - oghma_lore: 1951 entries (scholar knowledge) - oghma_basic: 1949 entries (commoner knowledge) - oghma_visual: 1151 entries (Omnisight perception) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
dafit
5
oghma-proxy/.gitignore
vendored
Normal file
5
oghma-proxy/.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
config.local.yaml
|
||||
.env
|
||||
|
||||
35
oghma-proxy/Dockerfile
Normal file
35
oghma-proxy/Dockerfile
Normal file
@@ -0,0 +1,35 @@
|
||||
# Oghma RAG Proxy - Container Image
|
||||
FROM python:3.11-slim
|
||||
|
||||
LABEL maintainer="dafit@eachpath.local"
|
||||
LABEL description="RAG Proxy for SkyrimNet - Injects Tamrielic lore into NPC conversations"
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy project files
|
||||
COPY pyproject.toml .
|
||||
COPY src/ ./src/
|
||||
COPY config.yaml .
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Create non-root user
|
||||
RUN useradd -m -u 1000 oghma
|
||||
USER oghma
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8100
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:8100/health || exit 1
|
||||
|
||||
# Run the proxy
|
||||
CMD ["python", "-m", "uvicorn", "oghma_proxy.main:app", "--host", "0.0.0.0", "--port", "8100"]
|
||||
54
oghma-proxy/README.md
Normal file
54
oghma-proxy/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Oghma RAG Proxy
|
||||
|
||||
RAG (Retrieval-Augmented Generation) proxy for SkyrimNet that injects Tamrielic lore into NPC conversations based on their knowledge profile.
|
||||
|
||||
## Overview
|
||||
|
||||
This proxy sits between SkyrimNet and your LLM inference endpoint (OpenRouter/vLLM), enriching prompts with relevant lore from CHIM's Oghma Infinium database.
|
||||
|
||||
**Key Features:**
|
||||
- Zero changes to SkyrimNet — just change the endpoint URL
|
||||
- NPC-aware filtering — guards don't know mage secrets
|
||||
- Two-tier knowledge — scholars get deep lore, commoners get basics
|
||||
- ChromaDB-powered semantic search
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -e .
|
||||
|
||||
# Ingest Oghma lore into ChromaDB
|
||||
python -m oghma_proxy.ingest --host iris-dev.eachpath.local --port 35000
|
||||
|
||||
# Run proxy
|
||||
python -m oghma_proxy.main
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Copy `config.yaml` to `config.local.yaml` and customize:
|
||||
|
||||
```yaml
|
||||
upstream:
|
||||
url: https://openrouter.ai/api/v1
|
||||
api_key: ${OPENROUTER_API_KEY}
|
||||
|
||||
chromadb:
|
||||
host: iris-dev.eachpath.local
|
||||
port: 35000
|
||||
```
|
||||
|
||||
## Kubernetes Deployment
|
||||
|
||||
```bash
|
||||
kubectl apply -k k8s/
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
See [TECHNICAL-SPEC.md](TECHNICAL-SPEC.md) for full design documentation.
|
||||
|
||||
---
|
||||
|
||||
Part of the [nimmerverse](https://github.com/dafit/nimmerverse) project.
|
||||
497
oghma-proxy/TECHNICAL-SPEC.md
Normal file
497
oghma-proxy/TECHNICAL-SPEC.md
Normal file
@@ -0,0 +1,497 @@
|
||||
# Oghma RAG Proxy — Technical Specification
|
||||
|
||||
**Project:** SkyrimNet Lore Enhancement via RAG Proxy
|
||||
**Status:** Design Phase
|
||||
**Author:** Chrysalis + dafit
|
||||
**Created:** 2026-03-30
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
SkyrimNet currently relies on:
|
||||
1. LLM's baked-in Skyrim knowledge (incomplete, potentially wrong)
|
||||
2. Dynamic memories (what the NPC witnessed)
|
||||
|
||||
**Missing:** Authoritative lore retrieval filtered by what each NPC *should* know.
|
||||
|
||||
**Result:**
|
||||
- Knowledge bleedover (guard knows Telvanni secrets)
|
||||
- Lore inaccuracies (mixing up timelines, factions)
|
||||
- No grounding in canon
|
||||
|
||||
---
|
||||
|
||||
## 2. Solution: Oghma RAG Proxy
|
||||
|
||||
A transparent proxy that sits between SkyrimNet and the LLM inference endpoint.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ OGHMA RAG PROXY │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ │
|
||||
│ │ SkyrimNet │ │ Oghma Proxy │ │ OpenRouter │ │
|
||||
│ │ SKSE Plugin │────────▶│ (FastAPI) │────────▶│ / vLLM │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ Port: N/A │ │ Port: 8100 │ │ Upstream │ │
|
||||
│ └──────────────┘ └────────┬─────────┘ └──────────────┘ │
|
||||
│ │ │
|
||||
│ │ Query │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ iris-dev │ │
|
||||
│ │ ChromaDB │ │
|
||||
│ │ Port: 35000 │ │
|
||||
│ │ │ │
|
||||
│ │ Collections: │ │
|
||||
│ │ - oghma_lore │ │
|
||||
│ │ - oghma_basic │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Core Principles
|
||||
|
||||
1. **Zero SkyrimNet Changes** — Only change the endpoint URL in config
|
||||
2. **Transparent Passthrough** — Unknown requests forward unchanged
|
||||
3. **NPC-Aware Filtering** — Lore filtered by extracted NPC profile
|
||||
4. **Two-Tier Content** — Scholars get deep lore, commoners get basics
|
||||
5. **Nimmerverse Native** — Runs on existing infrastructure (iris-dev)
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Components
|
||||
|
||||
### 4.1 Oghma Proxy Service
|
||||
|
||||
**Location:** VM on nimmerverse (could run on phoebe-dev or dedicated)
|
||||
**Tech Stack:** Python 3.11+, FastAPI, httpx, chromadb-client
|
||||
**Port:** 8100 (configurable)
|
||||
|
||||
**Responsibilities:**
|
||||
- Intercept OpenRouter-compatible API requests
|
||||
- Parse prompts to extract NPC context
|
||||
- Query ChromaDB for relevant lore
|
||||
- Inject lore into system prompt
|
||||
- Forward to upstream LLM
|
||||
- Stream response back to SkyrimNet
|
||||
|
||||
### 4.2 Oghma ChromaDB Collection
|
||||
|
||||
**Location:** iris-dev.eachpath.local:35000
|
||||
**Collections:**
|
||||
|
||||
| Collection | Content | Use Case |
|
||||
|------------|---------|----------|
|
||||
| `oghma_lore` | Full `topic_desc` entries | Scholars, mages, priests |
|
||||
| `oghma_basic` | Simplified `topic_desc_basic` | Commoners, guards, peasants |
|
||||
|
||||
**Metadata Schema:**
|
||||
```json
|
||||
{
|
||||
"topic": "Akatosh",
|
||||
"category": "Figures/Gods",
|
||||
"knowledge_classes": ["priest", "scholar", "dragon", "snowelf"],
|
||||
"knowledge_classes_basic": ["nord", "imperial", "breton"],
|
||||
"tags": ["divine", "time", "dragon-god"],
|
||||
"source_sheet": "Figures/Gods"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 NPC Profile Extractor
|
||||
|
||||
Parses SkyrimNet prompts to extract:
|
||||
- NPC name
|
||||
- Race
|
||||
- Profession/class (from bio or context)
|
||||
- Factions
|
||||
- Location (hold)
|
||||
- Special traits
|
||||
|
||||
**Extraction Patterns:**
|
||||
```python
|
||||
# From character bio section
|
||||
r"## (?P<name>\w+) Bio\n- Gender: (?P<gender>\w+)\n- Race: (?P<race>\w+)"
|
||||
|
||||
# From system context
|
||||
r"You are (?P<name>[^,]+), a (?P<race>\w+) (?P<profession>\w+)"
|
||||
|
||||
# From faction mentions
|
||||
r"member of (?:the )?(?P<faction>[\w\s]+)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Data Flow
|
||||
|
||||
### 5.1 Request Interception
|
||||
|
||||
```
|
||||
1. SkyrimNet sends POST /chat/completions to proxy
|
||||
2. Proxy extracts NPC profile from messages
|
||||
3. Proxy extracts conversation context/topic
|
||||
4. Proxy queries ChromaDB:
|
||||
- Collection: oghma_lore or oghma_basic (based on NPC education)
|
||||
- Filter: knowledge_classes intersects NPC's classes
|
||||
- Query: conversation context (semantic search)
|
||||
- Limit: 3-5 most relevant entries
|
||||
5. Proxy injects lore block into system message
|
||||
6. Proxy forwards enriched request to upstream
|
||||
7. Proxy streams response back to SkyrimNet
|
||||
```
|
||||
|
||||
### 5.2 Lore Injection Format
|
||||
|
||||
Injected after character bio, before conversation:
|
||||
|
||||
```
|
||||
## Relevant Lore Knowledge
|
||||
|
||||
Based on your background as a Nord priest in Whiterun, you would know:
|
||||
|
||||
- **Talos**: [Condensed lore about Talos worship, appropriate to character]
|
||||
- **Whiterun**: [Local knowledge about the hold]
|
||||
- **Companions**: [If character has connection]
|
||||
|
||||
Remember: This is knowledge your character possesses. Reference it naturally in conversation, don't recite it.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. NPC Knowledge Classification
|
||||
|
||||
### 6.1 Profile → Knowledge Classes Mapping
|
||||
|
||||
```python
|
||||
RACE_CLASSES = {
|
||||
"Nord": ["nord"],
|
||||
"Dunmer": ["darkelf", "dunmer"],
|
||||
"Altmer": ["highelf", "altmer"],
|
||||
"Bosmer": ["woodelf", "bosmer"],
|
||||
"Argonian": ["argonian"],
|
||||
"Khajiit": ["khajiit"],
|
||||
"Breton": ["breton"],
|
||||
"Redguard": ["redguard"],
|
||||
"Orsimer": ["orc", "orsimer"],
|
||||
"Imperial": ["imperial"],
|
||||
}
|
||||
|
||||
PROFESSION_CLASSES = {
|
||||
"priest": ["priest"],
|
||||
"mage": ["mage", "scholar"],
|
||||
"scholar": ["scholar"],
|
||||
"blacksmith": ["blacksmith"],
|
||||
"guard": ["guard", "warrior"],
|
||||
"thief": ["thief"],
|
||||
"merchant": ["merchant"],
|
||||
"innkeeper": ["innkeeper"],
|
||||
"hunter": ["hunter"],
|
||||
"farmer": ["peasant"],
|
||||
"noble": ["noble"],
|
||||
"bard": ["bard"],
|
||||
}
|
||||
|
||||
LOCATION_CLASSES = {
|
||||
"Whiterun": ["whiterun"],
|
||||
"Windhelm": ["eastmarch"],
|
||||
"Solitude": ["haafingar"],
|
||||
"Riften": ["rift"],
|
||||
"Markarth": ["reach"],
|
||||
"Morthal": ["hjaalmarch"],
|
||||
"Dawnstar": ["pale"],
|
||||
"Winterhold": ["winterhold"],
|
||||
"Falkreath": ["falkreath"],
|
||||
"Solstheim": ["solstheim"],
|
||||
}
|
||||
|
||||
FACTION_CLASSES = {
|
||||
"Companions": ["companions"],
|
||||
"College of Winterhold": ["college", "mage"],
|
||||
"Thieves Guild": ["thieves"],
|
||||
"Dark Brotherhood": ["darkbrotherhood"],
|
||||
"Stormcloaks": ["stormcloak"],
|
||||
"Imperial Legion": ["imperial"],
|
||||
"Thalmor": ["thalmor"],
|
||||
"Dawnguard": ["dawnguard"],
|
||||
"Volkihar": ["vampire", "volkihar"],
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 Education Level Detection
|
||||
|
||||
```python
|
||||
def get_education_level(npc_profile: NPCProfile) -> str:
|
||||
"""Determine if NPC gets full lore or basic summaries."""
|
||||
|
||||
educated_professions = {"mage", "scholar", "priest", "noble", "bard"}
|
||||
educated_factions = {"College of Winterhold", "Thalmor"}
|
||||
|
||||
if npc_profile.profession in educated_professions:
|
||||
return "scholar"
|
||||
if any(f in educated_factions for f in npc_profile.factions):
|
||||
return "scholar"
|
||||
|
||||
return "commoner"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. API Specification
|
||||
|
||||
### 7.1 Proxy Endpoints
|
||||
|
||||
**POST /v1/chat/completions** (OpenRouter compatible)
|
||||
- Intercepts, enriches, forwards
|
||||
- Supports streaming
|
||||
|
||||
**POST /v1/completions** (Legacy)
|
||||
- Same enrichment logic
|
||||
|
||||
**GET /health**
|
||||
- Returns proxy + ChromaDB status
|
||||
|
||||
**GET /stats**
|
||||
- Lore injection statistics
|
||||
- Cache hit rates
|
||||
- Average latency added
|
||||
|
||||
### 7.2 Configuration
|
||||
|
||||
```yaml
|
||||
# oghma-proxy.yaml
|
||||
proxy:
|
||||
host: 0.0.0.0
|
||||
port: 8100
|
||||
|
||||
upstream:
|
||||
# OpenRouter
|
||||
url: https://openrouter.ai/api/v1
|
||||
api_key: ${OPENROUTER_API_KEY}
|
||||
# Or local vLLM
|
||||
# url: http://localhost:8000/v1
|
||||
|
||||
chromadb:
|
||||
host: iris-dev.eachpath.local
|
||||
port: 35000
|
||||
collection_lore: oghma_lore
|
||||
collection_basic: oghma_basic
|
||||
|
||||
retrieval:
|
||||
max_results: 5
|
||||
min_score: 0.6
|
||||
|
||||
injection:
|
||||
enabled: true
|
||||
position: after_bio # after_bio | before_conversation | system_suffix
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
log_injections: true
|
||||
log_to_phoebe: true # Log to nimmerverse decision table
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Deployment Architecture
|
||||
|
||||
### 8.1 Host Options
|
||||
|
||||
**Option A: Dedicated VM (Recommended)**
|
||||
```
|
||||
VM ID: 109
|
||||
Hostname: oghma.eachpath.local
|
||||
IP: 10.0.40.109
|
||||
Resources: 2 vCPU, 4GB RAM
|
||||
OS: Rocky Linux 10
|
||||
```
|
||||
|
||||
**Option B: Co-locate on phoebe-dev**
|
||||
```
|
||||
Hostname: phoebe-dev.eachpath.local
|
||||
Port: 8100 (alongside PostgreSQL 35432)
|
||||
Pros: No new VM, shared resources
|
||||
Cons: Resource contention
|
||||
```
|
||||
|
||||
### 8.2 Service Configuration
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/oghma-proxy.service
|
||||
[Unit]
|
||||
Description=Oghma RAG Proxy for SkyrimNet
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=chrysalis
|
||||
WorkingDirectory=/opt/oghma-proxy
|
||||
ExecStart=/opt/oghma-proxy/venv/bin/python -m uvicorn main:app --host 0.0.0.0 --port 8100
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=OPENROUTER_API_KEY=your-key-here
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### 8.3 SkyrimNet Configuration Change
|
||||
|
||||
In `config/OpenRouter.yaml`:
|
||||
```yaml
|
||||
# Before
|
||||
openrouter:
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
# After
|
||||
openrouter:
|
||||
base_url: http://oghma.eachpath.local:8100/v1
|
||||
# Or if running locally:
|
||||
# base_url: http://localhost:8100/v1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Data Pipeline: Oghma Ingestion
|
||||
|
||||
### 9.1 Google Sheets → ChromaDB Pipeline
|
||||
|
||||
```python
|
||||
# ingest_oghma.py
|
||||
|
||||
import pandas as pd
|
||||
import chromadb
|
||||
from chromadb.config import Settings
|
||||
|
||||
SHEET_ID = "1dcfctU-iOqprwy2BOc7___4Awteczgdlv8886KalPsQ"
|
||||
SHEETS = [
|
||||
("Figures/Gods", 0),
|
||||
("Artifacts", 1),
|
||||
("Locations - Whiterun", 2),
|
||||
# ... etc
|
||||
]
|
||||
|
||||
def ingest_sheet(sheet_name: str, gid: int, chroma_client):
|
||||
url = f"https://docs.google.com/spreadsheets/d/{SHEET_ID}/export?format=csv&gid={gid}"
|
||||
df = pd.read_csv(url)
|
||||
|
||||
collection_lore = chroma_client.get_or_create_collection("oghma_lore")
|
||||
collection_basic = chroma_client.get_or_create_collection("oghma_basic")
|
||||
|
||||
for _, row in df.iterrows():
|
||||
topic = row['topic']
|
||||
|
||||
# Full lore for educated NPCs
|
||||
if pd.notna(row.get('topic_desc')):
|
||||
collection_lore.add(
|
||||
documents=[row['topic_desc']],
|
||||
ids=[f"{sheet_name}:{topic}"],
|
||||
metadatas=[{
|
||||
"topic": topic,
|
||||
"category": sheet_name,
|
||||
"knowledge_classes": row.get('knowledge_class', ''),
|
||||
"tags": row.get('tags', ''),
|
||||
}]
|
||||
)
|
||||
|
||||
# Basic lore for commoners
|
||||
if pd.notna(row.get('topic_desc_basic')):
|
||||
collection_basic.add(
|
||||
documents=[row['topic_desc_basic']],
|
||||
ids=[f"{sheet_name}:{topic}:basic"],
|
||||
metadatas=[{
|
||||
"topic": topic,
|
||||
"category": sheet_name,
|
||||
"knowledge_classes": row.get('knowledge_class_basic', ''),
|
||||
"tags": row.get('tags', ''),
|
||||
}]
|
||||
)
|
||||
```
|
||||
|
||||
### 9.2 Embedding Model
|
||||
|
||||
Use same embedding model as SkyrimNet memories for consistency:
|
||||
- **Model:** `all-MiniLM-L6-v2` (384 dimensions)
|
||||
- **Or:** Match whatever SkyrimNet uses in Memory.yaml
|
||||
|
||||
---
|
||||
|
||||
## 10. Implementation Phases
|
||||
|
||||
### Phase 1: Foundation (Week 1)
|
||||
- [ ] Set up oghma-proxy repository
|
||||
- [ ] Implement basic FastAPI proxy (passthrough mode)
|
||||
- [ ] Test with SkyrimNet (verify transparent forwarding)
|
||||
- [ ] Deploy on phoebe-dev for initial testing
|
||||
|
||||
### Phase 2: Oghma Ingestion (Week 1-2)
|
||||
- [ ] Write Google Sheets ingestion script
|
||||
- [ ] Ingest all Oghma sheets into iris-dev ChromaDB
|
||||
- [ ] Verify embeddings and metadata
|
||||
- [ ] Test semantic queries manually
|
||||
|
||||
### Phase 3: NPC Profile Extraction (Week 2)
|
||||
- [ ] Implement prompt parser for NPC context
|
||||
- [ ] Build knowledge class mapper
|
||||
- [ ] Test with various NPC types
|
||||
- [ ] Handle edge cases (unnamed NPCs, generic guards)
|
||||
|
||||
### Phase 4: RAG Integration (Week 2-3)
|
||||
- [ ] Implement ChromaDB query logic
|
||||
- [ ] Build lore injection formatter
|
||||
- [ ] Add education-level routing (scholar vs commoner)
|
||||
- [ ] Test end-to-end with SkyrimNet
|
||||
|
||||
### Phase 5: Polish & Deploy (Week 3)
|
||||
- [ ] Add streaming support
|
||||
- [ ] Implement caching (avoid re-querying same context)
|
||||
- [ ] Add metrics/logging to phoebe
|
||||
- [ ] Create dedicated VM or finalize deployment
|
||||
- [ ] Write operational runbook
|
||||
|
||||
### Phase 6: Iteration (Ongoing)
|
||||
- [ ] Tune retrieval parameters based on gameplay
|
||||
- [ ] Add more knowledge sources beyond Oghma
|
||||
- [ ] Consider contributing upstream to SkyrimNet
|
||||
|
||||
---
|
||||
|
||||
## 11. Success Metrics
|
||||
|
||||
| Metric | Target | How to Measure |
|
||||
|--------|--------|----------------|
|
||||
| Lore accuracy | NPCs reference correct lore | Manual spot-checks |
|
||||
| Knowledge scoping | Guards don't know mage secrets | Test with various NPC types |
|
||||
| Latency overhead | < 200ms added | Proxy metrics |
|
||||
| Stability | 99.9% uptime | Service monitoring |
|
||||
| User experience | Immersion improved | Subjective gameplay testing |
|
||||
|
||||
---
|
||||
|
||||
## 12. Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Latency too high | Breaks dialogue flow | Cache aggressively, async prefetch |
|
||||
| Wrong lore injected | Immersion broken | Strict knowledge filtering, fallback to no injection |
|
||||
| ChromaDB down | No lore enrichment | Graceful degradation (passthrough mode) |
|
||||
| Prompt parsing fails | NPC profile unknown | Default to basic/generic lore |
|
||||
| Oghma data incomplete | Missing topics | Supplement with UESP scraping |
|
||||
|
||||
---
|
||||
|
||||
## 13. Future Enhancements
|
||||
|
||||
1. **Player profile tracking** — Remember what player has learned, NPCs can reference shared knowledge
|
||||
2. **Gossip propagation** — Lore spreads through NPC network with degradation
|
||||
3. **Dynamic lore updates** — Events in-game add to lore corpus
|
||||
4. **Multi-source RAG** — Combine Oghma + UESP + custom worldbuilding
|
||||
5. **Upstream contribution** — Propose RAG API to SkyrimNet author
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0 | **Created:** 2026-03-30 | **Updated:** 2026-03-30
|
||||
76
oghma-proxy/config.yaml
Normal file
76
oghma-proxy/config.yaml
Normal file
@@ -0,0 +1,76 @@
|
||||
# Oghma RAG Proxy Configuration
|
||||
# Copy to config.local.yaml and customize for your environment
|
||||
|
||||
proxy:
|
||||
host: 0.0.0.0
|
||||
port: 8100
|
||||
workers: 1
|
||||
|
||||
upstream:
|
||||
# OpenRouter (cloud)
|
||||
url: https://openrouter.ai/api/v1
|
||||
api_key: ${OPENROUTER_API_KEY}
|
||||
|
||||
# Local vLLM alternative:
|
||||
# url: http://localhost:8000/v1
|
||||
# api_key: ""
|
||||
|
||||
timeout: 120 # seconds
|
||||
stream_timeout: 300 # for streaming responses
|
||||
|
||||
chromadb:
|
||||
host: iris-dev.eachpath.local
|
||||
port: 35000
|
||||
collection_lore: oghma_lore
|
||||
collection_basic: oghma_basic
|
||||
|
||||
retrieval:
|
||||
max_results: 5
|
||||
min_score: 0.55
|
||||
embedding_model: all-MiniLM-L6-v2 # Match SkyrimNet memory embeddings
|
||||
|
||||
injection:
|
||||
enabled: true
|
||||
position: after_bio # after_bio | before_conversation | system_suffix
|
||||
|
||||
# Injection template
|
||||
template: |
|
||||
## Relevant Lore Knowledge
|
||||
|
||||
Based on your background, you would know:
|
||||
|
||||
{% for entry in lore_entries %}
|
||||
- **{{ entry.topic }}**: {{ entry.content }}
|
||||
{% endfor %}
|
||||
|
||||
Remember: Reference this knowledge naturally in conversation when relevant.
|
||||
|
||||
npc_extraction:
|
||||
# Regex patterns for extracting NPC info from prompts
|
||||
patterns:
|
||||
bio_header: '## (?P<name>[\w\s]+) Bio\n- Gender: (?P<gender>\w+)\n- Race: (?P<race>\w+)'
|
||||
role_context: 'You are (?P<name>[^,]+), (?:a |an )?(?P<race>\w+)'
|
||||
faction_member: 'member of (?:the )?(?P<faction>[\w\s]+)'
|
||||
location_in: '(?:in|at|near) (?P<location>Whiterun|Windhelm|Solitude|Riften|Markarth|Morthal|Dawnstar|Winterhold|Falkreath)'
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
format: json # json | console
|
||||
log_injections: true
|
||||
|
||||
# Log decisions to phoebe for analysis
|
||||
phoebe:
|
||||
enabled: true
|
||||
host: phoebe-dev.eachpath.local
|
||||
port: 35432
|
||||
database: nimmerverse
|
||||
table: oghma_proxy_decisions
|
||||
|
||||
cache:
|
||||
enabled: true
|
||||
ttl_seconds: 300 # Cache lore lookups for 5 minutes
|
||||
max_size: 1000 # Max cached queries
|
||||
|
||||
metrics:
|
||||
enabled: true
|
||||
endpoint: /metrics # Prometheus-compatible
|
||||
50
oghma-proxy/k8s/configmap.yaml
Normal file
50
oghma-proxy/k8s/configmap.yaml
Normal file
@@ -0,0 +1,50 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: oghma-proxy-config
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
data:
|
||||
config.yaml: |
|
||||
proxy:
|
||||
host: 0.0.0.0
|
||||
port: 8100
|
||||
workers: 1
|
||||
|
||||
upstream:
|
||||
# Configure via secret or environment variables
|
||||
url: ${UPSTREAM_URL}
|
||||
timeout: 120
|
||||
stream_timeout: 300
|
||||
|
||||
chromadb:
|
||||
# ChromaDB service - adjust based on your deployment
|
||||
# Option 1: External (iris-dev VM)
|
||||
host: iris-dev.eachpath.local
|
||||
port: 35000
|
||||
# Option 2: In-cluster ChromaDB
|
||||
# host: chromadb.nimmersky.svc.cluster.local
|
||||
# port: 8000
|
||||
collection_lore: oghma_lore
|
||||
collection_basic: oghma_basic
|
||||
|
||||
retrieval:
|
||||
max_results: 5
|
||||
min_score: 0.55
|
||||
embedding_model: all-MiniLM-L6-v2
|
||||
|
||||
injection:
|
||||
enabled: true
|
||||
position: after_bio
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
format: json
|
||||
log_injections: true
|
||||
|
||||
cache:
|
||||
enabled: true
|
||||
ttl_seconds: 300
|
||||
max_size: 1000
|
||||
108
oghma-proxy/k8s/deployment.yaml
Normal file
108
oghma-proxy/k8s/deployment.yaml
Normal file
@@ -0,0 +1,108 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oghma-proxy
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
app.kubernetes.io/component: inference-proxy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8100"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
serviceAccountName: oghma-proxy
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
fsGroup: 1000
|
||||
|
||||
containers:
|
||||
- name: oghma-proxy
|
||||
image: registry.eachpath.local/nimmerverse/oghma-proxy:latest
|
||||
imagePullPolicy: Always
|
||||
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8100
|
||||
protocol: TCP
|
||||
|
||||
env:
|
||||
- name: OPENROUTER_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oghma-proxy-secrets
|
||||
key: OPENROUTER_API_KEY
|
||||
- name: UPSTREAM_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oghma-proxy-secrets
|
||||
key: UPSTREAM_URL
|
||||
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /app/config.yaml
|
||||
subPath: config.yaml
|
||||
readOnly: true
|
||||
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: oghma-proxy-config
|
||||
|
||||
# Prefer scheduling near inference workloads
|
||||
affinity:
|
||||
podAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: inference
|
||||
topologyKey: kubernetes.io/hostname
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: oghma-proxy
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
22
oghma-proxy/k8s/kustomization.yaml
Normal file
22
oghma-proxy/k8s/kustomization.yaml
Normal file
@@ -0,0 +1,22 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: nimmersky
|
||||
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- configmap.yaml
|
||||
- secret.yaml
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
|
||||
commonLabels:
|
||||
app.kubernetes.io/managed-by: kustomize
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
|
||||
images:
|
||||
- name: registry.eachpath.local/nimmerverse/oghma-proxy
|
||||
newTag: latest
|
||||
|
||||
# For production, use overlays:
|
||||
# kustomize build k8s/overlays/production | kubectl apply -f -
|
||||
7
oghma-proxy/k8s/namespace.yaml
Normal file
7
oghma-proxy/k8s/namespace.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
app.kubernetes.io/component: gaming
|
||||
33
oghma-proxy/k8s/secret.yaml
Normal file
33
oghma-proxy/k8s/secret.yaml
Normal file
@@ -0,0 +1,33 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: oghma-proxy-secrets
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Replace with actual values or use external secret management
|
||||
OPENROUTER_API_KEY: "your-openrouter-api-key-here"
|
||||
UPSTREAM_URL: "https://openrouter.ai/api/v1"
|
||||
|
||||
---
|
||||
# Alternative: Use ExternalSecret with Vault/Vaultwarden
|
||||
# apiVersion: external-secrets.io/v1beta1
|
||||
# kind: ExternalSecret
|
||||
# metadata:
|
||||
# name: oghma-proxy-secrets
|
||||
# namespace: nimmersky
|
||||
# spec:
|
||||
# refreshInterval: 1h
|
||||
# secretStoreRef:
|
||||
# name: vault-backend
|
||||
# kind: ClusterSecretStore
|
||||
# target:
|
||||
# name: oghma-proxy-secrets
|
||||
# data:
|
||||
# - secretKey: OPENROUTER_API_KEY
|
||||
# remoteRef:
|
||||
# key: nimmerverse/skyrimnet
|
||||
# property: openrouter_api_key
|
||||
39
oghma-proxy/k8s/service.yaml
Normal file
39
oghma-proxy/k8s/service.yaml
Normal file
@@ -0,0 +1,39 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oghma-proxy
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
ports:
|
||||
- name: http
|
||||
port: 8100
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
|
||||
---
|
||||
# Optional: Expose externally via LoadBalancer or NodePort
|
||||
# for SkyrimNet running on gaming PC outside the cluster
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oghma-proxy-external
|
||||
namespace: nimmersky
|
||||
labels:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
app.kubernetes.io/part-of: nimmerverse
|
||||
spec:
|
||||
type: NodePort
|
||||
selector:
|
||||
app.kubernetes.io/name: oghma-proxy
|
||||
ports:
|
||||
- name: http
|
||||
port: 8100
|
||||
targetPort: http
|
||||
nodePort: 30100 # Access via <node-ip>:30100
|
||||
protocol: TCP
|
||||
54
oghma-proxy/pyproject.toml
Normal file
54
oghma-proxy/pyproject.toml
Normal file
@@ -0,0 +1,54 @@
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "oghma-proxy"
|
||||
version = "0.1.0"
|
||||
description = "RAG Proxy for SkyrimNet - Injects Tamrielic lore into NPC conversations"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
license = "MIT"
|
||||
authors = [
|
||||
{ name = "dafit", email = "dafit@eachpath.local" },
|
||||
]
|
||||
|
||||
dependencies = [
|
||||
"fastapi>=0.109.0",
|
||||
"uvicorn[standard]>=0.27.0",
|
||||
"httpx>=0.26.0",
|
||||
"chromadb>=0.4.22",
|
||||
"pydantic>=2.5.0",
|
||||
"pydantic-settings>=2.1.0",
|
||||
"pyyaml>=6.0.1",
|
||||
"structlog>=24.1.0",
|
||||
"psycopg[binary]>=3.1.0", # For phoebe logging
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-asyncio>=0.23.0",
|
||||
"httpx>=0.26.0",
|
||||
"ruff>=0.1.0",
|
||||
]
|
||||
ingest = [
|
||||
"pandas>=2.1.0",
|
||||
"gspread>=5.12.0",
|
||||
"sentence-transformers>=2.2.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
oghma-proxy = "oghma_proxy.main:main"
|
||||
oghma-ingest = "oghma_proxy.ingest:main"
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py311"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "N", "W", "UP"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
3
oghma-proxy/src/oghma_proxy/__init__.py
Normal file
3
oghma-proxy/src/oghma_proxy/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
"""Oghma RAG Proxy - Lore enrichment for SkyrimNet."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
147
oghma-proxy/src/oghma_proxy/extractor.py
Normal file
147
oghma-proxy/src/oghma_proxy/extractor.py
Normal file
@@ -0,0 +1,147 @@
|
||||
"""NPC Profile Extractor - Parses SkyrimNet prompts to extract NPC context."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import structlog
|
||||
|
||||
from .models import NPCProfile
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
|
||||
class NPCExtractor:
|
||||
"""Extracts NPC profile information from SkyrimNet prompts."""
|
||||
|
||||
# Regex patterns for extraction
|
||||
PATTERNS = {
|
||||
# Character bio header
|
||||
"bio_header": re.compile(
|
||||
r"## (?P<name>[\w\s'-]+) Bio\s*\n"
|
||||
r"- Gender: (?P<gender>\w+)\s*\n"
|
||||
r"- Race: (?P<race>[\w\s]+)",
|
||||
re.MULTILINE,
|
||||
),
|
||||
# Alternative role description
|
||||
"role_intro": re.compile(
|
||||
r"You are (?P<name>[^,\n]+),?\s*(?:a |an )?(?P<descriptor>[^.\n]+)",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Faction membership
|
||||
"faction": re.compile(
|
||||
r"(?:member of|belongs to|joined|part of) (?:the )?(?P<faction>[\w\s]+?)(?:\.|,|\n|$)",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Location mentions
|
||||
"location": re.compile(
|
||||
r"(?:in|at|near|from) (?P<location>Whiterun|Windhelm|Solitude|Riften|"
|
||||
r"Markarth|Morthal|Dawnstar|Winterhold|Falkreath|Riverwood|Rorikstead|"
|
||||
r"Ivarstead|Solstheim|Raven Rock)",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
# Profession/occupation
|
||||
"occupation": re.compile(
|
||||
r"(?:works as|profession:|occupation:|is a|as a) (?P<profession>[\w\s]+?)(?:\.|,|\n|$)",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
}
|
||||
|
||||
# Known professions for fuzzy matching
|
||||
KNOWN_PROFESSIONS = {
|
||||
"priest", "priestess", "mage", "wizard", "scholar", "blacksmith",
|
||||
"guard", "soldier", "warrior", "thief", "merchant", "innkeeper",
|
||||
"hunter", "farmer", "peasant", "noble", "jarl", "bard", "alchemist",
|
||||
"healer", "assassin", "spy", "courier", "carriage driver", "fisherman",
|
||||
"miller", "brewer", "smith", "armorer", "fletcher", "jeweler",
|
||||
}
|
||||
|
||||
def extract(self, messages: list[dict]) -> NPCProfile:
|
||||
"""Extract NPC profile from chat messages."""
|
||||
# Combine all message content for analysis
|
||||
full_text = "\n".join(
|
||||
msg.get("content", "") for msg in messages if msg.get("content")
|
||||
)
|
||||
|
||||
profile = NPCProfile()
|
||||
|
||||
# Try bio header first (most reliable)
|
||||
if match := self.PATTERNS["bio_header"].search(full_text):
|
||||
profile.name = match.group("name").strip()
|
||||
profile.gender = match.group("gender").strip()
|
||||
profile.race = match.group("race").strip()
|
||||
logger.debug("Extracted from bio header", name=profile.name, race=profile.race)
|
||||
|
||||
# Fallback to role intro
|
||||
elif match := self.PATTERNS["role_intro"].search(full_text):
|
||||
profile.name = match.group("name").strip()
|
||||
descriptor = match.group("descriptor")
|
||||
# Try to parse race from descriptor
|
||||
profile.race = self._extract_race_from_descriptor(descriptor)
|
||||
logger.debug("Extracted from role intro", name=profile.name)
|
||||
|
||||
# Extract location
|
||||
if match := self.PATTERNS["location"].search(full_text):
|
||||
profile.location = match.group("location").strip()
|
||||
|
||||
# Extract factions
|
||||
for match in self.PATTERNS["faction"].finditer(full_text):
|
||||
faction = match.group("faction").strip()
|
||||
if faction and faction not in profile.factions:
|
||||
profile.factions.append(faction)
|
||||
|
||||
# Extract profession
|
||||
if match := self.PATTERNS["occupation"].search(full_text):
|
||||
profession = match.group("profession").strip().lower()
|
||||
# Validate against known professions
|
||||
for known in self.KNOWN_PROFESSIONS:
|
||||
if known in profession:
|
||||
profile.profession = known
|
||||
break
|
||||
|
||||
# Compute knowledge classes
|
||||
profile.compute_knowledge_classes()
|
||||
|
||||
logger.info(
|
||||
"Extracted NPC profile",
|
||||
name=profile.name,
|
||||
race=profile.race,
|
||||
profession=profile.profession,
|
||||
factions=profile.factions,
|
||||
location=profile.location,
|
||||
knowledge_classes=profile.knowledge_classes,
|
||||
education_level=profile.education_level.value,
|
||||
)
|
||||
|
||||
return profile
|
||||
|
||||
def _extract_race_from_descriptor(self, descriptor: str) -> str:
|
||||
"""Try to extract race from a descriptor string."""
|
||||
races = [
|
||||
"Nord", "Dunmer", "Dark Elf", "Altmer", "High Elf",
|
||||
"Bosmer", "Wood Elf", "Argonian", "Khajiit", "Breton",
|
||||
"Redguard", "Orsimer", "Orc", "Imperial",
|
||||
]
|
||||
descriptor_lower = descriptor.lower()
|
||||
for race in races:
|
||||
if race.lower() in descriptor_lower:
|
||||
# Normalize to single-word form
|
||||
return race.replace(" ", "")
|
||||
return "Unknown"
|
||||
|
||||
def extract_conversation_context(self, messages: list[dict]) -> str:
|
||||
"""Extract the current conversation topic for RAG query."""
|
||||
# Get the last few user/assistant exchanges
|
||||
recent_content = []
|
||||
for msg in reversed(messages[-6:]):
|
||||
content = msg.get("content", "")
|
||||
if content and msg.get("role") in ("user", "assistant"):
|
||||
# Skip very long content (likely system prompts)
|
||||
if len(content) < 500:
|
||||
recent_content.append(content)
|
||||
|
||||
if not recent_content:
|
||||
return ""
|
||||
|
||||
# Combine recent conversation as the query context
|
||||
return " ".join(reversed(recent_content[-3:]))
|
||||
444
oghma-proxy/src/oghma_proxy/ingest.py
Normal file
444
oghma-proxy/src/oghma_proxy/ingest.py
Normal file
@@ -0,0 +1,444 @@
|
||||
"""Oghma Data Ingestion - Loads CHIM's Oghma lore into ChromaDB."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import io
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from typing import Iterator
|
||||
|
||||
import chromadb
|
||||
import httpx
|
||||
import structlog
|
||||
from chromadb.config import Settings
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
# Google Sheet ID for CHIM's Oghma Infinium
|
||||
OGHMA_SHEET_ID = "1dcfctU-iOqprwy2BOc7___4Awteczgdlv8886KalPsQ"
|
||||
|
||||
|
||||
def discover_sheet_gids(sheet_id: str) -> dict[str, str]:
|
||||
"""
|
||||
Discover actual gids for all sheets by parsing the HTML page.
|
||||
|
||||
Returns:
|
||||
Dict mapping sheet name to gid
|
||||
"""
|
||||
url = f"https://docs.google.com/spreadsheets/d/{sheet_id}/htmlview"
|
||||
|
||||
with httpx.Client(follow_redirects=True, timeout=30.0) as client:
|
||||
response = client.get(url)
|
||||
response.raise_for_status()
|
||||
html = response.text
|
||||
|
||||
sheet_gids = {}
|
||||
|
||||
# Pattern 1: items.push({name: "Sheet Name", ...gid: "12345"...})
|
||||
# This is the format Google Sheets uses in the htmlview page
|
||||
for match in re.finditer(
|
||||
r'items\.push\(\{name:\s*"([^"]+)"[^}]*gid:\s*"(\d+)"',
|
||||
html
|
||||
):
|
||||
name = match.group(1)
|
||||
gid = match.group(2)
|
||||
sheet_gids[name] = gid
|
||||
|
||||
# Pattern 2: Also check for gid in URL patterns as backup
|
||||
# ...gid=12345", gid: "12345"...
|
||||
if not sheet_gids:
|
||||
for match in re.finditer(r'gid=(\d+)"[^}]*gid:\s*"(\d+)"', html):
|
||||
gid = match.group(2)
|
||||
# Try to find associated name nearby
|
||||
# This is a fallback pattern
|
||||
|
||||
# Pattern 3: Look for the sheet tabs JSON structure
|
||||
for match in re.finditer(
|
||||
r'\{name:\s*"([^"]+)"[^}]*gid:\s*"(\d+)"[^}]*\}',
|
||||
html
|
||||
):
|
||||
name = match.group(1)
|
||||
gid = match.group(2)
|
||||
if name not in sheet_gids:
|
||||
sheet_gids[name] = gid
|
||||
|
||||
logger.info("Discovered sheets", count=len(sheet_gids), sheets=list(sheet_gids.keys())[:10])
|
||||
return sheet_gids
|
||||
|
||||
|
||||
def fetch_sheet_csv(sheet_id: str, gid: str, sheet_name: str = "") -> str:
|
||||
"""Fetch a Google Sheet as CSV."""
|
||||
url = f"https://docs.google.com/spreadsheets/d/{sheet_id}/export?format=csv&gid={gid}"
|
||||
|
||||
with httpx.Client(follow_redirects=True, timeout=60.0) as client:
|
||||
response = client.get(url)
|
||||
if response.status_code == 400:
|
||||
logger.warning("Sheet fetch failed with 400", sheet=sheet_name, gid=gid)
|
||||
raise httpx.HTTPStatusError(
|
||||
f"Failed to fetch sheet {sheet_name}",
|
||||
request=response.request,
|
||||
response=response,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.text
|
||||
|
||||
|
||||
def parse_csv(csv_text: str) -> Iterator[dict]:
|
||||
"""Parse CSV text into rows."""
|
||||
import csv
|
||||
|
||||
reader = csv.DictReader(io.StringIO(csv_text))
|
||||
for row in reader:
|
||||
yield row
|
||||
|
||||
|
||||
def categorize_sheet(sheet_name: str) -> str | None:
|
||||
"""
|
||||
Determine category for a sheet and whether to process it.
|
||||
|
||||
Returns category name or None if sheet should be skipped.
|
||||
"""
|
||||
# Normalize escaped slashes from JSON
|
||||
normalized_name = sheet_name.replace("\\/", "/").replace("\\", "")
|
||||
|
||||
# Sheets to process and their categories
|
||||
sheet_categories = {
|
||||
"Figures/Gods": "figures_gods",
|
||||
"Artifacts": "artifacts",
|
||||
"Armor and Weapons": "armor_weapons",
|
||||
"Items": "items",
|
||||
"Spells": "spells",
|
||||
"Creatures": "creatures",
|
||||
"Groups/Lore/Books": "groups_lore",
|
||||
"Dynamic Oghma": "dynamic",
|
||||
"Visual Descriptions": "visual",
|
||||
}
|
||||
|
||||
# Check direct match
|
||||
if normalized_name in sheet_categories:
|
||||
return sheet_categories[normalized_name]
|
||||
|
||||
# Check location sheets - handles both "Locations - Whiterun" and "Locations (Whiterun)"
|
||||
if normalized_name.startswith("Locations"):
|
||||
# Try pattern: "Locations (Whiterun)"
|
||||
match = re.match(r"Locations\s*[\(\-]\s*([^\)]+)\)?", normalized_name)
|
||||
if match:
|
||||
hold = match.group(1).strip().lower().replace(" ", "_")
|
||||
return f"locations_{hold}"
|
||||
|
||||
# Skip meta/reference sheets
|
||||
skip_sheets = ["Project Oghma", "Knowledge Classes Reference", "Vanilla NPCS", "Template"]
|
||||
if any(skip in normalized_name for skip in skip_sheets):
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def ingest_oghma(
|
||||
chromadb_host: str = "iris-dev.eachpath.local",
|
||||
chromadb_port: int = 35000,
|
||||
dry_run: bool = False,
|
||||
) -> dict:
|
||||
"""
|
||||
Ingest all Oghma sheets into ChromaDB.
|
||||
|
||||
Returns:
|
||||
Statistics about ingestion
|
||||
"""
|
||||
stats = {
|
||||
"sheets_processed": 0,
|
||||
"sheets_skipped": 0,
|
||||
"lore_entries": 0,
|
||||
"basic_entries": 0,
|
||||
"visual_entries": 0,
|
||||
"errors": [],
|
||||
}
|
||||
|
||||
# Discover actual sheet gids
|
||||
logger.info("Discovering sheet gids...")
|
||||
try:
|
||||
sheet_gids = discover_sheet_gids(OGHMA_SHEET_ID)
|
||||
except Exception as e:
|
||||
logger.error("Failed to discover sheets", error=str(e))
|
||||
# Fallback to known gids (manually discovered)
|
||||
sheet_gids = {
|
||||
"Figures/Gods": "0",
|
||||
# Add more as we discover them
|
||||
}
|
||||
|
||||
if not sheet_gids:
|
||||
logger.error("No sheets discovered!")
|
||||
return stats
|
||||
|
||||
if not dry_run:
|
||||
client = chromadb.HttpClient(
|
||||
host=chromadb_host,
|
||||
port=chromadb_port,
|
||||
settings=Settings(anonymized_telemetry=False),
|
||||
)
|
||||
|
||||
# Get or create collections
|
||||
collection_lore = client.get_or_create_collection(
|
||||
name="oghma_lore",
|
||||
metadata={"description": "Full Tamrielic lore for educated NPCs"},
|
||||
)
|
||||
collection_basic = client.get_or_create_collection(
|
||||
name="oghma_basic",
|
||||
metadata={"description": "Basic Tamrielic lore for commoners"},
|
||||
)
|
||||
collection_visual = client.get_or_create_collection(
|
||||
name="oghma_visual",
|
||||
metadata={"description": "Visual descriptions for Omnisight perception"},
|
||||
)
|
||||
|
||||
logger.info("Connected to ChromaDB", host=chromadb_host, port=chromadb_port)
|
||||
else:
|
||||
logger.info("DRY RUN - not connecting to ChromaDB")
|
||||
collection_lore = None
|
||||
collection_basic = None
|
||||
collection_visual = None
|
||||
|
||||
for sheet_name, gid in sheet_gids.items():
|
||||
category = categorize_sheet(sheet_name)
|
||||
|
||||
if category is None:
|
||||
logger.debug("Skipping sheet", sheet=sheet_name)
|
||||
stats["sheets_skipped"] += 1
|
||||
continue
|
||||
|
||||
logger.info("Processing sheet", sheet=sheet_name, gid=gid, category=category)
|
||||
|
||||
# Rate limit to avoid Google blocking
|
||||
time.sleep(1.0)
|
||||
|
||||
try:
|
||||
csv_text = fetch_sheet_csv(OGHMA_SHEET_ID, gid, sheet_name)
|
||||
rows = list(parse_csv(csv_text))
|
||||
|
||||
if not rows:
|
||||
logger.warning("Empty sheet", sheet=sheet_name)
|
||||
continue
|
||||
|
||||
# Check if this sheet has the expected columns
|
||||
sample_row = rows[0]
|
||||
is_visual_sheet = category == "visual"
|
||||
|
||||
# Visual Descriptions has different schema: baseid, name, description
|
||||
if is_visual_sheet:
|
||||
if "name" not in sample_row or "description" not in sample_row:
|
||||
logger.warning("Visual sheet missing expected columns", sheet=sheet_name, columns=list(sample_row.keys())[:5])
|
||||
continue
|
||||
elif "topic" not in sample_row:
|
||||
logger.warning("Sheet missing 'topic' column", sheet=sheet_name, columns=list(sample_row.keys())[:5])
|
||||
continue
|
||||
|
||||
lore_docs = []
|
||||
lore_ids = []
|
||||
lore_metadatas = []
|
||||
|
||||
basic_docs = []
|
||||
basic_ids = []
|
||||
basic_metadatas = []
|
||||
|
||||
# Visual descriptions - universal perception (Omnisight)
|
||||
visual_docs = []
|
||||
visual_ids = []
|
||||
visual_metadatas = []
|
||||
|
||||
# Track seen IDs to handle duplicates
|
||||
seen_ids = set()
|
||||
duplicates_skipped = 0
|
||||
|
||||
for row in rows:
|
||||
# Handle visual descriptions separately
|
||||
if is_visual_sheet:
|
||||
name = row.get("name", "").strip()
|
||||
description = row.get("description", "").strip()
|
||||
baseid = row.get("baseid", "").strip()
|
||||
|
||||
# Clean up Excel scientific notation for zero (0.00E+00)
|
||||
if baseid and ("E+" in baseid or "E-" in baseid):
|
||||
try:
|
||||
if float(baseid) == 0:
|
||||
baseid = ""
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
if name and description:
|
||||
# Use baseid:name for uniqueness, fall back to name only
|
||||
doc_id = f"visual:{baseid}:{name}" if baseid else f"visual:{name}"
|
||||
|
||||
if doc_id in seen_ids:
|
||||
duplicates_skipped += 1
|
||||
continue
|
||||
seen_ids.add(doc_id)
|
||||
|
||||
visual_docs.append(description)
|
||||
visual_ids.append(doc_id)
|
||||
visual_metadatas.append({
|
||||
"name": name,
|
||||
"baseid": baseid,
|
||||
"category": "visual",
|
||||
"sheet": sheet_name,
|
||||
})
|
||||
continue
|
||||
|
||||
topic = row.get("topic", "").strip()
|
||||
if not topic:
|
||||
continue
|
||||
|
||||
# Full lore entry
|
||||
topic_desc = row.get("topic_desc", "").strip()
|
||||
if topic_desc:
|
||||
lore_id = f"{category}:{topic}"
|
||||
|
||||
if lore_id in seen_ids:
|
||||
duplicates_skipped += 1
|
||||
else:
|
||||
seen_ids.add(lore_id)
|
||||
knowledge_classes = row.get("knowledge_class", "").strip()
|
||||
|
||||
lore_docs.append(topic_desc)
|
||||
lore_ids.append(lore_id)
|
||||
lore_metadatas.append({
|
||||
"topic": topic,
|
||||
"category": category,
|
||||
"sheet": sheet_name,
|
||||
"knowledge_classes": knowledge_classes,
|
||||
"tags": row.get("tags", "").strip(),
|
||||
})
|
||||
|
||||
# Basic lore entry
|
||||
topic_desc_basic = row.get("topic_desc_basic", "").strip()
|
||||
if topic_desc_basic:
|
||||
basic_id = f"{category}:{topic}:basic"
|
||||
|
||||
if basic_id in seen_ids:
|
||||
duplicates_skipped += 1
|
||||
else:
|
||||
seen_ids.add(basic_id)
|
||||
knowledge_classes_basic = row.get("knowledge_class_basic", "").strip()
|
||||
|
||||
basic_docs.append(topic_desc_basic)
|
||||
basic_ids.append(basic_id)
|
||||
basic_metadatas.append({
|
||||
"topic": topic,
|
||||
"category": category,
|
||||
"sheet": sheet_name,
|
||||
"knowledge_classes": knowledge_classes_basic,
|
||||
"tags": row.get("tags", "").strip(),
|
||||
})
|
||||
|
||||
# Batch insert to ChromaDB
|
||||
if not dry_run:
|
||||
if lore_docs:
|
||||
collection_lore.upsert(
|
||||
documents=lore_docs,
|
||||
ids=lore_ids,
|
||||
metadatas=lore_metadatas,
|
||||
)
|
||||
if basic_docs:
|
||||
collection_basic.upsert(
|
||||
documents=basic_docs,
|
||||
ids=basic_ids,
|
||||
metadatas=basic_metadatas,
|
||||
)
|
||||
if visual_docs:
|
||||
collection_visual.upsert(
|
||||
documents=visual_docs,
|
||||
ids=visual_ids,
|
||||
metadatas=visual_metadatas,
|
||||
)
|
||||
|
||||
stats["sheets_processed"] += 1
|
||||
stats["lore_entries"] += len(lore_docs)
|
||||
stats["basic_entries"] += len(basic_docs)
|
||||
stats["visual_entries"] += len(visual_docs)
|
||||
|
||||
if duplicates_skipped > 0:
|
||||
logger.debug("Duplicates skipped", sheet=sheet_name, count=duplicates_skipped)
|
||||
|
||||
logger.info(
|
||||
"Sheet processed",
|
||||
sheet=sheet_name,
|
||||
rows=len(rows),
|
||||
lore_entries=len(lore_docs),
|
||||
basic_entries=len(basic_docs),
|
||||
visual_entries=len(visual_docs),
|
||||
)
|
||||
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error("HTTP error fetching sheet", sheet=sheet_name, status=e.response.status_code)
|
||||
stats["errors"].append({"sheet": sheet_name, "error": f"HTTP {e.response.status_code}"})
|
||||
except Exception as e:
|
||||
logger.error("Failed to process sheet", sheet=sheet_name, error=str(e))
|
||||
stats["errors"].append({"sheet": sheet_name, "error": str(e)})
|
||||
|
||||
logger.info(
|
||||
"Ingestion complete",
|
||||
sheets_processed=stats["sheets_processed"],
|
||||
sheets_skipped=stats["sheets_skipped"],
|
||||
lore_entries=stats["lore_entries"],
|
||||
basic_entries=stats["basic_entries"],
|
||||
visual_entries=stats["visual_entries"],
|
||||
errors=len(stats["errors"]),
|
||||
)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Ingest CHIM's Oghma Infinium lore into ChromaDB"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
default="iris-dev.eachpath.local",
|
||||
help="ChromaDB host",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=35000,
|
||||
help="ChromaDB port",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Fetch and parse sheets without writing to ChromaDB",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Configure logging
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.dev.ConsoleRenderer(),
|
||||
],
|
||||
)
|
||||
|
||||
try:
|
||||
stats = ingest_oghma(
|
||||
chromadb_host=args.host,
|
||||
chromadb_port=args.port,
|
||||
dry_run=args.dry_run,
|
||||
)
|
||||
|
||||
if stats["errors"]:
|
||||
logger.warning("Ingestion completed with errors", errors=stats["errors"])
|
||||
# Don't exit 1 if we processed some sheets successfully
|
||||
if stats["sheets_processed"] == 0:
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Ingestion failed", error=str(e))
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
153
oghma-proxy/src/oghma_proxy/injector.py
Normal file
153
oghma-proxy/src/oghma_proxy/injector.py
Normal file
@@ -0,0 +1,153 @@
|
||||
"""Lore Injector - Injects retrieved lore into SkyrimNet prompts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import structlog
|
||||
|
||||
from .models import InjectionResult, LoreEntry, NPCProfile
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
|
||||
class LoreInjector:
|
||||
"""Injects Oghma lore into SkyrimNet chat messages."""
|
||||
|
||||
DEFAULT_TEMPLATE = """
|
||||
## Relevant Lore Knowledge
|
||||
|
||||
Based on your background as a {race} {profession} in {location}, you would know:
|
||||
|
||||
{lore_items}
|
||||
|
||||
Note: Reference this knowledge naturally when relevant to the conversation. Do not recite it.
|
||||
"""
|
||||
|
||||
def __init__(self, template: str | None = None, position: str = "after_bio"):
|
||||
"""
|
||||
Initialize injector.
|
||||
|
||||
Args:
|
||||
template: Jinja-style template for injection block
|
||||
position: Where to inject - 'after_bio', 'before_conversation', 'system_suffix'
|
||||
"""
|
||||
self.template = template or self.DEFAULT_TEMPLATE
|
||||
self.position = position
|
||||
|
||||
def inject(
|
||||
self,
|
||||
messages: list[dict],
|
||||
npc_profile: NPCProfile,
|
||||
lore_entries: list[LoreEntry],
|
||||
query_time_ms: float,
|
||||
) -> tuple[list[dict], InjectionResult]:
|
||||
"""
|
||||
Inject lore into chat messages.
|
||||
|
||||
Args:
|
||||
messages: Original chat messages
|
||||
npc_profile: Extracted NPC profile
|
||||
lore_entries: Retrieved lore entries
|
||||
query_time_ms: Time taken for retrieval
|
||||
|
||||
Returns:
|
||||
Tuple of (modified messages, injection result)
|
||||
"""
|
||||
if not lore_entries:
|
||||
return messages, InjectionResult(
|
||||
npc_profile=npc_profile,
|
||||
lore_entries=[],
|
||||
injection_text="",
|
||||
query_time_ms=query_time_ms,
|
||||
)
|
||||
|
||||
# Build injection text
|
||||
injection_text = self._build_injection_text(npc_profile, lore_entries)
|
||||
|
||||
# Clone messages to avoid modifying original
|
||||
modified_messages = [dict(msg) for msg in messages]
|
||||
|
||||
# Find injection point
|
||||
injected = False
|
||||
for i, msg in enumerate(modified_messages):
|
||||
if msg.get("role") == "system":
|
||||
content = msg.get("content", "")
|
||||
|
||||
if self.position == "after_bio":
|
||||
# Inject after character bio section
|
||||
bio_markers = ["## Background", "## Personality", "## Speech Style"]
|
||||
for marker in bio_markers:
|
||||
if marker in content:
|
||||
# Insert before this section
|
||||
idx = content.index(marker)
|
||||
modified_messages[i]["content"] = (
|
||||
content[:idx] + injection_text + "\n\n" + content[idx:]
|
||||
)
|
||||
injected = True
|
||||
break
|
||||
|
||||
elif self.position == "system_suffix":
|
||||
# Append to end of system message
|
||||
modified_messages[i]["content"] = content + "\n\n" + injection_text
|
||||
injected = True
|
||||
|
||||
if injected:
|
||||
break
|
||||
|
||||
# Fallback: prepend to first user message if no system message found
|
||||
if not injected and self.position == "before_conversation":
|
||||
for i, msg in enumerate(modified_messages):
|
||||
if msg.get("role") == "user":
|
||||
content = msg.get("content", "")
|
||||
modified_messages[i]["content"] = (
|
||||
f"[Context for the NPC you're speaking with]\n{injection_text}\n\n"
|
||||
f"[Player speaks]\n{content}"
|
||||
)
|
||||
injected = True
|
||||
break
|
||||
|
||||
if injected:
|
||||
logger.info(
|
||||
"Injected lore",
|
||||
npc_name=npc_profile.name,
|
||||
entries_count=len(lore_entries),
|
||||
position=self.position,
|
||||
)
|
||||
else:
|
||||
logger.warning("Could not find injection point", position=self.position)
|
||||
|
||||
result = InjectionResult(
|
||||
npc_profile=npc_profile,
|
||||
lore_entries=lore_entries,
|
||||
injection_text=injection_text if injected else "",
|
||||
query_time_ms=query_time_ms,
|
||||
)
|
||||
|
||||
return modified_messages, result
|
||||
|
||||
def _build_injection_text(
|
||||
self,
|
||||
npc_profile: NPCProfile,
|
||||
lore_entries: list[LoreEntry],
|
||||
) -> str:
|
||||
"""Build the injection text block."""
|
||||
# Build lore items list
|
||||
lore_items = []
|
||||
for entry in lore_entries:
|
||||
# Truncate very long entries
|
||||
content = entry.content
|
||||
if len(content) > 300:
|
||||
content = content[:297] + "..."
|
||||
lore_items.append(f"- **{entry.topic}**: {content}")
|
||||
|
||||
lore_items_text = "\n".join(lore_items)
|
||||
|
||||
# Fill template
|
||||
injection_text = self.template.format(
|
||||
race=npc_profile.race or "person",
|
||||
profession=npc_profile.profession or "citizen",
|
||||
location=npc_profile.location or "Skyrim",
|
||||
lore_items=lore_items_text,
|
||||
name=npc_profile.name,
|
||||
)
|
||||
|
||||
return injection_text.strip()
|
||||
291
oghma-proxy/src/oghma_proxy/main.py
Normal file
291
oghma-proxy/src/oghma_proxy/main.py
Normal file
@@ -0,0 +1,291 @@
|
||||
"""Oghma RAG Proxy - Main FastAPI Application."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import structlog
|
||||
import yaml
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from .extractor import NPCExtractor
|
||||
from .injector import LoreInjector
|
||||
from .models import ChatCompletionRequest
|
||||
from .retriever import OghmaRetriever
|
||||
|
||||
# Configure structured logging
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.JSONRenderer(),
|
||||
],
|
||||
wrapper_class=structlog.stdlib.BoundLogger,
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
)
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
|
||||
def load_config(config_path: str = "config.yaml") -> dict:
|
||||
"""Load configuration from YAML file."""
|
||||
# Try local config first, then default
|
||||
for path in ["config.local.yaml", config_path]:
|
||||
if os.path.exists(path):
|
||||
with open(path) as f:
|
||||
config = yaml.safe_load(f)
|
||||
logger.info("Loaded config", path=path)
|
||||
return config
|
||||
return {}
|
||||
|
||||
|
||||
# Global instances
|
||||
config = load_config()
|
||||
extractor = NPCExtractor()
|
||||
retriever: OghmaRetriever | None = None
|
||||
injector: LoreInjector | None = None
|
||||
http_client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Application lifespan - setup and teardown."""
|
||||
global retriever, injector, http_client
|
||||
|
||||
# Initialize components
|
||||
chroma_config = config.get("chromadb", {})
|
||||
retriever = OghmaRetriever(
|
||||
host=chroma_config.get("host", "iris-dev.eachpath.local"),
|
||||
port=chroma_config.get("port", 35000),
|
||||
collection_lore=chroma_config.get("collection_lore", "oghma_lore"),
|
||||
collection_basic=chroma_config.get("collection_basic", "oghma_basic"),
|
||||
max_results=config.get("retrieval", {}).get("max_results", 5),
|
||||
min_score=config.get("retrieval", {}).get("min_score", 0.55),
|
||||
)
|
||||
|
||||
injection_config = config.get("injection", {})
|
||||
injector = LoreInjector(
|
||||
template=injection_config.get("template"),
|
||||
position=injection_config.get("position", "after_bio"),
|
||||
)
|
||||
|
||||
upstream_config = config.get("upstream", {})
|
||||
http_client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(
|
||||
connect=10.0,
|
||||
read=upstream_config.get("timeout", 120.0),
|
||||
write=30.0,
|
||||
pool=10.0,
|
||||
),
|
||||
limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Oghma RAG Proxy started",
|
||||
upstream_url=upstream_config.get("url", ""),
|
||||
chromadb_host=chroma_config.get("host"),
|
||||
)
|
||||
|
||||
yield
|
||||
|
||||
# Cleanup
|
||||
if http_client:
|
||||
await http_client.aclose()
|
||||
logger.info("Oghma RAG Proxy stopped")
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Oghma RAG Proxy",
|
||||
description="RAG Proxy for SkyrimNet - Injects Tamrielic lore into NPC conversations",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint."""
|
||||
chromadb_healthy = retriever.health_check() if retriever else False
|
||||
|
||||
return {
|
||||
"status": "healthy" if chromadb_healthy else "degraded",
|
||||
"components": {
|
||||
"proxy": "healthy",
|
||||
"chromadb": "healthy" if chromadb_healthy else "unhealthy",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Debug: track recent RAG operations
|
||||
_recent_rag_ops = []
|
||||
|
||||
@app.get("/stats")
|
||||
async def get_stats():
|
||||
"""Get proxy statistics."""
|
||||
return {
|
||||
"version": "0.1.0",
|
||||
"injection_enabled": config.get("injection", {}).get("enabled", True),
|
||||
"upstream_url": config.get("upstream", {}).get("url", ""),
|
||||
}
|
||||
|
||||
@app.get("/debug/rag")
|
||||
async def debug_rag():
|
||||
"""Debug endpoint to see recent RAG operations."""
|
||||
return {"recent_operations": _recent_rag_ops[-20:]}
|
||||
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat_completions(request: Request):
|
||||
"""
|
||||
Proxy chat completions with RAG enrichment.
|
||||
|
||||
This endpoint intercepts OpenRouter-compatible requests,
|
||||
enriches them with relevant Tamrielic lore, and forwards
|
||||
to the upstream LLM.
|
||||
"""
|
||||
start_time = time.perf_counter()
|
||||
|
||||
# Parse request body
|
||||
body = await request.json()
|
||||
messages = body.get("messages", [])
|
||||
stream = body.get("stream", False)
|
||||
|
||||
# Extract NPC profile from messages
|
||||
npc_profile = extractor.extract(messages)
|
||||
|
||||
# Get conversation context for RAG query
|
||||
context = extractor.extract_conversation_context(messages)
|
||||
|
||||
# Retrieve relevant lore
|
||||
lore_entries = []
|
||||
query_time_ms = 0.0
|
||||
if (
|
||||
retriever
|
||||
and injector
|
||||
and config.get("injection", {}).get("enabled", True)
|
||||
and context
|
||||
):
|
||||
lore_entries, query_time_ms = retriever.retrieve(context, npc_profile)
|
||||
|
||||
# Inject lore into messages
|
||||
if lore_entries:
|
||||
messages, injection_result = injector.inject(
|
||||
messages,
|
||||
npc_profile,
|
||||
lore_entries,
|
||||
query_time_ms,
|
||||
)
|
||||
body["messages"] = messages
|
||||
|
||||
# Track for debug endpoint
|
||||
_recent_rag_ops.append({
|
||||
"npc": npc_profile.name,
|
||||
"race": npc_profile.race,
|
||||
"query": context[:100] if context else "",
|
||||
"lore_found": len(lore_entries),
|
||||
"topics": [e.topic for e in lore_entries[:3]],
|
||||
"time_ms": round(query_time_ms, 2),
|
||||
})
|
||||
if len(_recent_rag_ops) > 50:
|
||||
_recent_rag_ops.pop(0)
|
||||
|
||||
# Forward to upstream
|
||||
upstream_config = config.get("upstream", {})
|
||||
upstream_url = upstream_config.get("url", "https://openrouter.ai/api/v1")
|
||||
api_key = upstream_config.get("api_key", os.environ.get("OPENROUTER_API_KEY", ""))
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Copy relevant headers from original request
|
||||
for header in ["HTTP-Referer", "X-Title"]:
|
||||
if value := request.headers.get(header):
|
||||
headers[header] = value
|
||||
|
||||
try:
|
||||
if stream:
|
||||
# Streaming response
|
||||
return StreamingResponse(
|
||||
stream_upstream(f"{upstream_url}/chat/completions", headers, body),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
else:
|
||||
# Regular response
|
||||
response = await http_client.post(
|
||||
f"{upstream_url}/chat/completions",
|
||||
json=body,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
total_time = (time.perf_counter() - start_time) * 1000
|
||||
logger.info(
|
||||
"Request completed",
|
||||
npc_name=npc_profile.name,
|
||||
lore_entries=len(lore_entries),
|
||||
rag_time_ms=round(query_time_ms, 2),
|
||||
total_time_ms=round(total_time, 2),
|
||||
)
|
||||
|
||||
return response.json()
|
||||
|
||||
except httpx.HTTPError as e:
|
||||
logger.error("Upstream request failed", error=str(e))
|
||||
raise HTTPException(status_code=502, detail=f"Upstream error: {e}")
|
||||
|
||||
|
||||
async def stream_upstream(url: str, headers: dict, body: dict):
|
||||
"""Stream response from upstream."""
|
||||
async with http_client.stream("POST", url, json=body, headers=headers) as response:
|
||||
async for chunk in response.aiter_bytes():
|
||||
yield chunk
|
||||
|
||||
|
||||
@app.post("/v1/completions")
|
||||
async def completions(request: Request):
|
||||
"""Legacy completions endpoint - passthrough."""
|
||||
body = await request.json()
|
||||
|
||||
upstream_config = config.get("upstream", {})
|
||||
upstream_url = upstream_config.get("url", "https://openrouter.ai/api/v1")
|
||||
api_key = upstream_config.get("api_key", os.environ.get("OPENROUTER_API_KEY", ""))
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
response = await http_client.post(
|
||||
f"{upstream_url}/completions",
|
||||
json=body,
|
||||
headers=headers,
|
||||
)
|
||||
return response.json()
|
||||
|
||||
|
||||
def main():
|
||||
"""Run the proxy server."""
|
||||
import uvicorn
|
||||
|
||||
proxy_config = config.get("proxy", {})
|
||||
uvicorn.run(
|
||||
"oghma_proxy.main:app",
|
||||
host=proxy_config.get("host", "0.0.0.0"),
|
||||
port=proxy_config.get("port", 8100),
|
||||
workers=proxy_config.get("workers", 1),
|
||||
log_level="info",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
169
oghma-proxy/src/oghma_proxy/models.py
Normal file
169
oghma-proxy/src/oghma_proxy/models.py
Normal file
@@ -0,0 +1,169 @@
|
||||
"""Data models for Oghma RAG Proxy."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class EducationLevel(str, Enum):
|
||||
"""NPC education level determines lore depth."""
|
||||
|
||||
SCHOLAR = "scholar" # Full lore access
|
||||
COMMONER = "commoner" # Basic summaries only
|
||||
|
||||
|
||||
class NPCProfile(BaseModel):
|
||||
"""Extracted NPC profile from SkyrimNet prompts."""
|
||||
|
||||
name: str = "Unknown"
|
||||
race: str = "Unknown"
|
||||
gender: str = "Unknown"
|
||||
profession: str | None = None
|
||||
factions: list[str] = Field(default_factory=list)
|
||||
location: str | None = None
|
||||
traits: list[str] = Field(default_factory=list)
|
||||
|
||||
# Computed
|
||||
knowledge_classes: list[str] = Field(default_factory=list)
|
||||
education_level: EducationLevel = EducationLevel.COMMONER
|
||||
|
||||
def compute_knowledge_classes(self) -> None:
|
||||
"""Compute knowledge classes from profile attributes."""
|
||||
classes = set()
|
||||
|
||||
# Race-based knowledge
|
||||
race_map = {
|
||||
"nord": ["nord"],
|
||||
"dunmer": ["darkelf", "dunmer"],
|
||||
"altmer": ["highelf", "altmer"],
|
||||
"bosmer": ["woodelf", "bosmer"],
|
||||
"argonian": ["argonian"],
|
||||
"khajiit": ["khajiit"],
|
||||
"breton": ["breton"],
|
||||
"redguard": ["redguard"],
|
||||
"orsimer": ["orc", "orsimer"],
|
||||
"orc": ["orc", "orsimer"],
|
||||
"imperial": ["imperial"],
|
||||
}
|
||||
race_lower = self.race.lower()
|
||||
if race_lower in race_map:
|
||||
classes.update(race_map[race_lower])
|
||||
|
||||
# Profession-based knowledge
|
||||
profession_map = {
|
||||
"priest": ["priest"],
|
||||
"mage": ["mage", "scholar"],
|
||||
"wizard": ["mage", "scholar"],
|
||||
"scholar": ["scholar"],
|
||||
"blacksmith": ["blacksmith"],
|
||||
"guard": ["guard", "warrior"],
|
||||
"soldier": ["warrior", "guard"],
|
||||
"warrior": ["warrior"],
|
||||
"thief": ["thief"],
|
||||
"merchant": ["merchant"],
|
||||
"innkeeper": ["innkeeper"],
|
||||
"hunter": ["hunter"],
|
||||
"farmer": ["peasant"],
|
||||
"peasant": ["peasant"],
|
||||
"noble": ["noble"],
|
||||
"jarl": ["noble"],
|
||||
"bard": ["bard"],
|
||||
"alchemist": ["alchemist"],
|
||||
}
|
||||
if self.profession:
|
||||
prof_lower = self.profession.lower()
|
||||
if prof_lower in profession_map:
|
||||
classes.update(profession_map[prof_lower])
|
||||
|
||||
# Location-based knowledge
|
||||
location_map = {
|
||||
"whiterun": ["whiterun"],
|
||||
"windhelm": ["eastmarch"],
|
||||
"solitude": ["haafingar"],
|
||||
"riften": ["rift"],
|
||||
"markarth": ["reach"],
|
||||
"morthal": ["hjaalmarch"],
|
||||
"dawnstar": ["pale"],
|
||||
"winterhold": ["winterhold"],
|
||||
"falkreath": ["falkreath"],
|
||||
"solstheim": ["solstheim"],
|
||||
}
|
||||
if self.location:
|
||||
loc_lower = self.location.lower()
|
||||
if loc_lower in location_map:
|
||||
classes.update(location_map[loc_lower])
|
||||
|
||||
# Faction-based knowledge
|
||||
faction_map = {
|
||||
"companions": ["companions"],
|
||||
"college of winterhold": ["college", "mage"],
|
||||
"college": ["college", "mage"],
|
||||
"thieves guild": ["thieves"],
|
||||
"dark brotherhood": ["darkbrotherhood"],
|
||||
"stormcloaks": ["stormcloak"],
|
||||
"stormcloak": ["stormcloak"],
|
||||
"imperial legion": ["imperial"],
|
||||
"legion": ["imperial"],
|
||||
"thalmor": ["thalmor"],
|
||||
"dawnguard": ["dawnguard"],
|
||||
"volkihar": ["vampire", "volkihar"],
|
||||
}
|
||||
for faction in self.factions:
|
||||
faction_lower = faction.lower()
|
||||
if faction_lower in faction_map:
|
||||
classes.update(faction_map[faction_lower])
|
||||
|
||||
self.knowledge_classes = list(classes)
|
||||
|
||||
# Determine education level
|
||||
educated_professions = {"mage", "wizard", "scholar", "priest", "noble", "bard"}
|
||||
educated_factions = {"college of winterhold", "thalmor", "college"}
|
||||
|
||||
if self.profession and self.profession.lower() in educated_professions:
|
||||
self.education_level = EducationLevel.SCHOLAR
|
||||
elif any(f.lower() in educated_factions for f in self.factions):
|
||||
self.education_level = EducationLevel.SCHOLAR
|
||||
else:
|
||||
self.education_level = EducationLevel.COMMONER
|
||||
|
||||
|
||||
class LoreEntry(BaseModel):
|
||||
"""A retrieved lore entry from Oghma."""
|
||||
|
||||
topic: str
|
||||
content: str
|
||||
category: str
|
||||
score: float
|
||||
knowledge_classes: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ChatMessage(BaseModel):
|
||||
"""OpenRouter-compatible chat message."""
|
||||
|
||||
role: str
|
||||
content: str
|
||||
name: str | None = None
|
||||
|
||||
|
||||
class ChatCompletionRequest(BaseModel):
|
||||
"""OpenRouter-compatible chat completion request."""
|
||||
|
||||
model: str
|
||||
messages: list[ChatMessage]
|
||||
temperature: float | None = None
|
||||
max_tokens: int | None = None
|
||||
stream: bool = False
|
||||
# Allow additional fields to pass through
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class InjectionResult(BaseModel):
|
||||
"""Result of lore injection."""
|
||||
|
||||
npc_profile: NPCProfile
|
||||
lore_entries: list[LoreEntry]
|
||||
injection_text: str
|
||||
query_time_ms: float
|
||||
173
oghma-proxy/src/oghma_proxy/retriever.py
Normal file
173
oghma-proxy/src/oghma_proxy/retriever.py
Normal file
@@ -0,0 +1,173 @@
|
||||
"""Oghma Lore Retriever - Queries ChromaDB for relevant Tamrielic lore."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from functools import lru_cache
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import chromadb
|
||||
import structlog
|
||||
from chromadb.config import Settings
|
||||
|
||||
from .models import EducationLevel, LoreEntry, NPCProfile
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from chromadb import Collection
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
|
||||
class OghmaRetriever:
|
||||
"""Retrieves relevant lore from Oghma ChromaDB collections."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
host: str = "iris-dev.eachpath.local",
|
||||
port: int = 35000,
|
||||
collection_lore: str = "oghma_lore",
|
||||
collection_basic: str = "oghma_basic",
|
||||
max_results: int = 5,
|
||||
min_score: float = 0.55,
|
||||
):
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.collection_lore_name = collection_lore
|
||||
self.collection_basic_name = collection_basic
|
||||
self.max_results = max_results
|
||||
self.min_score = min_score
|
||||
|
||||
self._client: chromadb.HttpClient | None = None
|
||||
self._collection_lore: Collection | None = None
|
||||
self._collection_basic: Collection | None = None
|
||||
|
||||
def _get_client(self) -> chromadb.HttpClient:
|
||||
"""Get or create ChromaDB client."""
|
||||
if self._client is None:
|
||||
self._client = chromadb.HttpClient(
|
||||
host=self.host,
|
||||
port=self.port,
|
||||
settings=Settings(anonymized_telemetry=False),
|
||||
)
|
||||
logger.info("Connected to ChromaDB", host=self.host, port=self.port)
|
||||
return self._client
|
||||
|
||||
def _get_collection(self, education_level: EducationLevel) -> Collection:
|
||||
"""Get the appropriate collection based on education level."""
|
||||
client = self._get_client()
|
||||
|
||||
if education_level == EducationLevel.SCHOLAR:
|
||||
if self._collection_lore is None:
|
||||
self._collection_lore = client.get_collection(self.collection_lore_name)
|
||||
return self._collection_lore
|
||||
else:
|
||||
if self._collection_basic is None:
|
||||
self._collection_basic = client.get_collection(self.collection_basic_name)
|
||||
return self._collection_basic
|
||||
|
||||
def retrieve(
|
||||
self,
|
||||
query: str,
|
||||
npc_profile: NPCProfile,
|
||||
) -> tuple[list[LoreEntry], float]:
|
||||
"""
|
||||
Retrieve relevant lore entries for an NPC.
|
||||
|
||||
Args:
|
||||
query: Conversation context to search for
|
||||
npc_profile: NPC profile for knowledge filtering
|
||||
|
||||
Returns:
|
||||
Tuple of (lore entries, query time in ms)
|
||||
"""
|
||||
if not query.strip():
|
||||
return [], 0.0
|
||||
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
collection = self._get_collection(npc_profile.education_level)
|
||||
|
||||
# Build metadata filter for knowledge classes
|
||||
# NOTE: Currently disabled because CHIM's Oghma data doesn't have
|
||||
# knowledge_class populated consistently. Enable when data is enriched.
|
||||
where_filter = None
|
||||
# TODO: Re-enable when knowledge_class data is available
|
||||
# if npc_profile.knowledge_classes:
|
||||
# if len(npc_profile.knowledge_classes) == 1:
|
||||
# where_filter = {"knowledge_classes": {"$contains": npc_profile.knowledge_classes[0]}}
|
||||
# else:
|
||||
# where_filter = {
|
||||
# "$or": [
|
||||
# {"knowledge_classes": {"$contains": kc}}
|
||||
# for kc in npc_profile.knowledge_classes
|
||||
# ]
|
||||
# }
|
||||
|
||||
# Query ChromaDB
|
||||
results = collection.query(
|
||||
query_texts=[query],
|
||||
n_results=self.max_results,
|
||||
where=where_filter,
|
||||
include=["documents", "metadatas", "distances"],
|
||||
)
|
||||
|
||||
# Parse results
|
||||
entries = []
|
||||
if results and results["documents"] and results["documents"][0]:
|
||||
for i, doc in enumerate(results["documents"][0]):
|
||||
metadata = results["metadatas"][0][i] if results["metadatas"] else {}
|
||||
distance = results["distances"][0][i] if results["distances"] else 1.0
|
||||
|
||||
# Convert distance to similarity score (ChromaDB uses L2 distance)
|
||||
# Lower distance = higher similarity
|
||||
score = 1.0 / (1.0 + distance)
|
||||
|
||||
if score >= self.min_score:
|
||||
entries.append(
|
||||
LoreEntry(
|
||||
topic=metadata.get("topic", "Unknown"),
|
||||
content=doc,
|
||||
category=metadata.get("category", "Unknown"),
|
||||
score=score,
|
||||
knowledge_classes=metadata.get("knowledge_classes", "").split(","),
|
||||
)
|
||||
)
|
||||
|
||||
query_time = (time.perf_counter() - start_time) * 1000
|
||||
|
||||
logger.info(
|
||||
"Retrieved lore entries",
|
||||
query_preview=query[:100],
|
||||
npc_name=npc_profile.name,
|
||||
education=npc_profile.education_level.value,
|
||||
entries_found=len(entries),
|
||||
query_time_ms=round(query_time, 2),
|
||||
)
|
||||
|
||||
return entries, query_time
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to retrieve lore", error=str(e))
|
||||
query_time = (time.perf_counter() - start_time) * 1000
|
||||
return [], query_time
|
||||
|
||||
def health_check(self) -> bool:
|
||||
"""Check if ChromaDB is reachable."""
|
||||
try:
|
||||
client = self._get_client()
|
||||
client.heartbeat()
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error("ChromaDB health check failed", error=str(e))
|
||||
return False
|
||||
|
||||
|
||||
# Cached retriever instance
|
||||
@lru_cache(maxsize=1)
|
||||
def get_retriever(
|
||||
host: str = "iris-dev.eachpath.local",
|
||||
port: int = 35000,
|
||||
) -> OghmaRetriever:
|
||||
"""Get cached retriever instance."""
|
||||
return OghmaRetriever(host=host, port=port)
|
||||
7
oghma-proxy/usage.txt
Normal file
7
oghma-proxy/usage.txt
Normal file
@@ -0,0 +1,7 @@
|
||||
cd /home/dafit/nimmerverse/nimmersky/oghma-proxy
|
||||
python -m oghma_proxy.main
|
||||
|
||||
Endpoints:
|
||||
- http://localhost:8100/health - Health check
|
||||
- http://localhost:8100/debug/rag - See recent RAG operations
|
||||
- http://localhost:8100/v1/chat/completions - The proxy endpoint (point SkyrimNet here)
|
||||
Reference in New Issue
Block a user