diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fa5aafc --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +#Source +SkyrimNet-GamePlugin-main/ \ No newline at end of file diff --git a/skyrimnet/README.md b/skyrimnet/README.md new file mode 100644 index 0000000..9c1412a --- /dev/null +++ b/skyrimnet/README.md @@ -0,0 +1,54 @@ +# SkyrimNet Architecture Archive + +A persistent reference for understanding the SkyrimNet SKSE plugin and its surrounding mod ecosystem. Maintained by **dafit + chrysalis** as part of the nimmerverse partnership. + +**Last verified pass:** 2026-04-18 (based on the live install, not upstream HEAD) + +--- + +## Why this archive exists + +Earlier sessions iterated on SkyrimNet bug fixes reactively — fix one symptom, expose the next. We hit a coupling bug (gamemaster loop fix → broke action firing for malformed `OPENTRADE` markers) and decided to stop playing whack-a-bug. This archive is the systematic map we build on top, so future-Chrysalis can load context in one read and make informed structural changes instead of guessing. + +## How to use this archive + +1. **Load order:** start with `architecture.md` for the big picture, then jump to the file relevant to your task. +2. **Trust tags:** every claim is tagged `[verified]`, `[hypothesis]`, or `[unknown]`. Don't promote a hypothesis without re-verifying. +3. **Citations:** `path/to/file.ext:line_number` references are how we trace claims. If a path looks stale, it probably is — re-verify rather than assume. +4. **When you fix or learn something new:** update `bugs-and-fixes.md` AND the relevant detail file, AND bump the "Last verified pass" date here. +5. **Don't assume continuity from chat history.** This archive is the source of truth across sessions; conversation context is ephemeral. + +## File index + +| File | Purpose | +|---|---| +| [`architecture.md`](architecture.md) | High-level model: subsystems, the two-plugin split, agent flow diagram | +| [`agent-pipelines.md`](agent-pipelines.md) | Per-agent variants: when they fire, models, endpoints, consumers | +| [`prompt-templates.md`](prompt-templates.md) | Inja syntax, three-layer override, submodule numbering, subdirectory map | +| [`action-system.md`](action-system.md) | The three registration paths, ACTION: parser, two action selectors, drilldown | +| [`config-knobs.md`](config-knobs.md) | Load-bearing YAML settings → code behavior mapping | +| [`bugs-and-fixes.md`](bugs-and-fixes.md) | Running log of bugs we've found and what we did (or didn't do) about them | +| [`open-questions.md`](open-questions.md) | Things still uncertain — needing in-game test or DLL disassembly | + +## Adjacent resources in this directory + +- `logs/` — captured `openrouter_input.log`, `openrouter_output.log`, `conversation_log.log`, `SkyrimNet.log`, `all_traces_*.json`. Use these to verify behavior claims against real traces. +- `SkyrimNet-GamePlugin-main/` — upstream git pull of the open-source side (Papyrus + .esp + headers). The C++ DLL is closed-source, not in this pull. + +## State of knowledge — quick summary + +**[verified]** — agent flow architecture, three action registration paths, three-layer prompt override, Inja syntax, action parser grammar, the two action selectors and their relationship, GM loop bug + fix. + +**[hypothesis]** — exact firing cadence of the GM in continuous mode (appears to be a polling timer ~30s but couldn't isolate the source), `defaults_manifest.json` role (looks like first-run seed schema), whether the native_action_selector always fires after every dialogue response or only conditionally. + +**[unknown]** — full decorator list with signatures (dynamically generated by DLL), what `agentEnabled` vs `enabled` toggle in the gamemaster block actually does differently, whether `gamemaster_scene_planner` ever fires in current config. + +## Conventions + +- Dates are absolute (e.g. `2026-04-18`), never relative (`yesterday`, `last week`). +- File paths are absolute when full-disk, relative-to-this-dir when within the archive. +- The user is `dafit`. The AI partner is `chrysalis`. Identity matters for git attribution. + +--- + +**Philosophy:** *This archive grows by addition and refinement, not by overwrite. Bug-and-fix entries stay even after the bug is gone — they're the only record of why a code path looks the way it does.* diff --git a/skyrimnet/action-system.md b/skyrimnet/action-system.md new file mode 100644 index 0000000..7a1c0ce --- /dev/null +++ b/skyrimnet/action-system.md @@ -0,0 +1,173 @@ +# Action System + +This is the most complex subsystem and the one we've already broken once. Read carefully before editing anything that touches actions. + +## The three registration paths + +`[verified]` from `Source/Scripts/SkyrimNetApi.psc:27-31` (Papyrus path), `SkyrimNet.log:14123-14266` (CustomActionManager loader), `CppAPI/PublicAPI.h:93-97` (C++ path), and on-disk YAML inspection. + +### 1. Papyrus registration + +A mod's Papyrus script calls: + +```papyrus +SkyrimNetApi.RegisterAction( + string actionName, ; UPPERCASED in the registry — "OpenTrade" → "OPENTRADE" + string actionDescription, + string eligibilityScriptName, + string eligibilityFunctionName, + string executionScriptName, + string executionFunctionName, + string triggeringEventTypesCsv, + string actionType, ; "PAPYRUS" or "PAPYRUS_CUSTOM" + int priority, + string parameterSchemaJson, + string customCategory, + string tagsCsv +) +``` + +The C++ DLL keeps a registry of these. When the LLM emits `ACTION: OPENTRADE PARAMS: {...}`, the DLL: +1. Looks up `OPENTRADE` in the registry. +2. Calls the eligibility function via Papyrus (`Action_IsEligible` style), waits up to `dialogue.eligibilityCheckTimeoutMs` (= 2500ms). +3. If eligible, calls the execution function with the parameters. + +**Shipped Papyrus actions** (from `Source/Scripts/skynet_Library.psc:130-315`): +`OpenTrade`, `AccompanyTarget`, `StopAccompanying`, `WaitHere`, `Gesture` (with anim enum), `RentRoom`, `CompanionFollow`, `CompanionWait`, `CompanionInventory`, `CompanionGiveTask`. + +`[verified]` `SkyrimNet.log:16393` shows: `ActionLibrary::RegisterPapyrusAction for action: OPENTRADE`. + +### 2. YAML registration + +A mod can drop YAML files into `SKSE/Plugins/SkyrimNet/config/actions/*.yaml` *in its own mod folder*. SkyrimNet's `CustomActionManager::Initialize` scans across all installed mods at load. + +**YAML actions preserve case** in the registry (unlike Papyrus actions). So `OpenTrade` stays `OpenTrade`. + +**Schema** — sample from `SeverActions - SkyrimNet Action Pack/.../config/actions/attacktarget.yaml:1-37`: +```yaml +customCategory: Combat +name: AttackTarget +description: ... +questEditorId: ... # the Skyrim Quest holding the function +scriptName: ... # Papyrus script attached to the quest +executionFunctionName: ... +parameterMapping: + - parameterName: target + decorator: get_nearby_actor # or other decorator → Skyrim object +enabled: true +defaultPriority: 50 +eligibilityRules: + - decorator: is_in_combat + operator: equals + value: true + # AND/OR combinations supported +``` + +**Contributing mods at last load** `[verified]` from `SkyrimNet.log`: +- **`SeverActions - SkyrimNet Action Pack`** — ~80 generic actions (Combat/Travel/Economy/etc.). The big one. +- **`IntelEngine`** — 15 actions (likely intelligence/observation-themed). +- **`OStimNet_v0.9.2`** — ~10 actions, prefixed `tton_*` (sex/affection — adult mod). + +### 3. C++ registration + +External SKSE plugins can call `PublicRegisterCPPAction()` from `CppAPI/PublicAPI.h:93-97` to register C++-implemented actions directly. Avoids the Papyrus VM round-trip; useful for performance-critical or low-level actions. + +`[unknown]` whether any currently-loaded mod uses this path in this install. + +--- + +## Action categories (the wrapper layer) + +Actions can be grouped under categories defined in `cat_*.yaml` files. Categories let the LLM "intent-pick" without seeing all leaf actions. + +**Known categories** `[verified]` from logs: +`Combat`, `Communication`, `Travel`, `Economy`, `Items`, `Magic`, `Outfit`, `Crafting`, `Scheduling`, `Command` (companion), `Arrest`. + +**Two-stage flow:** +1. GM (or native action selector) emits `ACTION: Communication PARAMS: {"intent": "express gratitude..."}` — picks the *category* and states the intent in natural language. +2. C++ orchestrator fires a second LLM call using `prompts/native_action_selector_drilldown.prompt`, which sees only the leaf actions under that category plus the intent. Picks the specific leaf. + +This pattern reduces cognitive load: instead of choosing among ~105 actions, the model picks one of ~10 categories then one of ~5-15 leaves. + +--- + +## The `ACTION:` parser grammar + +`[verified]` from log behavior at `ActionManager.cpp:1783` and `:1814`. + +**Required form (start of a line):** +``` +ACTION: ActionName +ACTION: ActionName PARAMS: {"param": "value", ...} +ACTION: None +``` + +**Strict requirements:** +- The literal **`ACTION: `** prefix (with the colon and space) at the start of a line. +- Action name must match a registered action **exactly** (case-sensitive, except Papyrus actions are uppercase in the registry). +- `PARAMS:` if present must be valid JSON. +- The action line must be on its own line, after any dialogue text (with `embed_actions_in_dialogue: true`). + +**No leniency:** +- Bare action names (e.g. `OPENTRADE` alone) are NOT recognized. They fall through to dialogue text. +- No fuzzy matching. No alternate prefixes (`Action:`, `ACT:`, `[ACTION]`). +- Logged when not found: `[ActionManager.cpp:1783] ParseEmbeddedAction: No ACTION: line found in response`. +- Logged when found: `[ActionManager.cpp:1814] ParseEmbeddedAction: Successfully parsed action 'X' with params: {...}`. + +**Strip behavior:** `FilterActionLines` (`ActionManager.cpp:1840`) removes the recognized `ACTION:` line from dialogue text *before* TTS, so it isn't spoken aloud. **It only strips lines it parsed successfully** — malformed lines (the bug #2 case) stay in the text and get spoken. + +--- + +## The two action selectors — how they relate + +| Aspect | `gamemaster_action_selector` | `native_action_selector` | +|---|---|---| +| **Scope** | scene-level orchestration | per-NPC, post-dialogue attribution | +| **When** | GM polling tick, player input | after Dialogue agent produces text | +| **Picks from** | StartConversation / ContinueConversation / Narrate / None / native actions | the eligible-actions registry, two-stage | +| **Variant** | `gamemaster_evaluation` | `action_evaluation` | +| **Model** | claude-sonnet (this install) | eva (this install) | +| **`max_tokens`** | 256 | 500 | +| **Output** | one ACTION line | one ACTION line (after drilldown: two LLM calls total) | + +They are **complementary, not competitive**: +- The GM directs *who speaks and what about*. +- The native action selector classifies *what physically/mechanically happens* once an NPC has spoken. + +**With `embed_actions_in_dialogue: true`:** the Dialogue agent itself can emit an `ACTION:` line inline, which **bypasses** the native action selector entirely. Faster, but more prone to format errors → bug #2 territory. + +--- + +## Where custom actions actually live (resolved) + +We initially looked for them in `mods/SkyrimNet/` itself and didn't find them. They live in **other mods' folders**: + +``` +/home/dafit/Games/Skyrim/nimmersky/mods/SeverActions - SkyrimNet Action Pack/SKSE/Plugins/SkyrimNet/config/actions/*.yaml +/home/dafit/Games/Skyrim/nimmersky/mods/IntelEngine/SKSE/Plugins/SkyrimNet/config/actions/*.yaml +/home/dafit/Games/Skyrim/nimmersky/mods/OStimNet_v0.9.2/SKSE/Plugins/SkyrimNet/config/actions/*.yaml +``` + +`[hypothesis]` — paths inferred from the contributing-mod names in the log; the agent didn't enumerate the actual files. Confirm by `find /home/dafit/Games/Skyrim/nimmersky/mods/ -name "*.yaml" -path "*/config/actions/*"`. + +This is also why "export from web UI" doesn't work cleanly — the UI sees actions in its registry but doesn't know they came from neighbor mods' folders, and can't reach back across mod boundaries to dump them. + +**Path forward for git-tracking:** symlink each contributing mod's `config/actions/` into a `nimmersky/skyrimnet/contributed-actions/{modname}/` archive, or write a one-shot collator script that copies them out. + +--- + +## The malformed-marker problem (bug #2) + +The Dialogue model occasionally emits `OPENTRADE` (bare uppercase token) instead of `ACTION: OpenTrade`. The parser ignores the bare token, doesn't strip it, and it gets TTS'd as plain text — Arcadia speaks "OPENTRADE" out loud. + +**Why it happens:** `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` lists actions to the LLM as bullets: +``` +**Available Actions:** +- `OPENTRADE` — Use ONLY if ... +- `OFFERQUEST` — ... +``` +The model sometimes emits the bare bullet name instead of the full `ACTION: OPENTRADE` line. The bullet list format encourages the mistake. + +**Why it became visible only after fixing bug #1:** the over-firing GM loop gave the Dialogue model 3-5 retries per scene; bug #2 was statistically masked. Fixing the loop removed the retries, exposing bug #2. + +See [`bugs-and-fixes.md`](bugs-and-fixes.md) for fix candidates. diff --git a/skyrimnet/agent-pipelines.md b/skyrimnet/agent-pipelines.md new file mode 100644 index 0000000..8c7d11a --- /dev/null +++ b/skyrimnet/agent-pipelines.md @@ -0,0 +1,145 @@ +# Agent Pipelines + +Each agent is selected via a "variant" in `overwrite/.../config/OpenRouter.yaml`, which maps to a model + endpoint + per-call parameters. Variants are referenced from the C++ DLL when it constructs each request. + +## Variant routing table `[verified]` from `OpenRouter.yaml` + trace dump cross-check + +| Variant | Model alias | Endpoint (this install) | `max_tokens` | Purpose | +|---|---|---|---|---| +| `gamemaster_evaluation` | `claude-sonnet-4-5-20250929` | `127.0.0.1:8000` (local Claude proxy) | **256** | "Should I act and how?" — fires GM action selector | +| `AgentDefault` (Dialogue) | `eva` (custom local alias) | `10.0.30.21:31000` | 4096 | NPC dialogue generation | +| `meta` | `omega` | `10.0.30.22:31004` | 100 | Mood eval, memory query gen, classifiers, target selection | +| `vision` | `Qwen3-VL-8B-Instruct-abliterated-v2.Q4_K_M.gguf` | `10.0.30.22:31005` | 4000 | OmniSight scene description from screenshot | +| `combat` / `action_evaluation` | `eva` | same as AgentDefault | 500 | Combat-flavor dialogue / native action selection | +| `gamemaster_scene_planner` | (no dedicated variant captured — likely uses `AgentDefault`) | — | — | Pre-plans 4-6-beat scenes (consumed by `gamemaster_action_selector.prompt:96-119` via `scene_plan` context var) | +| `intel_story_dm` | `claude-sonnet-4-5-20250929` | local proxy | — | IntelEngine plugin's persistent narrative DM | +| `gamemaster_evaluation` (TTON) | `claude-sonnet-4-5-20250929` | local proxy | — | OstimNet plugin's nearby-NPC GM | + +`[note]` Models can be reconfigured per-agent by editing `OpenRouter.yaml`. The aliases (`eva`, `omega`) are user-defined and resolved by the OpenRouter routing layer. + +--- + +## Gamemaster (GM) + +**Prompt:** `prompts/gamemaster_action_selector.prompt` (action selector) + `prompts/gamemaster_scene_planner.prompt` (optional scene-plan generator). + +**When it fires** `[verified]` from trace dump: +- Polling tick in continuous mode, roughly every `gamemaster.continuousSceneCooldownSeconds` (= 30s in this install). `[hypothesis]` exact timer source not isolated. +- On player input arrival. +- After a non-trivial in-game event (combat start, NPC death, location change) — though events are filterable via `Events.yaml`. + +**Input context:** +- Recent events (`gamemaster.recentEventsCount: 25` controls volume). +- Nearby actors (`gamemaster.nearbyActorRadius: 600`). +- Eligible actions list (populated dynamically from C++; in continuous mode `ACTION: None` is allowed only if exposed by the prompt — see `bugs-and-fixes.md` Bug #1). +- Optional `scene_plan` if scene planner has run. + +**Output:** exactly one line of the form +``` +ACTION: ActionName PARAMS: {"key": "value", ...} +``` +or `ACTION: None`. `max_tokens: 256` enforces this — no room for prose. + +**Consumers:** the C++ orchestrator parses the ACTION line and dispatches: +- `StartConversation` / `ContinueConversation` → kicks the Dialogue pipeline for the named speaker/target with the given topic. +- `Narrate` → triggers a narration-mode LLM call (no specific TTS speaker). +- `None` → no-op, scene breathes. +- Native actions (e.g. `OpenTrade`) when registered as eligible — fires the Papyrus/C++ callback. + +--- + +## Dialogue Agent (`AgentDefault`) + +**Prompt:** `prompts/dialogue_response.prompt` (16-line wrapper) + `submodules/system_head/*` (load-ordered) + `submodules/user_final_instructions/*` (load-ordered) + `submodules/character_bio/*` (per-NPC). + +**When it fires:** +- GM dispatches `StartConversation` or `ContinueConversation` with a target speaker. +- Player provides text or voice input addressed to an NPC. +- An NPC's AI Package activates one of SkyrimNet's custom packages (Player Dialogue, NPC Dialogue, TalkToPlayer). + +**Input context:** the NPC's full character bio (assembled from `character_bio/` submodules), the recent dialogue history (`event_history` component), the eligible actions list (if `embed_actions_in_dialogue: true`), the OmniSight scene description (if vision is enabled), and the topic from the GM (when GM-initiated). + +**Output:** the NPC's spoken line(s), optionally followed on a separate line by `ACTION: ActionName ...` if `embed_actions_in_dialogue: true`. + +**Consumers:** +- `FilterActionLines` (in DLL, ~`ActionManager.cpp:1840`) strips any `ACTION:` line from the dialogue text before TTS. +- `ParseEmbeddedAction` (in DLL, ~`ActionManager.cpp:1783`) extracts the action and dispatches it the same way as a GM-emitted action. +- The remaining dialogue text is fed to the TTS pipeline (`tts_generation` span in trace). +- After dialogue completes, **mood evaluation** and **memory search query generation** fire in parallel (both `meta` variant). + +--- + +## Meta Agents (`meta` variant) + +A family of small classifier/helper calls, all capped at `max_tokens: 100`. + +**Prompts:** +- `prompts/helpers/evaluate_mood.prompt` — post-dialogue mood update for the speaker +- `prompts/helpers/generate_search_query.prompt` — turns a dialogue into a memory-retrieval query +- `prompts/helpers/generate_profile.prompt` — generates/updates a character profile +- `prompts/target_selectors/dialogue_speaker_selector.prompt` — picks who in a group should respond to player +- `prompts/target_selectors/player_dialogue_target_selector.prompt` — picks the best NPC for player to address +- `prompts/memory/generate_memory.prompt` and `memory_ranker.prompt` — memory creation/ranking +- `prompts/transformers/native_dialogue_transformer.prompt` — text→text transformations +- `prompts/transformers/universal_translator.prompt` — translation pipeline + +**When they fire:** mostly post-dialogue. `target_selection_llm` runs *before* dialogue when player input arrives. + +**Output:** small structured responses (mood enum, search query string, JSON profile, NPC UUID). + +--- + +## Native Action Selector (`action_evaluation` variant) + +**Prompts:** `prompts/native_action_selector.prompt` (stage 1: pick category) + `prompts/native_action_selector_drilldown.prompt` (stage 2: pick leaf action under that category). + +**When it fires:** *after* the Dialogue agent produces text, asking "what in-game action does this dialogue imply?". `[hypothesis]` may not fire if `embed_actions_in_dialogue: true` and the Dialogue agent already emitted a valid `ACTION:` line — needs verification (see `open-questions.md`). + +**Input:** the NPC's just-spoken line + the eligible action list with category groupings. + +**Output:** `ACTION: CategoryName PARAMS: {"intent": "..."}` from stage 1, then `ACTION: LeafActionName PARAMS: {...}` from stage 2. + +**Why two stages:** with up to ~105 actions across contributor mods, asking the LLM to pick directly from a flat list is cognitively expensive. Categorizing first (Combat/Communication/Travel/Economy/etc.) narrows the choice set dramatically for stage 2. + +`[verified]` action firing pattern from `SkyrimNet.log:104547`: `ACTION: Communication PARAMS: {"intent": "express gratitude..."}`. + +--- + +## Vision Agent (`vision` variant — OmniSight) + +**Prompts:** `prompts/omnisight/describe_actor.prompt`, `describe_scene.prompt`, `describe_item.prompt`, `describe_location.prompt`, `describe_furniture.prompt`, with rendering-mode submodules in `submodules/omnisight_*/`. + +**When it fires:** on `player_text_input` and `player_direct_input_voice` events. Captures a Skyrim screenshot via `omnisight_capture_image`, then feeds it to the local Qwen3-VL model. + +**Output:** scene description text (up to 4000 tokens), inserted into the Dialogue agent's context as the `omnisight` block. + +**Consumers:** the Dialogue agent uses this to ground its response in what's visually present (objects, characters, environment) — not just what's in event logs. + +--- + +## How agents chain + +`[verified]` agent chains observed in trace `trace_1776469194689_100` (583 spans): + +``` +GM tick → ACTION: ContinueConversation → DialoguePipeline kicks + └─ target_selection_llm (meta) → picks speaker + └─ Dialogue agent generates text + optional ACTION line + ├─ inline ACTION parsed → action dispatched + ├─ remaining text → TTS + ├─ mood_evaluation (meta) parallel + └─ memory_search_query_generation (meta) parallel +``` + +For player input: +``` +player_text_input → OmniSight (vision) capture → DialoguePipeline kicks + └─ same downstream as above +``` + +## Open questions about pipelines + +See [`open-questions.md`](open-questions.md) for unresolved items: +- Does `native_action_selector` always fire, or only when `embed_actions_in_dialogue: false`? +- What event types currently trigger non-GM agent firings? (`Events.yaml` lists ~40 event types with toggles.) +- Does `gamemaster_scene_planner` ever fire in current config? diff --git a/skyrimnet/architecture.md b/skyrimnet/architecture.md new file mode 100644 index 0000000..b270ec7 --- /dev/null +++ b/skyrimnet/architecture.md @@ -0,0 +1,151 @@ +# SkyrimNet Architecture — High-Level Model + +## What SkyrimNet is + +A multi-agent LLM orchestrator that hijacks vanilla Skyrim NPC behavior — replacing static dialogue topics and idle routines with context-aware, LLM-driven scenes. NPCs talk to each other and the player through generated dialogue; their world-affecting actions are picked from a registry of "actions" contributed by SkyrimNet itself and any cooperating mod. + +`[verified]` from `SkyrimNet.log:14123-14266` (action library initialization), `Source/Scripts/SkyrimNetApi.psc` (public API), `prompts/gamemaster_action_selector.prompt` (GM orchestrator prompt). + +## The two-plugin architecture + +SkyrimNet ships alongside a sibling SKSE plugin called **IntelEngine**. They're independent SKSE plugins that share the SQLite-backed persistence layer. + +| Plugin | Role | Storage | +|---|---|---| +| **SkyrimNet** | LLM orchestration, dialogue generation, agent pipelines, TTS/STT, action dispatch | `overwrite/SKSE/Plugins/SkyrimNet/data/SkyrimNet-{epoch}-{nnnnnn}.db` | +| **IntelEngine** | Persistent narrative/intelligence layer (third-party "story DM"-style agent) | `overwrite/SKSE/Plugins/IntelEngine/data/IntelEngine-{epoch}-{nnnnnn}.db` | + +`[verified]` from disk layout. Per-game-session DB sharding (epoch suffix = save game timestamp). + +## The four code layers + +``` +┌─────────────────────────────────────────────────┐ +│ Closed-source C++ DLL │ +│ SKSE/Plugins/SkyrimNet.dll │ +│ - LLM orchestration, agent dispatch │ +│ - Action parser (ParseEmbeddedAction) │ +│ - Decorator implementation │ +│ - SQLite persistence + vector embeddings │ +└─────────────────────────────────────────────────┘ + ▲ ▼ +┌─────────────────────────────────────────────────┐ +│ Open-source Papyrus glue │ +│ mods/SkyrimNet/Source/Scripts/*.psc │ +│ - SkyrimNetApi.psc (public API surface) │ +│ - SkyrimNetInternal.psc (DLL callbacks) │ +│ - skynet_MainController.psc (quest entry) │ +│ - skynet_Library.psc (shipped action impls) │ +│ - skynet_VoiceInput*.psc (STT integration) │ +└─────────────────────────────────────────────────┘ + ▲ ▼ +┌─────────────────────────────────────────────────┐ +│ Open-source .esp content (Spriggit JSON) │ +│ mods/SkyrimNet/plugins/SkyrimNet/ │ +│ - 8 custom AI Packages (NPC/Player Dialogue, │ +│ Follow, TalkToPlayer) │ +│ - Custom Magic Effects (voice input spells) │ +│ - Factions (Whitelist/Blacklist/Following) │ +│ - Keywords (DialogueTarget/FollowTarget) │ +│ - Quests (skynet_MainController, skynet_Mcm) │ +└─────────────────────────────────────────────────┘ + ▲ ▼ +┌─────────────────────────────────────────────────┐ +│ Configuration & content (text files) │ +│ mods/SkyrimNet/SKSE/Plugins/SkyrimNet/ │ +│ - prompts/ (Inja templates, three-layer) │ +│ - sql/migrations/ (17 schema migrations) │ +│ overwrite/SKSE/Plugins/SkyrimNet/ │ +│ - config/ (38 YAML files + defaults_manifest)│ +│ - data/ (SQLite per-session DBs) │ +│ - prompts/ (runtime UI overrides) │ +│ Plus contributing mods' config/actions/*.yaml │ +└─────────────────────────────────────────────────┘ +``` + +`[verified]` All layers exist. The closed-source DLL is the only piece we cannot read directly — we infer behavior from logs, headers, Papyrus callbacks, and traces. + +## The four agent families + +Each agent maps to a "variant" in `OpenRouter.yaml`, which maps to a model/endpoint. See `agent-pipelines.md` for the full table. + +1. **Gamemaster (GM)** — scene-level orchestrator. Decides "should anything happen now, and if so what?" Polls every ~30s in continuous mode + fires on player input. Emits one `ACTION:` line. +2. **Dialogue** — generates the actual NPC speech. Triggered by GM actions like `StartConversation` / `ContinueConversation` or by player dialogue input. Can optionally append an `ACTION:` line for inline action firing. +3. **Meta** — classifiers and helpers (mood eval, memory query generation, dialogue speaker selection). Capped at ~100 tokens per call. +4. **Vision (OmniSight)** — describes the current scene from a screenshot. Uses a local Qwen3-VL model. Fires on `player_text_input` and `player_direct_input_voice` events. + +Plus a fifth implicit agent type: + +5. **Native Action Selector** — *post-dialogue* classifier that asks "what in-game action does this NPC's spoken line imply?" Two-stage: category → leaf. Distinct from the GM's scene-level action selection. + +## End-to-end orchestration trace + +For a player text-input event (verified against `all_traces_1776478948530.json`): + +``` +event_received + ├─ papyrus_decorator_cache_warmup + │ ├─ get_player + │ ├─ get_nearby_actors + │ └─ papyrus_decorators_async ← warm caches before LLM render + ├─ scene_capture + │ └─ omnisight_immediate_scene_capture + │ └─ omnisight_capture_image ← screenshot for vision model + ├─ chat_ui_open ← UI block for input + ├─ warmup_player_dialogue + │ └─ many decorator:* spans (decnpc, render_subcomponent, …) + └─ dialogue_manager_handle_player_speech + ├─ target_selection_llm ← meta-model: who responds? + └─ generate_response + ├─ initiate_eligibility_checks (Papyrus IsEligible callbacks) + ├─ build_action_context + │ ├─ wait_eligibility_results (≤ 2500ms) + │ ├─ filter_eligible_actions + │ └─ build_action_schemas (JSON schema list for LLM) + ├─ build_payload + │ └─ render_template (Inja render of dialogue_response.prompt) + ├─ llm_request (variant=AgentDefault → eva) + ├─ tts_generation + │ └─ tts_segment_0…N + ├─ mood_evaluation (variant=meta → omega, parallel) + └─ memory_search_query_generation (variant=meta → omega, parallel) +``` + +For a continuous-mode GM tick (also `[verified]` from trace): + +``` +gamemaster_evaluation_llm + └─ gamemaster_async_llm + └─ llm_request (variant=gamemaster_evaluation → claude-sonnet-4-5, max_tokens=256) + ↓ +[parser extracts ACTION: line] + ↓ +if action == StartConversation or ContinueConversation: + player_dialogue_manager_process_event + └─ dialogue_manager_handle_perceived_event + └─ generate_response (full pipeline above) +``` + +## Where the bottlenecks are + +`[hypothesis]` based on the trace structure and log volumes: + +- **GM `max_tokens: 256`** is a hard ceiling. With three contributor mods registering ~105 actions total, the GM has to reason over a large `eligible_actions` list and emit one ACTION line — the two-stage drilldown and category wrapper exist precisely to compress this cognitive load. +- **`wait_eligibility_results` blocks for up to 2500ms.** Slow Papyrus eligibility callbacks shrink the available action set. This is a Skyrim-VM-side performance dependency that no LLM tuning can fix. +- **OmniSight vision** runs locally on a Qwen3-VL model. Image capture + inference adds latency before any text generation can begin. + +## Adjacent technologies in the substrate + +- **whisper.cpp** for local STT (`SKSE/Plugins/SkyrimNet/libs/whisper.dll` + `ggml*.dll` for CPU/CUDA/Vulkan/OpenCL backends). +- **all-MiniLM-L6-v2** sentence-transformer for semantic embedding of NPC memories (`SKSE/Plugins/SkyrimNet/models/all-MiniLM-L6-v2-tokenizer.json`). +- **ONNX runtime** (`onnxruntime_skyrimnet.dll`) — likely VAD or auxiliary model inference. +- **espeak-ng** voice data (`SKSE/Plugins/SkyrimNet/models/espeak-ng-data/`) — TTS phoneme tables for Piper/PocketTTS. +- **Spriggit** to git-track the .esp content as JSON. + +## Cross-references + +- For per-agent firing details see [`agent-pipelines.md`](agent-pipelines.md). +- For the prompt template system see [`prompt-templates.md`](prompt-templates.md). +- For action registration and the `ACTION:` parser see [`action-system.md`](action-system.md). +- For YAML config behavior see [`config-knobs.md`](config-knobs.md). +- For known bugs and what was tried see [`bugs-and-fixes.md`](bugs-and-fixes.md). diff --git a/skyrimnet/bugs-and-fixes.md b/skyrimnet/bugs-and-fixes.md new file mode 100644 index 0000000..d077d92 --- /dev/null +++ b/skyrimnet/bugs-and-fixes.md @@ -0,0 +1,157 @@ +# Bugs and Fixes — Running Log + +A chronological record of bugs we've found in SkyrimNet behavior, what we did about them (or chose not to do), and the latent issues that surface as a result. **Entries stay even after fixes** — they document why a code path looks the way it does. + +## How to read this log + +Each entry has: +- **Status:** `[FIXED]`, `[PARTIAL FIX]`, `[KNOWN, UNFIXED]`, `[REGRESSION]`, `[OBSERVED, NOT REPRODUCED]` +- **Discovered:** absolute date +- **Symptom:** what the user/we saw +- **Root cause:** what's actually broken +- **Fix:** what we changed (or "none — see notes") +- **Side effects:** anything the fix exposed or changed downstream +- **Files touched:** absolute paths + +--- + +## Bug #0 — First sentence not played in TTS `[FIXED]` + +**Discovered:** 2026-04-18 (early in this session) +**Status:** Fixed by user via in-game UI before we instrumented the cause. + +**Symptom:** The first sentence of any NPC dialogue line was generated by the LLM (visible on the web UI debug view) but never spoken out loud. Subsequent sentences played fine. + +**Root cause:** Race in the DBVO (Dialogue Background Voice Over) pipeline. SkyrimNet routes TTS audio through vanilla Skyrim's dialogue topic system; the first audio chunk would arrive before the dialogue topic was fully "mounted," and the audio buffer dispatched into a non-listening topic — silent drop. Subsequent chunks landed on the now-live topic and played fine. + +**Fix:** User enabled two settings via in-game UI (later persisted to YAML): +- `enableNPCNodeUpdates: false → true` (`SkyrimNet.yaml:26`) — forces audio node re-binding when a new line begins, so the first buffer has a live node to attach to. +- `dbvo.fixSubtitleDuringTTS: false → true` (`SkyrimNet.yaml:31`) — extends subtitle (and carrier line lifetime) to cover the actual TTS audio. + +Plus belt-and-braces: `tts.maxSegmentWords: 24 → 4` so even residual races lose at most 4 words instead of a full sentence. + +**Side effects:** The default-off settings have small visual costs (skin texture flicker with mods that inject textures at runtime; brief subtitle flicker). Worth the trade for this user. + +**Files touched:** `SkyrimNet.yaml` (in `overwrite/.../config/`). + +--- + +## Bug #1 — Recursive farewell loop in continuous mode `[REVERTED — fix preserved, see notes]` + +**Discovered:** 2026-04-18 +**Status:** Fix was applied to `gamemaster_action_selector.prompt` and verified working (GM returned `ACTION: None` in subsequent traces). **Reverted later same day** because the fix exposed Bug #2 (malformed action markers were no longer being masked by the GM's retry loop). The reverted edit is preserved at `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/gamemaster_action_selector.prompt.edit` for reference. Active prompt is back to upstream baseline. Plan: re-apply the fix paired with a sister edit to `submodules/user_final_instructions/0750_embedded_actions.prompt` that prevents Bug #2. + +**Symptom:** `conversation_log.log:79-106` shows Severio Pelagia delivering 5+ farewells over ~1m50s after player said "Please leave me alone": +``` +[2026-04-17 00:53:12] Severio: As you wish, lad. I'll be on my way then. Take care of yourself out here. The roads can be dangerous... +[2026-04-17 00:53:22] Severio: Farewell, lad. May the road be kind to you. I'll be heading back to the farm now. Take care of yourself out here. +[2026-04-17 00:53:30] Severio: As you wish, lad. I'll take my leave then. The road can be treacherous... +[2026-04-17 00:53:39] Severio: I'll be on my way then, lad. Take care of yourself out here. The roads can be dangerous... +[2026-04-17 00:53:48] Severio: Muttering to myself as I turn back towards home... +[2026-04-17 00:53:55] Severio: These young'uns, always in such a hurry... +``` +Same intent, different paraphrasings, every ~8-10 seconds. + +**Root cause:** Structural in `prompts/gamemaster_action_selector.prompt`: +- **Line 9 (pre-fix):** `{% if not is_continuous_mode %}- ACTION: None — no action needed{% endif %}` — the `None` action was *removed from the action menu* when in continuous mode. +- **Lines 75-77 (pre-fix):** "You are actively directing this scene. **You must select an action**—do not select None." — explicit prohibition. +- **Lines 109-114 (pre-fix):** the wise `None` description ("a conversation just concluded naturally, silence serves the mood better") was gated to non-continuous mode only. + +So in continuous-roleplay mode (which user keeps on), the GM had **no syntactic way** to signal "the scene is done." Its only choices were `StartConversation`, `ContinueConversation`, or fail. Combined with line 763's bias ("Shape the world actively—don't just facilitate dialogue; make things happen") and the `ContinueConversation` reason at line 791 ("the dialogue needs another beat to conclude naturally"), the GM was structurally pushed to keep firing `ContinueConversation`. Each `ContinueConversation` re-prompted the Dialogue agent with the prior farewell in context, and the Dialogue agent dutifully restated. + +**Fix:** Edited `prompts/gamemaster_action_selector.prompt` with seven changes: +1. Line 9: Removed `{% if not is_continuous_mode %}` guard so `ACTION: None` is always in the format help. +2. Line 57: Qualified "needs another beat" with "NOT applicable if prior beat was a conclusion." +3. NEW Anti-Restatement Rule under ContinueConversation: explicit instruction that topic must drive *new* content, never rephrasing. +4. Lines 75-77: Replaced "you must select an action—do not select None" with "Prefer action over inaction. **However, recognize natural endings.**" +5. NEW Concrete signals list: explicit triggers for `ACTION: None` ("farewell," "goodbye," "I'll be on my way," "take care," "no more words," "leave me alone"). +6. Lines 150-152: Removed `{% if not is_continuous_mode %}` guard so `None` is always listed in Available Actions. +7. Line 161: Continuous user message mentions `ACTION: None` as a valid choice. + +Backup naming convention: the upstream baseline (now active again after revert) had been kept at `gamemaster_action_selector.prompt.backup` while the fix was active. After revert, the user renamed our edited version to `gamemaster_action_selector.prompt.edit` — semantically clearer than `.backup` (it labels "this is OUR edit," not "this is what was here before"). This `.edit` suffix is the convention going forward for preserving reverted-but-not-discarded edits. + +**Side effects:** **Exposed Bug #2** — the over-firing GM loop had been masking the malformed-marker bug by giving the Dialogue model multiple retries to emit a parser-compatible action line. With the loop stopped, malformed markers go through to TTS uncorrected. **This is the reason for the revert** — we want both fixes shipped together, not a partial regression. + +**Files touched:** `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/gamemaster_action_selector.prompt`. + +**Verification:** Web UI trace screenshot at 04:43 showed three consecutive `ACTION: None` returns from the GM after Arcadia's `OPENTRADE` greeting — exactly the desired behavior (no spurious continuation). + +--- + +## Bug #2 — Malformed action markers spoken aloud `[KNOWN, UNFIXED]` + +**Discovered:** 2026-04-18 (immediately after Bug #1 fix exposed it) +**Status:** Known, unfixed. Was previously masked by Bug #1. + +**Symptom:** The Dialogue model emits a bare uppercase token (e.g., `OPENTRADE`) at the end of its dialogue text without the required `ACTION: ` prefix. The C++ parser ignores it as a non-match, and `FilterActionLines` doesn't strip it (it only strips successfully-parsed action lines). The token then falls through to TTS and gets spoken verbatim. No in-game action fires. + +**Concrete example** (from web UI screenshot, 04:42:47): +``` +Dialogue agent output: +"Welcome, Davies Nullshari. My name is Arcadia. How may I assist you today? +OPENTRADE" + +→ Arcadia speaks: "Welcome, Davies Nullshari. My name is Arcadia. How may I assist you today? OPENTRADE." +→ No trade UI opens. +``` + +**Root cause:** Two-layer: +1. **Parser is strict** (`ActionManager.cpp:1783` "No ACTION: line found in response"). Requires literal `ACTION: ` prefix at start of line. No fuzzy matching, no fallback. +2. **Prompt format encourages the mistake.** `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` lists actions as bullets: + ``` + **Available Actions:** + - `OPENTRADE` — Use ONLY if ... + - `OFFERQUEST` — ... + ``` + The model sometimes emits the bare bullet name instead of the full `ACTION: OPENTRADE` line. + +**Why it became visible after fixing Bug #1:** The over-firing GM loop gave the Dialogue model 3-5 retries per scene. Eventually one retry would emit the marker correctly and the action would fire. With the loop stopped, the first malformed emission is the only one — no retry, no recovery. + +**Fix candidates** (none applied): + +A) **Prompt-side enforcement.** Edit `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` to add a "WRONG vs RIGHT" example: +``` +**CRITICAL — common mistake:** +- ✗ WRONG: "Welcome to my shop. OPENTRADE" +- ✓ RIGHT: "Welcome to my shop.\nACTION: OpenTrade" +The literal `ACTION: ` prefix at start of a new line is REQUIRED. Bare action names will NOT fire. +``` + +B) **Parser-side leniency.** Modify the C++ DLL to recognize bare uppercase tokens at end of dialogue as candidates. **Not feasible** — closed source. + +C) **Add an "action-marker recognition" rule to the GM** so the GM sees the orphaned marker in recent dialogue and fires the corresponding action on its next tick. Brittle — adds responsibility to the wrong agent and depends on GM tick timing. + +**Recommendation:** Option A is the durable fix. Cheap, additive, treats the cause not the symptom. + +**Files touched:** none yet. Sister edit would land at `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/submodules/user_final_instructions/0750_embedded_actions.prompt`. + +**User's stance:** "all fine dear it's just a game B." — willing to live with this for now while we map the architecture systematically. + +--- + +## Bug #3 — `OPENTRADE` reflects an action-name uppercasing surprise `[OBSERVED, NOT REPRODUCED]` + +**Related to Bug #2**, deserves its own note. + +**Observation:** Papyrus-registered action names are **uppercased in the action registry** (`OpenTrade` → `OPENTRADE`). YAML-registered actions preserve case. So `OPENTRADE` IS the correct registry name when emitted via the Papyrus path — the only thing the Dialogue agent gets "wrong" is the missing `ACTION: ` prefix. + +**Implication:** Any prompt-side fix to bug #2 should reference `ACTION: OPENTRADE` (uppercase, matching the registry), not `ACTION: OpenTrade`. Mismatch would cause the parser to also reject the corrected form. + +**Verified:** `SkyrimNet.log:16393` `ActionLibrary::RegisterPapyrusAction for action: OPENTRADE`. + +--- + +## Pattern observation: "two bugs canceling out" + +Bug #1 + Bug #2 form a classic interaction pattern: +- The visible bug (recursive farewell) was *caused by* Bug #1. +- Bug #2 (malformed markers) had been silently present but **statistically masked** by Bug #1's retry loop. +- Fixing Bug #1 exposed Bug #2. + +This is worth remembering when planning future fixes: **a clean fix can reveal latent bugs that were invisible while the bigger bug was running**. Always consider what the broken behavior was inadvertently working around. + +--- + +## Future-bug log slots + +(Reserved for entries we haven't discovered yet. When you land here from a future session having found a new bug, add an entry following the format above and update the README's "Last verified pass" date.) diff --git a/skyrimnet/config-knobs.md b/skyrimnet/config-knobs.md new file mode 100644 index 0000000..f957767 --- /dev/null +++ b/skyrimnet/config-knobs.md @@ -0,0 +1,129 @@ +# Config Knobs — load-bearing settings + +YAML configs live in `overwrite/SKSE/Plugins/SkyrimNet/config/` (with active edits) and `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/config/` (shipped baselines). 38 YAML files total, plus `defaults_manifest.json`. + +This document focuses on the **load-bearing** settings — ones we've discovered have outsized effects on behavior. Full enumeration would be brittle and date-sensitive; the goal here is to capture the gotchas. + +## SkyrimNet.yaml — the central config + +`[verified]` Direct observations from this install + log/trace correlation. + +| Setting | This install | What it controls | Gotcha | +|---|---|---|---| +| `dialogue.embedActionsInDialogue` | `true` | Lets the Dialogue model emit `ACTION:` lines inline after speech. | When `true`, the native_action_selector may be bypassed — and bug #2 (malformed markers) surfaces. When `false`, actions are strictly post-hoc via native selector. | +| `dialogue.eligibilityCheckTimeoutMs` | `2500` | Bounds the `wait_eligibility_results` span. | Slow Papyrus eligibility callbacks (>2.5s) get dropped from the available action list for that turn. | +| `dialogue.interruptDialogueOnVoiceStart` | `true` | Player STT triggers a barge-in on the NPC. | Distinct from TTS-side interrupts; affects whose voice cuts off whom. | +| `dialogue.maxSegmentWords` (`tts.maxSegmentWords`) | `4` | TTS chunk size for streaming. | Was raised by us as part of the first-sentence-drop fix. Small chunks can amplify chunker race conditions; suspected innocent in NPC-NPC restatement. | +| `gamemaster.enabled` | `true` | Master switch for the GM. | — | +| `gamemaster.agentEnabled` | `true` | `[unknown]` Distinct from `enabled`? Both default `true`; toggling untested. | See open-questions.md. | +| `gamemaster.continuousSceneCooldownSeconds` | `30` | Minimum gap between GM polling ticks in continuous mode. | Lower = more LLM cost, more "alive" feel. Higher = cheaper, quieter scenes. | +| `gamemaster.actionCooldownSeconds` | `180` | Per-NPC action cooldown. | Prevents the same NPC from being action-targeted repeatedly. | +| `gamemaster.nearbyActorRadius` | `600` | Radius around player the GM considers. | Skyrim units (~10cm each, so ~60m). | +| `gamemaster.recentEventsCount` | `25` | How much event history goes into the GM prompt. | Higher = more context, more tokens, more cost. | +| `gamemaster.requestTimeoutSeconds` | `30` | Per-call LLM timeout for the GM. | — | +| `gamemaster.continuousSceneCooldownSeconds` | `30` | (duplicate row — see above) | — | +| `interaction.maxDistance` | `250` | Default interaction distance. | Affects who hears what. | +| `interaction.normalMaxDistance` | `250` | Normal speech audibility. | — | +| `interaction.whisperMaxDistance` | `100` | Whisper audibility. | — | +| `events.preloadCount` | `1000` | Events warmed at startup. | Affects load time + initial memory query corpus. | +| `narration.enabled` | `false` | Suppresses asterisk-style narration globally. | Referenced by `is_narration_enabled()` decorator in many prompts; toggling changes prompt content materially. | +| `subtitles.enableNPCSubtitles` | `true` | Whether NPC subtitles render. | — | +| `dbvo.enabled` | `true` | Dialogue Background Voice Over — routes TTS through vanilla Skyrim dialogue topics. | The "first-sentence-dropped" bug we hit was traced to this. Disabling skips lip-sync and topic-coupling. | +| `shouldSilenceActors` | `true` | Silences actors before TTS plays (prevents vanilla audio overlap). | Can race the first audio chunk and silence it. | + +`[verified]` `enableNPCNodeUpdates: true` and `dbvo.fixSubtitleDuringTTS: true` were the toggles that fixed the user's "first sentence not played" issue (see `bugs-and-fixes.md` historical context). + +## Agents.yaml + +Defines per-agent variant assignments and behavior. Each agent has: +- `variant` — which OpenRouter variant to use (cross-references `OpenRouter.yaml`) +- `enabled` — master toggle for the agent +- Possibly `cooldown`, `timeout`, model-specific overrides + +`[hypothesis]` Editing `Agents.yaml` is how a user changes which model handles which agent (e.g., "use Claude for dialogue too, not just GM"). Confirm by inspection. + +## OpenRouter.yaml + +Variant → model + endpoint + per-call defaults. See `agent-pipelines.md` for the full table extracted from this install. + +Key sections: +- `default_params` per variant — `max_tokens`, `temperature`, `top_p`, etc. +- `endpoint` — the URL the OpenRouter routing layer sends to. In this install, mostly local IPs (`10.0.30.x`) plus a Claude proxy on `127.0.0.1:8000`. +- `model` — the OpenRouter alias or local model ID. + +## Actions.yaml + +`[hypothesis]` Defines global action-system policies: which categories are enabled, default priorities, eligibility-check defaults. Doesn't define individual actions — those come from `config/actions/*.yaml` in *contributing mods'* folders (see `action-system.md`). + +## Events.yaml + +~40 event types, each with toggles: +- `enabled` — does SkyrimNet listen for this event at all +- `persistent` — does the event get stored in the long-term log +- `shortLivedEnabled` — short-lived event variant +- `allowNPCReaction` — can NPCs react to this event via the GM +- `npcReactionCooldown` — per-NPC cooldown for reacting +- `interrupt` — does this event interrupt ongoing dialogue + +**Currently disabled in user's config** `[verified]` from the user's install: +- `quest_stage` +- `cell_attach_detach` +- `quest_objective_state` +- `scene*` (multiple scene_* event types) + +These were likely disabled to cut event noise. Re-enabling would feed more triggers to the GM and increase LLM call frequency. + +## Memory.yaml + +Controls the vector memory system (semantic embeddings via all-MiniLM-L6-v2): +- Embedding model selection +- Memory retention windows +- Number of memories retrieved per query +- Memory generation thresholds + +`[hypothesis]` These tune the RAG-for-NPCs behavior. Defaults probably reasonable for most users. + +## OmniSight.yaml + +Vision-agent settings — when to capture screenshots, how often, scene description verbosity. + +## MCP.yaml + +`[hypothesis]` Model Context Protocol integration? Not yet inspected. Could be relevant if SkyrimNet exposes MCP servers/clients for external tool integration. + +## Per-TTS provider configs + +One YAML per supported TTS backend: +- `XTTS.yaml`, `Zonos.yaml`, `Chatterbox.yaml`, `Piper.yaml`, `PocketTTS.yaml`, `ElevenLabs.yaml`, `Inworld.yaml`, `VastAI.yaml` + +Active backend selected by `tts.engine` in `SkyrimNet.yaml`. This install uses `pocket_tts`. + +Streaming params worth knowing (in `PocketTTS.yaml`): +- `streaming.max_words_to_process: 32` — max chunk size +- `streaming.min_words_to_process_initial: 16` — first-chunk minimum (was a suspect for the first-sentence-drop bug) +- `streaming.process_on_sentence_boundary: true` +- `streaming.process_on_word_count: true` +- `streaming.max_buffers_per_actor: 4` + +## STT.yaml + +Speech-to-text settings (whisper.cpp local). VAD thresholds, language, sample rate, voice activity gate. + +## defaults_manifest.json + +`[hypothesis]` JSON dump of every config key with its default value, structured by section name. Looks like the **schema the DLL seeds first-run config from** — when a YAML doesn't exist, the DLL writes one populated from this manifest. + +**Sections present** (28 total): `ActorFilter`, `Agents`, `BardSinging`, `ChatUI`, `Chatterbox`, `DialogueFilter`, `Diary`, `DynamicBio`, `ElevenLabs`, `Entity`, `Events`, `Hotkey`, `Inworld`, `Memory`, `MemoryFilter`, `OmniSight`, `OpenRouter`, `Piper`, `PlayerDialogue`, `PocketTTS`, `STT`, `UniversalTranslator`, `VastAI`, `VirtualEntities`, `VoiceSamples`, `WebServer`, `XTTS`, `Zonos`. + +Each section name matches a YAML file. Verify role by deleting a YAML and watching what regenerates. + +## Plugins subtree + +`config/plugins/IntelEngine/` — the IntelEngine sibling plugin keeps its config under SkyrimNet's config root. Notable files: `factions.yaml`, `settings.yaml`. `[unknown]` exact contents. + +## Editing safety + +- **Always edit `overwrite/.../config/`**, never `mods/.../config/` directly. The overwrite layer is what the game reads. +- **Most settings hot-reload.** Restart only needed for backend swaps (TTS engine, STT) or fundamental architecture changes. +- **Backup before changes:** `cp foo.yaml foo.yaml.backup` is the pattern we've used. +- **The in-game MCM (Mod Configuration Menu)** writes back to these YAMLs — UI tweaks persist as YAML edits, which is convenient but means the UI is yet another writer to be aware of. diff --git a/skyrimnet/open-questions.md b/skyrimnet/open-questions.md new file mode 100644 index 0000000..e55b8c9 --- /dev/null +++ b/skyrimnet/open-questions.md @@ -0,0 +1,127 @@ +# Open Questions + +Things we believe but couldn't fully verify in the first discovery pass, or that we observed but couldn't explain. Each question has a **verification path** — what would resolve it. + +When a question gets resolved, move the answer into the relevant detail file (`architecture.md`, `agent-pipelines.md`, etc.) and delete the entry here. Or move it to `bugs-and-fixes.md` if it turned out to be a bug. + +--- + +## Q1: Does `native_action_selector` always fire after dialogue, or only sometimes? + +**Why we care:** Determines whether `embed_actions_in_dialogue: true` truly bypasses the native selector, or whether they run in parallel and fight for the action slot. + +**What we observed:** The trace dump (`all_traces_1776478948530.json`) shows `mood_evaluation` and `memory_search_query_generation` after `generate_response`, but **no obvious `native_action_selection` span**. Either: + +- (a) It's triggered on demand only when `embed_actions_in_dialogue: false` and didn't fire in the captured session. +- (b) It's named something else in the trace tree. +- (c) It was firing in parallel and the trace just didn't capture it under a recognizable name. + +**Verification path:** Set `embed_actions_in_dialogue: false` in `SkyrimNet.yaml`, trigger an NPC dialogue, capture a fresh trace, look for new spans that weren't there before. If new `*action_evaluation*` spans appear, hypothesis (a) is confirmed. + +--- + +## Q2: What's the actual GM polling timer source? + +**Why we care:** We've been calling it "polling every ~30s" based on `gamemaster.continuousSceneCooldownSeconds`, but the trace doesn't show a clean periodic tick — fires correlate with events too. + +**What we observed:** GM `gamemaster_evaluation` calls fire at irregular intervals in the captured logs. Sometimes ~30s apart, sometimes seconds after a player input or NPC event. + +**Hypothesis:** The "polling" is actually event-driven — every relevant Skyrim event (per `Events.yaml` filters) triggers a GM evaluation, with `continuousSceneCooldownSeconds` as a *minimum* gap to prevent spam. So idle silence + cooldown = ~30s tick; active scene = much faster. + +**Verification path:** Disable all event-triggered firings in `Events.yaml` (set `allowNPCReaction: false` everywhere), enable continuous mode, sit in an empty cell with no NPCs, watch trace timing. Should see clean 30s ticks if cooldown is the only timer. + +--- + +## Q3: What does `defaults_manifest.json` actually do? + +**Why we care:** Understanding the first-run / reset path is important for users restoring config. + +**Hypothesis:** It's the schema the DLL seeds first-run config from. When a YAML doesn't exist, the DLL writes one populated from this manifest. + +**Verification path:** Delete `overwrite/.../config/Memory.yaml` (a low-stakes one), launch Skyrim, check whether a fresh `Memory.yaml` regenerates with default values. If yes, hypothesis confirmed. + +--- + +## Q4: Is `gamemaster.agentEnabled` distinct from `gamemaster.enabled`? + +**Why we care:** Both default `true` in this install. If they're the same, that's tech debt; if they're different, we need to know what each gates. + +**Verification path:** Toggle `agentEnabled: false` while keeping `enabled: true`, observe behavior. Then swap. Document whichever toggle changes what. + +--- + +## Q5: Does `gamemaster_scene_planner` ever fire in current config? + +**Why we care:** `prompts/gamemaster_action_selector.prompt:96-119` references a `scene_plan` context variable, suggesting a separate planner agent populates it. But the captured trace dump shows no `scene_planner` spans — either it never fires, or it fires under a different name, or it's gated behind an MCM toggle that's currently off. + +**Verification path:** Search `Agents.yaml` for any scene_planner-related variant. Check MCM for a "Scene Planner" toggle. Enable if found, observe traces. + +--- + +## Q6: What's the canonical decorator list? + +**Why we care:** Decorators are the bridge from prompts to game state. Without the full list with signatures, we can't reason about what's possible to surface in a prompt. + +**What we have:** ~30+ decorator names spotted in trace spans: +`decnpc`, `is_in_faction`, `is_player`, `is_in_combat`, `get_arousal_state`, `render_character_profile`, `render_subcomponent`, `format_event`, `short_time`, `get_name`, `get_nearby_npc_list`, `get_recent_events`, `get_quest_stage`, `is_narration_enabled`, `get_relevant_memories`, `get_scene_context`, `get_world_knowledge`, `papyrus_util`, `outfit_context`, … + +**Verification path:** The full list is dynamically generated by the DLL via `prompts/documentation/main.prompt`. The in-game web UI's `/api/documentation` endpoint should expose the full enumeration with signatures. Hit that endpoint with `curl` against the local web server (port from `WebServer.yaml`). + +--- + +## Q7: Where do contributing mods' YAML actions live on this disk? + +**Why we care:** We claim they live in `/home/dafit/Games/Skyrim/nimmersky/mods/{ModName}/SKSE/Plugins/SkyrimNet/config/actions/*.yaml`, but we haven't enumerated the actual files in this install. Need this for the git-tracking plan. + +**Verification path:** `find /home/dafit/Games/Skyrim/nimmersky/mods/ -name "*.yaml" -path "*/config/actions/*"`. List all paths, count by mod, sample one schema to confirm consistency with `action-system.md` description. + +--- + +## Q8: What's IntelEngine doing exactly? + +**Why we care:** It's a sibling SKSE plugin that registers 15 actions and has its own SQLite DB. We don't know its scope. + +**What we have:** Plugin name suggests "intelligence" — possibly classification, world observation, scene tagging, or persistent narrative state. + +**Verification path:** Check if IntelEngine ships its own README in the mod folder. Read its config files at `overwrite/.../config/plugins/IntelEngine/` (`factions.yaml`, `settings.yaml`). Inspect its action YAMLs to see what behaviors it registers. + +--- + +## Q9: How does the SQLite vector_memory_system actually retrieve? + +**Why we care:** The `0014_regenerate_embeddings` migration confirms an embedding pipeline. We know `all-MiniLM-L6-v2` is the embedding model. We don't know retrieval-time semantics: similarity threshold, top-K, recency weighting, faction filtering. + +**Verification path:** Read `Memory.yaml` for tunables. Trace a `memory_search_query_generation` span end-to-end — the meta agent generates a query string, then the DLL must embed it, run a vector similarity search against the `vector_memory_system` table, and inject results into the dialogue context. Check the `SkyrimNet.log` for SQL query patterns. + +--- + +## Q10: Knowledge Pack format (`0017_knowledge_packs` migration) + +**Why we care:** dafit already authors `nimmerverse/nimmersky/oghma-sknpack/` knowledge packs (Whiterun, Winterhold, Solstheim, etc.). Understanding the runtime ingestion + storage format would let us validate that our packs are well-formed. + +**Verification path:** Read `0017_knowledge_packs.sql` to see the table schema. Inspect an oghma-sknpack file to see the source format. Trace ingestion in `SkyrimNet.log` — search for "knowledge_pack" or "sknpack" patterns. The `oghma-proxy` tool probably has documentation we wrote ourselves. + +--- + +## Q11: Does the chunker overlap suspicion for NPC↔NPC restatement still hold? + +**Why we care:** During Bug #2 diagnosis we hypothesized `tts.maxSegmentWords: 4` might amplify chunker race conditions in NPC-NPC scenes. We later determined the GM-loop bug fully explained restatement and the chunker was innocent. But the small chunk size remains as a stress test for any other latent chunker bugs. + +**Verification path:** Once Bug #2 is fixed, observe NPC-NPC scenes (Whiterun market with Ysolda↔Brenuin) with `maxSegmentWords` at 4 vs 24. If 4-word chunks cause clipping or audio gaps not seen at 24, the chunker has a chunk-size-dependent issue worth filing as Bug #4. + +--- + +## Q12: How does the in-game prompt editor's `propose_prompt_edit` tool flow back into the override file? + +**Why we care:** The runtime override at `overwrite/prompts/agent_chat.prompt.backup.1776372078` proves edits via the in-game LLM-driven prompt editor (`agent_prompt_helper.prompt`) get persisted with epoch-stamped backups. We don't know the exact write semantics — does it always backup-and-replace, or only on accept? + +**Verification path:** Use the in-game prompt editor on a low-stakes prompt, watch what files appear/change in `overwrite/prompts/`. Document the backup naming and write-on-accept behavior. + +--- + +## Conventions for adding new entries + +- Number monotonically (Q13 next). +- Each entry needs **why we care** + **verification path** at minimum. +- If a question gets answered, *move the answer into the appropriate detail file* and delete the entry here. Don't let resolved questions pile up. +- If a question gets *partially* answered, update it with the partial answer and what's still unknown. diff --git a/skyrimnet/prompt-templates.md b/skyrimnet/prompt-templates.md new file mode 100644 index 0000000..21206e7 --- /dev/null +++ b/skyrimnet/prompt-templates.md @@ -0,0 +1,116 @@ +# Prompt Template System + +## Template language: Inja, NOT Jinja2 + +`[verified]` from `prompts/agent_prompt_helper.prompt:138-310`, which contains the canonical syntax reference embedded into the in-game prompt editor. + +Inja is a C++ Jinja2-inspired template engine. It looks similar but is **strictly more limited**. + +### Key differences from Jinja2 worth knowing + +- **Limited filter set.** Standard Jinja filters like `default`, `upper`, `length`, `join`, `sort` are present; less common ones may not be. +- **`{% set %}` only affects render context** — does NOT mutate input data. +- **Array manipulation functions return new arrays** — `append`, `extend`, etc. don't modify originals. +- **Template inheritance is limited:** `{% extends "..." %}` + `{% block name %}...{% endblock %}` works; multiple inheritance does not. +- **No macros** in the Jinja2 sense. +- **Backslashed paths in includes:** `render_template("components\\event_history_compact")` — the `\\` is the path separator in include calls. + +### SkyrimNet-specific Inja extensions + +**Section markers** split a single `.prompt` file into multiple chat messages: +``` +[ system ] +... system message content ... +[ end system ] + +[ user ] +... user message content ... +[ end user ] + +[ assistant ] +... assistant message content (rare) ... +[ end assistant ] +``` + +**Special blocks:** +- `[ raw ] ... [ end raw ]` — prevents Inja parsing of `{{...}}` etc. Used for embedded examples that contain template syntax literally. +- `[ cache ] ... [ end cache ]` — marks a block as cacheable for prompt caching across requests. + +**Include functions:** +- `render_template("path\\with\\backslashes")` — include another `.prompt` from `prompts/`. +- `render_subcomponent("subdir", render_mode)` — render numbered submodules from a subdir in load order. +- `render_character_profile("mode", uuid)` — render an NPC's bio with a render mode (`full`, `target`, `transform`, `thoughts`, `short_inline`, `interject_inline`). + +**Decorators** (DLL-registered functions callable from templates): +- `decnpc(uuid)` — universal NPC info accessor; returns object with `name`, `firstName`, `lastName`, `race`, `gender`, pronouns (`subjectivePronoun`, `objectivePronoun`, `possessivePronoun`, `reflexivePronoun`), `level`, `health`, `magicka`, `stamina`, `faction[]`, `isInCombat`, `isHostile`, `isDead`, `isBusy`, all skill values. +- 30+ other decorators observed in trace dump: `is_in_faction`, `is_player`, `get_arousal_state`, `format_event`, `short_time`, `get_name`, `get_nearby_npc_list`, `get_recent_events`, `get_quest_stage`, `is_narration_enabled`, `get_relevant_memories`, `get_scene_context`, `get_world_knowledge`, `papyrus_util`, `outfit_context`, etc. + +`[unknown]` Full decorator list with signatures — dynamically generated by the DLL via `prompts/documentation/main.prompt`. Would require dumping the in-game web UI's `/api/documentation` endpoint to enumerate. + +--- + +## The three-layer override architecture `[verified]` + +Load priority **high → low** (later layers shadow earlier ones for the same path): + +| # | Path | Purpose | Notes | +|---|---|---|---| +| 1 | `overwrite/SKSE/Plugins/SkyrimNet/prompts/` | **Runtime overrides.** Edits via the in-game prompt editor land here. | Selectively populated. Contains epoch-stamped backups (`agent_chat.prompt.backup.1776372078`). Also hosts runtime-generated content: `dynamic_character_bio.prompt` (~173KB regenerated per session), `_saves/`, `characters/` (per-actor profiles). | +| 2 | `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/` | **Shipped baseline + dev/user edits.** | Where our `gamemaster_action_selector.prompt` fix lives. The `.backup` from the fix is here too: `gamemaster_action_selector.prompt.backup`. | +| 3 | `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/original_prompts/` | **Pristine upstream baseline.** Useful for diffing to detect upstream changes. | `[hypothesis]` Only `characters/` and `submodules/system_head/` are populated; the rest of upstream prompts may be assumed copied to `prompts/` at install time, or this directory only holds pristine *overrideable* defaults. | + +**Practical implication:** before editing any prompt, check if a runtime override exists in `overwrite/prompts/`. If it does, *that* is the active version — editing in `mods/prompts/` will be silently shadowed. + +`[verified]` overlay pattern is the same shape as Linux `/etc/foo.conf` + `/etc/foo.conf.d/` + package defaults. + +--- + +## Submodule numbering convention `[verified]` + +Submodules in `prompts/submodules//` are loaded in numerical filename order (e.g. `0010_*.prompt` before `0750_*.prompt`). The numbering carves out conventional ranges: + +| Range | Conventional purpose | Examples | +|---|---|---| +| `0010_…0099_` | Meta/header content (instructions, scene context bootstrap) | `0010_instructions.prompt`, `0250_omnisight.prompt` | +| `0100_…0499_` | Content sections (summary, background, personality, appearance, equipment) | character bio sections | +| `0500_…0799_` | Guidelines and behavioral rules | `0750_embedded_actions.prompt` (the ACTION: format spec) | +| `0800_…0999_` | Late instructions and special toggles | direct narration, recent state changes | +| `7000_…7999_` | Memories and progression | NPC memory blocks | +| `9990_` | Speech style — rendered last to override earlier voice instructions | speech style submodules | + +Earlier numbers establish baseline; higher numbers customize/override. Fits the late-binding-wins pattern. + +--- + +## Subdirectory map of `prompts/` `[verified]` + +| Dir | Purpose | Sample contents | +|---|---|---| +| `prompts/` (root) | Top-level entry-point prompts (one per agent type) | `dialogue_response.prompt`, `gamemaster_action_selector.prompt`, `native_action_selector.prompt`, `agent_chat.prompt`, `gamemaster_scene_planner.prompt`, `player_dialogue.prompt`, `player_thoughts.prompt` | +| `prompts/components/` | Reusable building blocks for `render_template("components\\X")` | `event_history.prompt`, `event_history_compact.prompt`, `event_history_verbose.prompt`, `agent_tools_base.prompt`, `memory_access.prompt`, character_bio variants | +| `prompts/components/context/` | Context-specific component variants | `scene_context.prompt`, `component_npc_state_summary.prompt` | +| `prompts/submodules/system_head/` | System-prompt prelude pieces | `0010_instructions.prompt`, `0250_omnisight.prompt`, … | +| `prompts/submodules/user_final_instructions/` | User-message tail pieces | `0750_embedded_actions.prompt` (ACTION: format), audio tag rules, recent state changes | +| `prompts/submodules/guidelines/` | Cross-cutting style rules | `0900_response_format.prompt` (asterisk narration rules, length limits) | +| `prompts/submodules/character_bio/` | Per-actor bio submodules merged into `dynamic_character_bio.prompt` | 17 numbered files: header, summary, personality, equipment, etc. | +| `prompts/submodules/omnisight_*/` | Vision-model description templates (one dir per target type: actor, scene, item, location, furniture, default) | matched 1:1 by `prompts/omnisight/describe_*.prompt` | +| `prompts/submodules/test_decorators/` | Test-data harness for the in-game template debugger | enabled by MCM toggle | +| `prompts/target_selectors/` | Meta classifier prompts for "who speaks next" | `dialogue_speaker_selector.prompt`, `player_dialogue_target_selector.prompt` | +| `prompts/transformers/` | Text→text transformations | `native_dialogue_transformer.prompt`, `universal_translator.prompt` | +| `prompts/memory/` | Memory generation, ranking, mood evaluation | `generate_memory.prompt`, `memory_ranker.prompt`, `mood_evaluator.prompt` | +| `prompts/helpers/` | Standalone classifier/profile generation prompts | `evaluate_mood.prompt`, `generate_profile.prompt`, `generate_search_query.prompt` | +| `prompts/omnisight/` | Vision-model prompts (one per target type) | `describe_actor.prompt`, `describe_scene.prompt`, `describe_item.prompt`, `describe_location.prompt`, `describe_furniture.prompt` | +| `prompts/dev/` | Developer test prompts | `mcm_test.prompt` | +| `prompts/documentation/` | Auto-generated decorator documentation rendered into the in-game web UI | `main.prompt`, `category.prompt` | +| `prompts/web/` | Web-UI bundled base templates | `bundled_base.prompt` | +| `prompts/translation/{generic,unique}/` | Localization CSVs | `00_SkyrimNet_generic.csv`, `00_SkyrimNet_unique.csv` | + +--- + +## Editing prompts safely — checklist + +1. **Is there an override in `overwrite/prompts/`?** If yes, that's the active version. Edit it (the in-game UI does this) or delete it to fall back to `mods/prompts/`. +2. **Diff against `original_prompts/`** to see what the upstream baseline says. +3. **Make a `.backup` next to the file** before significant edits — SkyrimNet itself does this convention with epoch-stamped backups; we use the simpler `.backup` suffix. +4. **Hot-reload picks up changes** — no need to restart Skyrim for prompt edits, but model behavior may take effect on the next agent firing. +5. **Test against logs** — watch `openrouter_input.log` to see what prompt text the model actually receives after your edit.