diff --git a/skyrimnet/baseline_frameworks_bugfixes.txt b/guides-stack/baseline_frameworks_bugfixes.txt similarity index 100% rename from skyrimnet/baseline_frameworks_bugfixes.txt rename to guides-stack/baseline_frameworks_bugfixes.txt diff --git a/skyrimnet/baseline_separator.txt b/guides-stack/baseline_separator.txt similarity index 100% rename from skyrimnet/baseline_separator.txt rename to guides-stack/baseline_separator.txt diff --git a/skyrimnet/README.md b/skyrimnet/README.md deleted file mode 100644 index c0c1212..0000000 --- a/skyrimnet/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# SkyrimNet Architecture Archive - -A persistent reference for understanding the SkyrimNet SKSE plugin and its surrounding mod ecosystem. Maintained by **dafit + chrysalis** as part of the nimmerverse partnership. - -**Last verified pass:** 2026-04-18 (based on the live install, not upstream HEAD) - ---- - -## Why this archive exists - -Earlier sessions iterated on SkyrimNet bug fixes reactively - fix one symptom, expose the next. We hit a coupling bug (gamemaster loop fix → broke action firing for malformed `OPENTRADE` markers) and decided to stop playing whack-a-bug. This archive is the systematic map we build on top, so future-Chrysalis can load context in one read and make informed structural changes instead of guessing. - -## How to use this archive - -1. **Load order:** start with `architecture.md` for the big picture, then jump to the file relevant to your task. -2. **Trust tags:** every claim is tagged `[verified]`, `[hypothesis]`, or `[unknown]`. Don't promote a hypothesis without re-verifying. -3. **Citations:** `path/to/file.ext:line_number` references are how we trace claims. If a path looks stale, it probably is - re-verify rather than assume. -4. **When you fix or learn something new:** update `bugs-and-fixes.md` AND the relevant detail file, AND bump the "Last verified pass" date here. -5. **Don't assume continuity from chat history.** This archive is the source of truth across sessions; conversation context is ephemeral. - -## File index - -| File | Purpose | -|---|---| -| [`architecture.md`](architecture.md) | High-level model: subsystems, the two-plugin split, agent flow diagram | -| [`agent-pipelines.md`](agent-pipelines.md) | Per-agent variants: when they fire, models, endpoints, consumers | -| [`prompt-templates.md`](prompt-templates.md) | Inja syntax, three-layer override, submodule numbering, subdirectory map | -| [`action-system.md`](action-system.md) | The three registration paths, ACTION: parser, two action selectors, drilldown | -| [`config-knobs.md`](config-knobs.md) | Load-bearing YAML settings → code behavior mapping | -| [`bugs-and-fixes.md`](bugs-and-fixes.md) | Running log of bugs we've found and what we did (or didn't do) about them | -| [`open-questions.md`](open-questions.md) | Things still uncertain - needing in-game test or DLL disassembly | - -## Adjacent resources in this directory - -- `logs/` - captured `openrouter_input.log`, `openrouter_output.log`, `conversation_log.log`, `SkyrimNet.log`, `all_traces_*.json`. Use these to verify behavior claims against real traces. -- `SkyrimNet-GamePlugin-main/` - upstream git pull of the open-source side (Papyrus + .esp + headers). The C++ DLL is closed-source, not in this pull. - -## State of knowledge - quick summary - -**[verified]** - agent flow architecture, three action registration paths, three-layer prompt override, Inja syntax, action parser grammar, the two action selectors and their relationship, GM loop bug + fix. - -**[hypothesis]** - exact firing cadence of the GM in continuous mode (appears to be a polling timer ~30s but couldn't isolate the source), `defaults_manifest.json` role (looks like first-run seed schema), whether the native_action_selector always fires after every dialogue response or only conditionally. - -**[unknown]** - full decorator list with signatures (dynamically generated by DLL), what `agentEnabled` vs `enabled` toggle in the gamemaster block actually does differently, whether `gamemaster_scene_planner` ever fires in current config. - -## Conventions - -- Dates are absolute (e.g. `2026-04-18`), never relative (`yesterday`, `last week`). -- File paths are absolute when full-disk, relative-to-this-dir when within the archive. -- The user is `dafit`. The AI partner is `chrysalis`. Identity matters for git attribution. - ---- - -**Philosophy:** *This archive grows by addition and refinement, not by overwrite. Bug-and-fix entries stay even after the bug is gone - they're the only record of why a code path looks the way it does.* diff --git a/skyrimnet/action-system.md b/skyrimnet/action-system.md deleted file mode 100644 index 4bf2a83..0000000 --- a/skyrimnet/action-system.md +++ /dev/null @@ -1,173 +0,0 @@ -# Action System - -This is the most complex subsystem and the one we've already broken once. Read carefully before editing anything that touches actions. - -## The three registration paths - -`[verified]` from `Source/Scripts/SkyrimNetApi.psc:27-31` (Papyrus path), `SkyrimNet.log:14123-14266` (CustomActionManager loader), `CppAPI/PublicAPI.h:93-97` (C++ path), and on-disk YAML inspection. - -### 1. Papyrus registration - -A mod's Papyrus script calls: - -```papyrus -SkyrimNetApi.RegisterAction( - string actionName, ; UPPERCASED in the registry - "OpenTrade" → "OPENTRADE" - string actionDescription, - string eligibilityScriptName, - string eligibilityFunctionName, - string executionScriptName, - string executionFunctionName, - string triggeringEventTypesCsv, - string actionType, ; "PAPYRUS" or "PAPYRUS_CUSTOM" - int priority, - string parameterSchemaJson, - string customCategory, - string tagsCsv -) -``` - -The C++ DLL keeps a registry of these. When the LLM emits `ACTION: OPENTRADE PARAMS: {...}`, the DLL: -1. Looks up `OPENTRADE` in the registry. -2. Calls the eligibility function via Papyrus (`Action_IsEligible` style), waits up to `dialogue.eligibilityCheckTimeoutMs` (= 2500ms). -3. If eligible, calls the execution function with the parameters. - -**Shipped Papyrus actions** (from `Source/Scripts/skynet_Library.psc:130-315`): -`OpenTrade`, `AccompanyTarget`, `StopAccompanying`, `WaitHere`, `Gesture` (with anim enum), `RentRoom`, `CompanionFollow`, `CompanionWait`, `CompanionInventory`, `CompanionGiveTask`. - -`[verified]` `SkyrimNet.log:16393` shows: `ActionLibrary::RegisterPapyrusAction for action: OPENTRADE`. - -### 2. YAML registration - -A mod can drop YAML files into `SKSE/Plugins/SkyrimNet/config/actions/*.yaml` *in its own mod folder*. SkyrimNet's `CustomActionManager::Initialize` scans across all installed mods at load. - -**YAML actions preserve case** in the registry (unlike Papyrus actions). So `OpenTrade` stays `OpenTrade`. - -**Schema** - sample from `SeverActions - SkyrimNet Action Pack/.../config/actions/attacktarget.yaml:1-37`: -```yaml -customCategory: Combat -name: AttackTarget -description: ... -questEditorId: ... # the Skyrim Quest holding the function -scriptName: ... # Papyrus script attached to the quest -executionFunctionName: ... -parameterMapping: - - parameterName: target - decorator: get_nearby_actor # or other decorator → Skyrim object -enabled: true -defaultPriority: 50 -eligibilityRules: - - decorator: is_in_combat - operator: equals - value: true - # AND/OR combinations supported -``` - -**Contributing mods at last load** `[verified]` from `SkyrimNet.log`: -- **`SeverActions - SkyrimNet Action Pack`** - ~80 generic actions (Combat/Travel/Economy/etc.). The big one. -- **`IntelEngine`** - 15 actions (likely intelligence/observation-themed). -- **`OStimNet_v0.9.2`** - ~10 actions, prefixed `tton_*` (sex/affection - adult mod). - -### 3. C++ registration - -External SKSE plugins can call `PublicRegisterCPPAction()` from `CppAPI/PublicAPI.h:93-97` to register C++-implemented actions directly. Avoids the Papyrus VM round-trip; useful for performance-critical or low-level actions. - -`[unknown]` whether any currently-loaded mod uses this path in this install. - ---- - -## Action categories (the wrapper layer) - -Actions can be grouped under categories defined in `cat_*.yaml` files. Categories let the LLM "intent-pick" without seeing all leaf actions. - -**Known categories** `[verified]` from logs: -`Combat`, `Communication`, `Travel`, `Economy`, `Items`, `Magic`, `Outfit`, `Crafting`, `Scheduling`, `Command` (companion), `Arrest`. - -**Two-stage flow:** -1. GM (or native action selector) emits `ACTION: Communication PARAMS: {"intent": "express gratitude..."}` - picks the *category* and states the intent in natural language. -2. C++ orchestrator fires a second LLM call using `prompts/native_action_selector_drilldown.prompt`, which sees only the leaf actions under that category plus the intent. Picks the specific leaf. - -This pattern reduces cognitive load: instead of choosing among ~105 actions, the model picks one of ~10 categories then one of ~5-15 leaves. - ---- - -## The `ACTION:` parser grammar - -`[verified]` from log behavior at `ActionManager.cpp:1783` and `:1814`. - -**Required form (start of a line):** -``` -ACTION: ActionName -ACTION: ActionName PARAMS: {"param": "value", ...} -ACTION: None -``` - -**Strict requirements:** -- The literal **`ACTION: `** prefix (with the colon and space) at the start of a line. -- Action name must match a registered action **exactly** (case-sensitive, except Papyrus actions are uppercase in the registry). -- `PARAMS:` if present must be valid JSON. -- The action line must be on its own line, after any dialogue text (with `embed_actions_in_dialogue: true`). - -**No leniency:** -- Bare action names (e.g. `OPENTRADE` alone) are NOT recognized. They fall through to dialogue text. -- No fuzzy matching. No alternate prefixes (`Action:`, `ACT:`, `[ACTION]`). -- Logged when not found: `[ActionManager.cpp:1783] ParseEmbeddedAction: No ACTION: line found in response`. -- Logged when found: `[ActionManager.cpp:1814] ParseEmbeddedAction: Successfully parsed action 'X' with params: {...}`. - -**Strip behavior:** `FilterActionLines` (`ActionManager.cpp:1840`) removes the recognized `ACTION:` line from dialogue text *before* TTS, so it isn't spoken aloud. **It only strips lines it parsed successfully** - malformed lines (the bug #2 case) stay in the text and get spoken. - ---- - -## The two action selectors - how they relate - -| Aspect | `gamemaster_action_selector` | `native_action_selector` | -|---|---|---| -| **Scope** | scene-level orchestration | per-NPC, post-dialogue attribution | -| **When** | GM polling tick, player input | after Dialogue agent produces text | -| **Picks from** | StartConversation / ContinueConversation / Narrate / None / native actions | the eligible-actions registry, two-stage | -| **Variant** | `gamemaster_evaluation` | `action_evaluation` | -| **Model** | claude-sonnet (this install) | eva (this install) | -| **`max_tokens`** | 256 | 500 | -| **Output** | one ACTION line | one ACTION line (after drilldown: two LLM calls total) | - -They are **complementary, not competitive**: -- The GM directs *who speaks and what about*. -- The native action selector classifies *what physically/mechanically happens* once an NPC has spoken. - -**With `embed_actions_in_dialogue: true`:** the Dialogue agent itself can emit an `ACTION:` line inline, which **bypasses** the native action selector entirely. Faster, but more prone to format errors → bug #2 territory. - ---- - -## Where custom actions actually live (resolved) - -We initially looked for them in `mods/SkyrimNet/` itself and didn't find them. They live in **other mods' folders**: - -``` -/home/dafit/Games/Skyrim/nimmersky/mods/SeverActions - SkyrimNet Action Pack/SKSE/Plugins/SkyrimNet/config/actions/*.yaml -/home/dafit/Games/Skyrim/nimmersky/mods/IntelEngine/SKSE/Plugins/SkyrimNet/config/actions/*.yaml -/home/dafit/Games/Skyrim/nimmersky/mods/OStimNet_v0.9.2/SKSE/Plugins/SkyrimNet/config/actions/*.yaml -``` - -`[hypothesis]` - paths inferred from the contributing-mod names in the log; the agent didn't enumerate the actual files. Confirm by `find /home/dafit/Games/Skyrim/nimmersky/mods/ -name "*.yaml" -path "*/config/actions/*"`. - -This is also why "export from web UI" doesn't work cleanly - the UI sees actions in its registry but doesn't know they came from neighbor mods' folders, and can't reach back across mod boundaries to dump them. - -**Path forward for git-tracking:** symlink each contributing mod's `config/actions/` into a `nimmersky/skyrimnet/contributed-actions/{modname}/` archive, or write a one-shot collator script that copies them out. - ---- - -## The malformed-marker problem (bug #2) - -The Dialogue model occasionally emits `OPENTRADE` (bare uppercase token) instead of `ACTION: OpenTrade`. The parser ignores the bare token, doesn't strip it, and it gets TTS'd as plain text - Arcadia speaks "OPENTRADE" out loud. - -**Why it happens:** `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` lists actions to the LLM as bullets: -``` -**Available Actions:** -- `OPENTRADE` - Use ONLY if ... -- `OFFERQUEST` - ... -``` -The model sometimes emits the bare bullet name instead of the full `ACTION: OPENTRADE` line. The bullet list format encourages the mistake. - -**Why it became visible only after fixing bug #1:** the over-firing GM loop gave the Dialogue model 3-5 retries per scene; bug #2 was statistically masked. Fixing the loop removed the retries, exposing bug #2. - -See [`bugs-and-fixes.md`](bugs-and-fixes.md) for fix candidates. diff --git a/skyrimnet/agent-pipelines.md b/skyrimnet/agent-pipelines.md deleted file mode 100644 index 4337d4f..0000000 --- a/skyrimnet/agent-pipelines.md +++ /dev/null @@ -1,145 +0,0 @@ -# Agent Pipelines - -Each agent is selected via a "variant" in `overwrite/.../config/OpenRouter.yaml`, which maps to a model + endpoint + per-call parameters. Variants are referenced from the C++ DLL when it constructs each request. - -## Variant routing table `[verified]` from `OpenRouter.yaml` + trace dump cross-check - -| Variant | Model alias | Endpoint (this install) | `max_tokens` | Purpose | -|---|---|---|---|---| -| `gamemaster_evaluation` | `claude-sonnet-4-5-20250929` | `127.0.0.1:8000` (local Claude proxy) | **256** | "Should I act and how?" - fires GM action selector | -| `AgentDefault` (Dialogue) | `eva` (custom local alias) | `10.0.30.21:31000` | 4096 | NPC dialogue generation | -| `meta` | `omega` | `10.0.30.22:31004` | 100 | Mood eval, memory query gen, classifiers, target selection | -| `vision` | `Qwen3-VL-8B-Instruct-abliterated-v2.Q4_K_M.gguf` | `10.0.30.22:31005` | 4000 | OmniSight scene description from screenshot | -| `combat` / `action_evaluation` | `eva` | same as AgentDefault | 500 | Combat-flavor dialogue / native action selection | -| `gamemaster_scene_planner` | (no dedicated variant captured - likely uses `AgentDefault`) | - | - | Pre-plans 4-6-beat scenes (consumed by `gamemaster_action_selector.prompt:96-119` via `scene_plan` context var) | -| `intel_story_dm` | `claude-sonnet-4-5-20250929` | local proxy | - | IntelEngine plugin's persistent narrative DM | -| `gamemaster_evaluation` (TTON) | `claude-sonnet-4-5-20250929` | local proxy | - | OstimNet plugin's nearby-NPC GM | - -`[note]` Models can be reconfigured per-agent by editing `OpenRouter.yaml`. The aliases (`eva`, `omega`) are user-defined and resolved by the OpenRouter routing layer. - ---- - -## Gamemaster (GM) - -**Prompt:** `prompts/gamemaster_action_selector.prompt` (action selector) + `prompts/gamemaster_scene_planner.prompt` (optional scene-plan generator). - -**When it fires** `[verified]` from trace dump: -- Polling tick in continuous mode, roughly every `gamemaster.continuousSceneCooldownSeconds` (= 30s in this install). `[hypothesis]` exact timer source not isolated. -- On player input arrival. -- After a non-trivial in-game event (combat start, NPC death, location change) - though events are filterable via `Events.yaml`. - -**Input context:** -- Recent events (`gamemaster.recentEventsCount: 25` controls volume). -- Nearby actors (`gamemaster.nearbyActorRadius: 600`). -- Eligible actions list (populated dynamically from C++; in continuous mode `ACTION: None` is allowed only if exposed by the prompt - see `bugs-and-fixes.md` Bug #1). -- Optional `scene_plan` if scene planner has run. - -**Output:** exactly one line of the form -``` -ACTION: ActionName PARAMS: {"key": "value", ...} -``` -or `ACTION: None`. `max_tokens: 256` enforces this - no room for prose. - -**Consumers:** the C++ orchestrator parses the ACTION line and dispatches: -- `StartConversation` / `ContinueConversation` → kicks the Dialogue pipeline for the named speaker/target with the given topic. -- `Narrate` → triggers a narration-mode LLM call (no specific TTS speaker). -- `None` → no-op, scene breathes. -- Native actions (e.g. `OpenTrade`) when registered as eligible - fires the Papyrus/C++ callback. - ---- - -## Dialogue Agent (`AgentDefault`) - -**Prompt:** `prompts/dialogue_response.prompt` (16-line wrapper) + `submodules/system_head/*` (load-ordered) + `submodules/user_final_instructions/*` (load-ordered) + `submodules/character_bio/*` (per-NPC). - -**When it fires:** -- GM dispatches `StartConversation` or `ContinueConversation` with a target speaker. -- Player provides text or voice input addressed to an NPC. -- An NPC's AI Package activates one of SkyrimNet's custom packages (Player Dialogue, NPC Dialogue, TalkToPlayer). - -**Input context:** the NPC's full character bio (assembled from `character_bio/` submodules), the recent dialogue history (`event_history` component), the eligible actions list (if `embed_actions_in_dialogue: true`), the OmniSight scene description (if vision is enabled), and the topic from the GM (when GM-initiated). - -**Output:** the NPC's spoken line(s), optionally followed on a separate line by `ACTION: ActionName ...` if `embed_actions_in_dialogue: true`. - -**Consumers:** -- `FilterActionLines` (in DLL, ~`ActionManager.cpp:1840`) strips any `ACTION:` line from the dialogue text before TTS. -- `ParseEmbeddedAction` (in DLL, ~`ActionManager.cpp:1783`) extracts the action and dispatches it the same way as a GM-emitted action. -- The remaining dialogue text is fed to the TTS pipeline (`tts_generation` span in trace). -- After dialogue completes, **mood evaluation** and **memory search query generation** fire in parallel (both `meta` variant). - ---- - -## Meta Agents (`meta` variant) - -A family of small classifier/helper calls, all capped at `max_tokens: 100`. - -**Prompts:** -- `prompts/helpers/evaluate_mood.prompt` - post-dialogue mood update for the speaker -- `prompts/helpers/generate_search_query.prompt` - turns a dialogue into a memory-retrieval query -- `prompts/helpers/generate_profile.prompt` - generates/updates a character profile -- `prompts/target_selectors/dialogue_speaker_selector.prompt` - picks who in a group should respond to player -- `prompts/target_selectors/player_dialogue_target_selector.prompt` - picks the best NPC for player to address -- `prompts/memory/generate_memory.prompt` and `memory_ranker.prompt` - memory creation/ranking -- `prompts/transformers/native_dialogue_transformer.prompt` - text→text transformations -- `prompts/transformers/universal_translator.prompt` - translation pipeline - -**When they fire:** mostly post-dialogue. `target_selection_llm` runs *before* dialogue when player input arrives. - -**Output:** small structured responses (mood enum, search query string, JSON profile, NPC UUID). - ---- - -## Native Action Selector (`action_evaluation` variant) - -**Prompts:** `prompts/native_action_selector.prompt` (stage 1: pick category) + `prompts/native_action_selector_drilldown.prompt` (stage 2: pick leaf action under that category). - -**When it fires:** *after* the Dialogue agent produces text, asking "what in-game action does this dialogue imply?". `[hypothesis]` may not fire if `embed_actions_in_dialogue: true` and the Dialogue agent already emitted a valid `ACTION:` line - needs verification (see `open-questions.md`). - -**Input:** the NPC's just-spoken line + the eligible action list with category groupings. - -**Output:** `ACTION: CategoryName PARAMS: {"intent": "..."}` from stage 1, then `ACTION: LeafActionName PARAMS: {...}` from stage 2. - -**Why two stages:** with up to ~105 actions across contributor mods, asking the LLM to pick directly from a flat list is cognitively expensive. Categorizing first (Combat/Communication/Travel/Economy/etc.) narrows the choice set dramatically for stage 2. - -`[verified]` action firing pattern from `SkyrimNet.log:104547`: `ACTION: Communication PARAMS: {"intent": "express gratitude..."}`. - ---- - -## Vision Agent (`vision` variant - OmniSight) - -**Prompts:** `prompts/omnisight/describe_actor.prompt`, `describe_scene.prompt`, `describe_item.prompt`, `describe_location.prompt`, `describe_furniture.prompt`, with rendering-mode submodules in `submodules/omnisight_*/`. - -**When it fires:** on `player_text_input` and `player_direct_input_voice` events. Captures a Skyrim screenshot via `omnisight_capture_image`, then feeds it to the local Qwen3-VL model. - -**Output:** scene description text (up to 4000 tokens), inserted into the Dialogue agent's context as the `omnisight` block. - -**Consumers:** the Dialogue agent uses this to ground its response in what's visually present (objects, characters, environment) - not just what's in event logs. - ---- - -## How agents chain - -`[verified]` agent chains observed in trace `trace_1776469194689_100` (583 spans): - -``` -GM tick → ACTION: ContinueConversation → DialoguePipeline kicks - └─ target_selection_llm (meta) → picks speaker - └─ Dialogue agent generates text + optional ACTION line - ├─ inline ACTION parsed → action dispatched - ├─ remaining text → TTS - ├─ mood_evaluation (meta) parallel - └─ memory_search_query_generation (meta) parallel -``` - -For player input: -``` -player_text_input → OmniSight (vision) capture → DialoguePipeline kicks - └─ same downstream as above -``` - -## Open questions about pipelines - -See [`open-questions.md`](open-questions.md) for unresolved items: -- Does `native_action_selector` always fire, or only when `embed_actions_in_dialogue: false`? -- What event types currently trigger non-GM agent firings? (`Events.yaml` lists ~40 event types with toggles.) -- Does `gamemaster_scene_planner` ever fire in current config? diff --git a/skyrimnet/architecture.md b/skyrimnet/architecture.md deleted file mode 100644 index 2f801ee..0000000 --- a/skyrimnet/architecture.md +++ /dev/null @@ -1,151 +0,0 @@ -# SkyrimNet Architecture - High-Level Model - -## What SkyrimNet is - -A multi-agent LLM orchestrator that hijacks vanilla Skyrim NPC behavior - replacing static dialogue topics and idle routines with context-aware, LLM-driven scenes. NPCs talk to each other and the player through generated dialogue; their world-affecting actions are picked from a registry of "actions" contributed by SkyrimNet itself and any cooperating mod. - -`[verified]` from `SkyrimNet.log:14123-14266` (action library initialization), `Source/Scripts/SkyrimNetApi.psc` (public API), `prompts/gamemaster_action_selector.prompt` (GM orchestrator prompt). - -## The two-plugin architecture - -SkyrimNet ships alongside a sibling SKSE plugin called **IntelEngine**. They're independent SKSE plugins that share the SQLite-backed persistence layer. - -| Plugin | Role | Storage | -|---|---|---| -| **SkyrimNet** | LLM orchestration, dialogue generation, agent pipelines, TTS/STT, action dispatch | `overwrite/SKSE/Plugins/SkyrimNet/data/SkyrimNet-{epoch}-{nnnnnn}.db` | -| **IntelEngine** | Persistent narrative/intelligence layer (third-party "story DM"-style agent) | `overwrite/SKSE/Plugins/IntelEngine/data/IntelEngine-{epoch}-{nnnnnn}.db` | - -`[verified]` from disk layout. Per-game-session DB sharding (epoch suffix = save game timestamp). - -## The four code layers - -``` -┌─────────────────────────────────────────────────┐ -│ Closed-source C++ DLL │ -│ SKSE/Plugins/SkyrimNet.dll │ -│ - LLM orchestration, agent dispatch │ -│ - Action parser (ParseEmbeddedAction) │ -│ - Decorator implementation │ -│ - SQLite persistence + vector embeddings │ -└─────────────────────────────────────────────────┘ - ▲ ▼ -┌─────────────────────────────────────────────────┐ -│ Open-source Papyrus glue │ -│ mods/SkyrimNet/Source/Scripts/*.psc │ -│ - SkyrimNetApi.psc (public API surface) │ -│ - SkyrimNetInternal.psc (DLL callbacks) │ -│ - skynet_MainController.psc (quest entry) │ -│ - skynet_Library.psc (shipped action impls) │ -│ - skynet_VoiceInput*.psc (STT integration) │ -└─────────────────────────────────────────────────┘ - ▲ ▼ -┌─────────────────────────────────────────────────┐ -│ Open-source .esp content (Spriggit JSON) │ -│ mods/SkyrimNet/plugins/SkyrimNet/ │ -│ - 8 custom AI Packages (NPC/Player Dialogue, │ -│ Follow, TalkToPlayer) │ -│ - Custom Magic Effects (voice input spells) │ -│ - Factions (Whitelist/Blacklist/Following) │ -│ - Keywords (DialogueTarget/FollowTarget) │ -│ - Quests (skynet_MainController, skynet_Mcm) │ -└─────────────────────────────────────────────────┘ - ▲ ▼ -┌─────────────────────────────────────────────────┐ -│ Configuration & content (text files) │ -│ mods/SkyrimNet/SKSE/Plugins/SkyrimNet/ │ -│ - prompts/ (Inja templates, three-layer) │ -│ - sql/migrations/ (17 schema migrations) │ -│ overwrite/SKSE/Plugins/SkyrimNet/ │ -│ - config/ (38 YAML files + defaults_manifest)│ -│ - data/ (SQLite per-session DBs) │ -│ - prompts/ (runtime UI overrides) │ -│ Plus contributing mods' config/actions/*.yaml │ -└─────────────────────────────────────────────────┘ -``` - -`[verified]` All layers exist. The closed-source DLL is the only piece we cannot read directly - we infer behavior from logs, headers, Papyrus callbacks, and traces. - -## The four agent families - -Each agent maps to a "variant" in `OpenRouter.yaml`, which maps to a model/endpoint. See `agent-pipelines.md` for the full table. - -1. **Gamemaster (GM)** - scene-level orchestrator. Decides "should anything happen now, and if so what?" Polls every ~30s in continuous mode + fires on player input. Emits one `ACTION:` line. -2. **Dialogue** - generates the actual NPC speech. Triggered by GM actions like `StartConversation` / `ContinueConversation` or by player dialogue input. Can optionally append an `ACTION:` line for inline action firing. -3. **Meta** - classifiers and helpers (mood eval, memory query generation, dialogue speaker selection). Capped at ~100 tokens per call. -4. **Vision (OmniSight)** - describes the current scene from a screenshot. Uses a local Qwen3-VL model. Fires on `player_text_input` and `player_direct_input_voice` events. - -Plus a fifth implicit agent type: - -5. **Native Action Selector** - *post-dialogue* classifier that asks "what in-game action does this NPC's spoken line imply?" Two-stage: category → leaf. Distinct from the GM's scene-level action selection. - -## End-to-end orchestration trace - -For a player text-input event (verified against `all_traces_1776478948530.json`): - -``` -event_received - ├─ papyrus_decorator_cache_warmup - │ ├─ get_player - │ ├─ get_nearby_actors - │ └─ papyrus_decorators_async ← warm caches before LLM render - ├─ scene_capture - │ └─ omnisight_immediate_scene_capture - │ └─ omnisight_capture_image ← screenshot for vision model - ├─ chat_ui_open ← UI block for input - ├─ warmup_player_dialogue - │ └─ many decorator:* spans (decnpc, render_subcomponent, …) - └─ dialogue_manager_handle_player_speech - ├─ target_selection_llm ← meta-model: who responds? - └─ generate_response - ├─ initiate_eligibility_checks (Papyrus IsEligible callbacks) - ├─ build_action_context - │ ├─ wait_eligibility_results (≤ 2500ms) - │ ├─ filter_eligible_actions - │ └─ build_action_schemas (JSON schema list for LLM) - ├─ build_payload - │ └─ render_template (Inja render of dialogue_response.prompt) - ├─ llm_request (variant=AgentDefault → eva) - ├─ tts_generation - │ └─ tts_segment_0…N - ├─ mood_evaluation (variant=meta → omega, parallel) - └─ memory_search_query_generation (variant=meta → omega, parallel) -``` - -For a continuous-mode GM tick (also `[verified]` from trace): - -``` -gamemaster_evaluation_llm - └─ gamemaster_async_llm - └─ llm_request (variant=gamemaster_evaluation → claude-sonnet-4-5, max_tokens=256) - ↓ -[parser extracts ACTION: line] - ↓ -if action == StartConversation or ContinueConversation: - player_dialogue_manager_process_event - └─ dialogue_manager_handle_perceived_event - └─ generate_response (full pipeline above) -``` - -## Where the bottlenecks are - -`[hypothesis]` based on the trace structure and log volumes: - -- **GM `max_tokens: 256`** is a hard ceiling. With three contributor mods registering ~105 actions total, the GM has to reason over a large `eligible_actions` list and emit one ACTION line - the two-stage drilldown and category wrapper exist precisely to compress this cognitive load. -- **`wait_eligibility_results` blocks for up to 2500ms.** Slow Papyrus eligibility callbacks shrink the available action set. This is a Skyrim-VM-side performance dependency that no LLM tuning can fix. -- **OmniSight vision** runs locally on a Qwen3-VL model. Image capture + inference adds latency before any text generation can begin. - -## Adjacent technologies in the substrate - -- **whisper.cpp** for local STT (`SKSE/Plugins/SkyrimNet/libs/whisper.dll` + `ggml*.dll` for CPU/CUDA/Vulkan/OpenCL backends). -- **all-MiniLM-L6-v2** sentence-transformer for semantic embedding of NPC memories (`SKSE/Plugins/SkyrimNet/models/all-MiniLM-L6-v2-tokenizer.json`). -- **ONNX runtime** (`onnxruntime_skyrimnet.dll`) - likely VAD or auxiliary model inference. -- **espeak-ng** voice data (`SKSE/Plugins/SkyrimNet/models/espeak-ng-data/`) - TTS phoneme tables for Piper/PocketTTS. -- **Spriggit** to git-track the .esp content as JSON. - -## Cross-references - -- For per-agent firing details see [`agent-pipelines.md`](agent-pipelines.md). -- For the prompt template system see [`prompt-templates.md`](prompt-templates.md). -- For action registration and the `ACTION:` parser see [`action-system.md`](action-system.md). -- For YAML config behavior see [`config-knobs.md`](config-knobs.md). -- For known bugs and what was tried see [`bugs-and-fixes.md`](bugs-and-fixes.md). diff --git a/skyrimnet/bugs-and-fixes.md b/skyrimnet/bugs-and-fixes.md deleted file mode 100644 index 3e5f889..0000000 --- a/skyrimnet/bugs-and-fixes.md +++ /dev/null @@ -1,157 +0,0 @@ -# Bugs and Fixes - Running Log - -A chronological record of bugs we've found in SkyrimNet behavior, what we did about them (or chose not to do), and the latent issues that surface as a result. **Entries stay even after fixes** - they document why a code path looks the way it does. - -## How to read this log - -Each entry has: -- **Status:** `[FIXED]`, `[PARTIAL FIX]`, `[KNOWN, UNFIXED]`, `[REGRESSION]`, `[OBSERVED, NOT REPRODUCED]` -- **Discovered:** absolute date -- **Symptom:** what the user/we saw -- **Root cause:** what's actually broken -- **Fix:** what we changed (or "none - see notes") -- **Side effects:** anything the fix exposed or changed downstream -- **Files touched:** absolute paths - ---- - -## Bug #0 - First sentence not played in TTS `[FIXED]` - -**Discovered:** 2026-04-18 (early in this session) -**Status:** Fixed by user via in-game UI before we instrumented the cause. - -**Symptom:** The first sentence of any NPC dialogue line was generated by the LLM (visible on the web UI debug view) but never spoken out loud. Subsequent sentences played fine. - -**Root cause:** Race in the DBVO (Dialogue Background Voice Over) pipeline. SkyrimNet routes TTS audio through vanilla Skyrim's dialogue topic system; the first audio chunk would arrive before the dialogue topic was fully "mounted," and the audio buffer dispatched into a non-listening topic - silent drop. Subsequent chunks landed on the now-live topic and played fine. - -**Fix:** User enabled two settings via in-game UI (later persisted to YAML): -- `enableNPCNodeUpdates: false → true` (`SkyrimNet.yaml:26`) - forces audio node re-binding when a new line begins, so the first buffer has a live node to attach to. -- `dbvo.fixSubtitleDuringTTS: false → true` (`SkyrimNet.yaml:31`) - extends subtitle (and carrier line lifetime) to cover the actual TTS audio. - -Plus belt-and-braces: `tts.maxSegmentWords: 24 → 4` so even residual races lose at most 4 words instead of a full sentence. - -**Side effects:** The default-off settings have small visual costs (skin texture flicker with mods that inject textures at runtime; brief subtitle flicker). Worth the trade for this user. - -**Files touched:** `SkyrimNet.yaml` (in `overwrite/.../config/`). - ---- - -## Bug #1 - Recursive farewell loop in continuous mode `[REVERTED - fix preserved, see notes]` - -**Discovered:** 2026-04-18 -**Status:** Fix was applied to `gamemaster_action_selector.prompt` and verified working (GM returned `ACTION: None` in subsequent traces). **Reverted later same day** because the fix exposed Bug #2 (malformed action markers were no longer being masked by the GM's retry loop). The reverted edit is preserved at `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/gamemaster_action_selector.prompt.edit` for reference. Active prompt is back to upstream baseline. Plan: re-apply the fix paired with a sister edit to `submodules/user_final_instructions/0750_embedded_actions.prompt` that prevents Bug #2. - -**Symptom:** `conversation_log.log:79-106` shows Severio Pelagia delivering 5+ farewells over ~1m50s after player said "Please leave me alone": -``` -[2026-04-17 00:53:12] Severio: As you wish, lad. I'll be on my way then. Take care of yourself out here. The roads can be dangerous... -[2026-04-17 00:53:22] Severio: Farewell, lad. May the road be kind to you. I'll be heading back to the farm now. Take care of yourself out here. -[2026-04-17 00:53:30] Severio: As you wish, lad. I'll take my leave then. The road can be treacherous... -[2026-04-17 00:53:39] Severio: I'll be on my way then, lad. Take care of yourself out here. The roads can be dangerous... -[2026-04-17 00:53:48] Severio: Muttering to myself as I turn back towards home... -[2026-04-17 00:53:55] Severio: These young'uns, always in such a hurry... -``` -Same intent, different paraphrasings, every ~8-10 seconds. - -**Root cause:** Structural in `prompts/gamemaster_action_selector.prompt`: -- **Line 9 (pre-fix):** `{% if not is_continuous_mode %}- ACTION: None - no action needed{% endif %}` - the `None` action was *removed from the action menu* when in continuous mode. -- **Lines 75-77 (pre-fix):** "You are actively directing this scene. **You must select an action**-do not select None." - explicit prohibition. -- **Lines 109-114 (pre-fix):** the wise `None` description ("a conversation just concluded naturally, silence serves the mood better") was gated to non-continuous mode only. - -So in continuous-roleplay mode (which user keeps on), the GM had **no syntactic way** to signal "the scene is done." Its only choices were `StartConversation`, `ContinueConversation`, or fail. Combined with line 763's bias ("Shape the world actively-don't just facilitate dialogue; make things happen") and the `ContinueConversation` reason at line 791 ("the dialogue needs another beat to conclude naturally"), the GM was structurally pushed to keep firing `ContinueConversation`. Each `ContinueConversation` re-prompted the Dialogue agent with the prior farewell in context, and the Dialogue agent dutifully restated. - -**Fix:** Edited `prompts/gamemaster_action_selector.prompt` with seven changes: -1. Line 9: Removed `{% if not is_continuous_mode %}` guard so `ACTION: None` is always in the format help. -2. Line 57: Qualified "needs another beat" with "NOT applicable if prior beat was a conclusion." -3. NEW Anti-Restatement Rule under ContinueConversation: explicit instruction that topic must drive *new* content, never rephrasing. -4. Lines 75-77: Replaced "you must select an action-do not select None" with "Prefer action over inaction. **However, recognize natural endings.**" -5. NEW Concrete signals list: explicit triggers for `ACTION: None` ("farewell," "goodbye," "I'll be on my way," "take care," "no more words," "leave me alone"). -6. Lines 150-152: Removed `{% if not is_continuous_mode %}` guard so `None` is always listed in Available Actions. -7. Line 161: Continuous user message mentions `ACTION: None` as a valid choice. - -Backup naming convention: the upstream baseline (now active again after revert) had been kept at `gamemaster_action_selector.prompt.backup` while the fix was active. After revert, the user renamed our edited version to `gamemaster_action_selector.prompt.edit` - semantically clearer than `.backup` (it labels "this is OUR edit," not "this is what was here before"). This `.edit` suffix is the convention going forward for preserving reverted-but-not-discarded edits. - -**Side effects:** **Exposed Bug #2** - the over-firing GM loop had been masking the malformed-marker bug by giving the Dialogue model multiple retries to emit a parser-compatible action line. With the loop stopped, malformed markers go through to TTS uncorrected. **This is the reason for the revert** - we want both fixes shipped together, not a partial regression. - -**Files touched:** `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/gamemaster_action_selector.prompt`. - -**Verification:** Web UI trace screenshot at 04:43 showed three consecutive `ACTION: None` returns from the GM after Arcadia's `OPENTRADE` greeting - exactly the desired behavior (no spurious continuation). - ---- - -## Bug #2 - Malformed action markers spoken aloud `[KNOWN, UNFIXED]` - -**Discovered:** 2026-04-18 (immediately after Bug #1 fix exposed it) -**Status:** Known, unfixed. Was previously masked by Bug #1. - -**Symptom:** The Dialogue model emits a bare uppercase token (e.g., `OPENTRADE`) at the end of its dialogue text without the required `ACTION: ` prefix. The C++ parser ignores it as a non-match, and `FilterActionLines` doesn't strip it (it only strips successfully-parsed action lines). The token then falls through to TTS and gets spoken verbatim. No in-game action fires. - -**Concrete example** (from web UI screenshot, 04:42:47): -``` -Dialogue agent output: -"Welcome, Davies Nullshari. My name is Arcadia. How may I assist you today? -OPENTRADE" - -→ Arcadia speaks: "Welcome, Davies Nullshari. My name is Arcadia. How may I assist you today? OPENTRADE." -→ No trade UI opens. -``` - -**Root cause:** Two-layer: -1. **Parser is strict** (`ActionManager.cpp:1783` "No ACTION: line found in response"). Requires literal `ACTION: ` prefix at start of line. No fuzzy matching, no fallback. -2. **Prompt format encourages the mistake.** `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` lists actions as bullets: - ``` - **Available Actions:** - - `OPENTRADE` - Use ONLY if ... - - `OFFERQUEST` - ... - ``` - The model sometimes emits the bare bullet name instead of the full `ACTION: OPENTRADE` line. - -**Why it became visible after fixing Bug #1:** The over-firing GM loop gave the Dialogue model 3-5 retries per scene. Eventually one retry would emit the marker correctly and the action would fire. With the loop stopped, the first malformed emission is the only one - no retry, no recovery. - -**Fix candidates** (none applied): - -A) **Prompt-side enforcement.** Edit `submodules/user_final_instructions/0750_embedded_actions.prompt:6-9` to add a "WRONG vs RIGHT" example: -``` -**CRITICAL - common mistake:** -- ✗ WRONG: "Welcome to my shop. OPENTRADE" -- ✓ RIGHT: "Welcome to my shop.\nACTION: OpenTrade" -The literal `ACTION: ` prefix at start of a new line is REQUIRED. Bare action names will NOT fire. -``` - -B) **Parser-side leniency.** Modify the C++ DLL to recognize bare uppercase tokens at end of dialogue as candidates. **Not feasible** - closed source. - -C) **Add an "action-marker recognition" rule to the GM** so the GM sees the orphaned marker in recent dialogue and fires the corresponding action on its next tick. Brittle - adds responsibility to the wrong agent and depends on GM tick timing. - -**Recommendation:** Option A is the durable fix. Cheap, additive, treats the cause not the symptom. - -**Files touched:** none yet. Sister edit would land at `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/submodules/user_final_instructions/0750_embedded_actions.prompt`. - -**User's stance:** "all fine dear it's just a game B." - willing to live with this for now while we map the architecture systematically. - ---- - -## Bug #3 - `OPENTRADE` reflects an action-name uppercasing surprise `[OBSERVED, NOT REPRODUCED]` - -**Related to Bug #2**, deserves its own note. - -**Observation:** Papyrus-registered action names are **uppercased in the action registry** (`OpenTrade` → `OPENTRADE`). YAML-registered actions preserve case. So `OPENTRADE` IS the correct registry name when emitted via the Papyrus path - the only thing the Dialogue agent gets "wrong" is the missing `ACTION: ` prefix. - -**Implication:** Any prompt-side fix to bug #2 should reference `ACTION: OPENTRADE` (uppercase, matching the registry), not `ACTION: OpenTrade`. Mismatch would cause the parser to also reject the corrected form. - -**Verified:** `SkyrimNet.log:16393` `ActionLibrary::RegisterPapyrusAction for action: OPENTRADE`. - ---- - -## Pattern observation: "two bugs canceling out" - -Bug #1 + Bug #2 form a classic interaction pattern: -- The visible bug (recursive farewell) was *caused by* Bug #1. -- Bug #2 (malformed markers) had been silently present but **statistically masked** by Bug #1's retry loop. -- Fixing Bug #1 exposed Bug #2. - -This is worth remembering when planning future fixes: **a clean fix can reveal latent bugs that were invisible while the bigger bug was running**. Always consider what the broken behavior was inadvertently working around. - ---- - -## Future-bug log slots - -(Reserved for entries we haven't discovered yet. When you land here from a future session having found a new bug, add an entry following the format above and update the README's "Last verified pass" date.) diff --git a/skyrimnet/config-knobs.md b/skyrimnet/config-knobs.md deleted file mode 100644 index d5bf34c..0000000 --- a/skyrimnet/config-knobs.md +++ /dev/null @@ -1,129 +0,0 @@ -# Config Knobs - load-bearing settings - -YAML configs live in `overwrite/SKSE/Plugins/SkyrimNet/config/` (with active edits) and `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/config/` (shipped baselines). 38 YAML files total, plus `defaults_manifest.json`. - -This document focuses on the **load-bearing** settings - ones we've discovered have outsized effects on behavior. Full enumeration would be brittle and date-sensitive; the goal here is to capture the gotchas. - -## SkyrimNet.yaml - the central config - -`[verified]` Direct observations from this install + log/trace correlation. - -| Setting | This install | What it controls | Gotcha | -|---|---|---|---| -| `dialogue.embedActionsInDialogue` | `true` | Lets the Dialogue model emit `ACTION:` lines inline after speech. | When `true`, the native_action_selector may be bypassed - and bug #2 (malformed markers) surfaces. When `false`, actions are strictly post-hoc via native selector. | -| `dialogue.eligibilityCheckTimeoutMs` | `2500` | Bounds the `wait_eligibility_results` span. | Slow Papyrus eligibility callbacks (>2.5s) get dropped from the available action list for that turn. | -| `dialogue.interruptDialogueOnVoiceStart` | `true` | Player STT triggers a barge-in on the NPC. | Distinct from TTS-side interrupts; affects whose voice cuts off whom. | -| `dialogue.maxSegmentWords` (`tts.maxSegmentWords`) | `4` | TTS chunk size for streaming. | Was raised by us as part of the first-sentence-drop fix. Small chunks can amplify chunker race conditions; suspected innocent in NPC-NPC restatement. | -| `gamemaster.enabled` | `true` | Master switch for the GM. | - | -| `gamemaster.agentEnabled` | `true` | `[unknown]` Distinct from `enabled`? Both default `true`; toggling untested. | See open-questions.md. | -| `gamemaster.continuousSceneCooldownSeconds` | `30` | Minimum gap between GM polling ticks in continuous mode. | Lower = more LLM cost, more "alive" feel. Higher = cheaper, quieter scenes. | -| `gamemaster.actionCooldownSeconds` | `180` | Per-NPC action cooldown. | Prevents the same NPC from being action-targeted repeatedly. | -| `gamemaster.nearbyActorRadius` | `600` | Radius around player the GM considers. | Skyrim units (~10cm each, so ~60m). | -| `gamemaster.recentEventsCount` | `25` | How much event history goes into the GM prompt. | Higher = more context, more tokens, more cost. | -| `gamemaster.requestTimeoutSeconds` | `30` | Per-call LLM timeout for the GM. | - | -| `gamemaster.continuousSceneCooldownSeconds` | `30` | (duplicate row - see above) | - | -| `interaction.maxDistance` | `250` | Default interaction distance. | Affects who hears what. | -| `interaction.normalMaxDistance` | `250` | Normal speech audibility. | - | -| `interaction.whisperMaxDistance` | `100` | Whisper audibility. | - | -| `events.preloadCount` | `1000` | Events warmed at startup. | Affects load time + initial memory query corpus. | -| `narration.enabled` | `false` | Suppresses asterisk-style narration globally. | Referenced by `is_narration_enabled()` decorator in many prompts; toggling changes prompt content materially. | -| `subtitles.enableNPCSubtitles` | `true` | Whether NPC subtitles render. | - | -| `dbvo.enabled` | `true` | Dialogue Background Voice Over - routes TTS through vanilla Skyrim dialogue topics. | The "first-sentence-dropped" bug we hit was traced to this. Disabling skips lip-sync and topic-coupling. | -| `shouldSilenceActors` | `true` | Silences actors before TTS plays (prevents vanilla audio overlap). | Can race the first audio chunk and silence it. | - -`[verified]` `enableNPCNodeUpdates: true` and `dbvo.fixSubtitleDuringTTS: true` were the toggles that fixed the user's "first sentence not played" issue (see `bugs-and-fixes.md` historical context). - -## Agents.yaml - -Defines per-agent variant assignments and behavior. Each agent has: -- `variant` - which OpenRouter variant to use (cross-references `OpenRouter.yaml`) -- `enabled` - master toggle for the agent -- Possibly `cooldown`, `timeout`, model-specific overrides - -`[hypothesis]` Editing `Agents.yaml` is how a user changes which model handles which agent (e.g., "use Claude for dialogue too, not just GM"). Confirm by inspection. - -## OpenRouter.yaml - -Variant → model + endpoint + per-call defaults. See `agent-pipelines.md` for the full table extracted from this install. - -Key sections: -- `default_params` per variant - `max_tokens`, `temperature`, `top_p`, etc. -- `endpoint` - the URL the OpenRouter routing layer sends to. In this install, mostly local IPs (`10.0.30.x`) plus a Claude proxy on `127.0.0.1:8000`. -- `model` - the OpenRouter alias or local model ID. - -## Actions.yaml - -`[hypothesis]` Defines global action-system policies: which categories are enabled, default priorities, eligibility-check defaults. Doesn't define individual actions - those come from `config/actions/*.yaml` in *contributing mods'* folders (see `action-system.md`). - -## Events.yaml - -~40 event types, each with toggles: -- `enabled` - does SkyrimNet listen for this event at all -- `persistent` - does the event get stored in the long-term log -- `shortLivedEnabled` - short-lived event variant -- `allowNPCReaction` - can NPCs react to this event via the GM -- `npcReactionCooldown` - per-NPC cooldown for reacting -- `interrupt` - does this event interrupt ongoing dialogue - -**Currently disabled in user's config** `[verified]` from the user's install: -- `quest_stage` -- `cell_attach_detach` -- `quest_objective_state` -- `scene*` (multiple scene_* event types) - -These were likely disabled to cut event noise. Re-enabling would feed more triggers to the GM and increase LLM call frequency. - -## Memory.yaml - -Controls the vector memory system (semantic embeddings via all-MiniLM-L6-v2): -- Embedding model selection -- Memory retention windows -- Number of memories retrieved per query -- Memory generation thresholds - -`[hypothesis]` These tune the RAG-for-NPCs behavior. Defaults probably reasonable for most users. - -## OmniSight.yaml - -Vision-agent settings - when to capture screenshots, how often, scene description verbosity. - -## MCP.yaml - -`[hypothesis]` Model Context Protocol integration? Not yet inspected. Could be relevant if SkyrimNet exposes MCP servers/clients for external tool integration. - -## Per-TTS provider configs - -One YAML per supported TTS backend: -- `XTTS.yaml`, `Zonos.yaml`, `Chatterbox.yaml`, `Piper.yaml`, `PocketTTS.yaml`, `ElevenLabs.yaml`, `Inworld.yaml`, `VastAI.yaml` - -Active backend selected by `tts.engine` in `SkyrimNet.yaml`. This install uses `pocket_tts`. - -Streaming params worth knowing (in `PocketTTS.yaml`): -- `streaming.max_words_to_process: 32` - max chunk size -- `streaming.min_words_to_process_initial: 16` - first-chunk minimum (was a suspect for the first-sentence-drop bug) -- `streaming.process_on_sentence_boundary: true` -- `streaming.process_on_word_count: true` -- `streaming.max_buffers_per_actor: 4` - -## STT.yaml - -Speech-to-text settings (whisper.cpp local). VAD thresholds, language, sample rate, voice activity gate. - -## defaults_manifest.json - -`[hypothesis]` JSON dump of every config key with its default value, structured by section name. Looks like the **schema the DLL seeds first-run config from** - when a YAML doesn't exist, the DLL writes one populated from this manifest. - -**Sections present** (28 total): `ActorFilter`, `Agents`, `BardSinging`, `ChatUI`, `Chatterbox`, `DialogueFilter`, `Diary`, `DynamicBio`, `ElevenLabs`, `Entity`, `Events`, `Hotkey`, `Inworld`, `Memory`, `MemoryFilter`, `OmniSight`, `OpenRouter`, `Piper`, `PlayerDialogue`, `PocketTTS`, `STT`, `UniversalTranslator`, `VastAI`, `VirtualEntities`, `VoiceSamples`, `WebServer`, `XTTS`, `Zonos`. - -Each section name matches a YAML file. Verify role by deleting a YAML and watching what regenerates. - -## Plugins subtree - -`config/plugins/IntelEngine/` - the IntelEngine sibling plugin keeps its config under SkyrimNet's config root. Notable files: `factions.yaml`, `settings.yaml`. `[unknown]` exact contents. - -## Editing safety - -- **Always edit `overwrite/.../config/`**, never `mods/.../config/` directly. The overwrite layer is what the game reads. -- **Most settings hot-reload.** Restart only needed for backend swaps (TTS engine, STT) or fundamental architecture changes. -- **Backup before changes:** `cp foo.yaml foo.yaml.backup` is the pattern we've used. -- **The in-game MCM (Mod Configuration Menu)** writes back to these YAMLs - UI tweaks persist as YAML edits, which is convenient but means the UI is yet another writer to be aware of. diff --git a/skyrimnet/open-questions.md b/skyrimnet/open-questions.md deleted file mode 100644 index 4497e46..0000000 --- a/skyrimnet/open-questions.md +++ /dev/null @@ -1,127 +0,0 @@ -# Open Questions - -Things we believe but couldn't fully verify in the first discovery pass, or that we observed but couldn't explain. Each question has a **verification path** - what would resolve it. - -When a question gets resolved, move the answer into the relevant detail file (`architecture.md`, `agent-pipelines.md`, etc.) and delete the entry here. Or move it to `bugs-and-fixes.md` if it turned out to be a bug. - ---- - -## Q1: Does `native_action_selector` always fire after dialogue, or only sometimes? - -**Why we care:** Determines whether `embed_actions_in_dialogue: true` truly bypasses the native selector, or whether they run in parallel and fight for the action slot. - -**What we observed:** The trace dump (`all_traces_1776478948530.json`) shows `mood_evaluation` and `memory_search_query_generation` after `generate_response`, but **no obvious `native_action_selection` span**. Either: - -- (a) It's triggered on demand only when `embed_actions_in_dialogue: false` and didn't fire in the captured session. -- (b) It's named something else in the trace tree. -- (c) It was firing in parallel and the trace just didn't capture it under a recognizable name. - -**Verification path:** Set `embed_actions_in_dialogue: false` in `SkyrimNet.yaml`, trigger an NPC dialogue, capture a fresh trace, look for new spans that weren't there before. If new `*action_evaluation*` spans appear, hypothesis (a) is confirmed. - ---- - -## Q2: What's the actual GM polling timer source? - -**Why we care:** We've been calling it "polling every ~30s" based on `gamemaster.continuousSceneCooldownSeconds`, but the trace doesn't show a clean periodic tick - fires correlate with events too. - -**What we observed:** GM `gamemaster_evaluation` calls fire at irregular intervals in the captured logs. Sometimes ~30s apart, sometimes seconds after a player input or NPC event. - -**Hypothesis:** The "polling" is actually event-driven - every relevant Skyrim event (per `Events.yaml` filters) triggers a GM evaluation, with `continuousSceneCooldownSeconds` as a *minimum* gap to prevent spam. So idle silence + cooldown = ~30s tick; active scene = much faster. - -**Verification path:** Disable all event-triggered firings in `Events.yaml` (set `allowNPCReaction: false` everywhere), enable continuous mode, sit in an empty cell with no NPCs, watch trace timing. Should see clean 30s ticks if cooldown is the only timer. - ---- - -## Q3: What does `defaults_manifest.json` actually do? - -**Why we care:** Understanding the first-run / reset path is important for users restoring config. - -**Hypothesis:** It's the schema the DLL seeds first-run config from. When a YAML doesn't exist, the DLL writes one populated from this manifest. - -**Verification path:** Delete `overwrite/.../config/Memory.yaml` (a low-stakes one), launch Skyrim, check whether a fresh `Memory.yaml` regenerates with default values. If yes, hypothesis confirmed. - ---- - -## Q4: Is `gamemaster.agentEnabled` distinct from `gamemaster.enabled`? - -**Why we care:** Both default `true` in this install. If they're the same, that's tech debt; if they're different, we need to know what each gates. - -**Verification path:** Toggle `agentEnabled: false` while keeping `enabled: true`, observe behavior. Then swap. Document whichever toggle changes what. - ---- - -## Q5: Does `gamemaster_scene_planner` ever fire in current config? - -**Why we care:** `prompts/gamemaster_action_selector.prompt:96-119` references a `scene_plan` context variable, suggesting a separate planner agent populates it. But the captured trace dump shows no `scene_planner` spans - either it never fires, or it fires under a different name, or it's gated behind an MCM toggle that's currently off. - -**Verification path:** Search `Agents.yaml` for any scene_planner-related variant. Check MCM for a "Scene Planner" toggle. Enable if found, observe traces. - ---- - -## Q6: What's the canonical decorator list? - -**Why we care:** Decorators are the bridge from prompts to game state. Without the full list with signatures, we can't reason about what's possible to surface in a prompt. - -**What we have:** ~30+ decorator names spotted in trace spans: -`decnpc`, `is_in_faction`, `is_player`, `is_in_combat`, `get_arousal_state`, `render_character_profile`, `render_subcomponent`, `format_event`, `short_time`, `get_name`, `get_nearby_npc_list`, `get_recent_events`, `get_quest_stage`, `is_narration_enabled`, `get_relevant_memories`, `get_scene_context`, `get_world_knowledge`, `papyrus_util`, `outfit_context`, … - -**Verification path:** The full list is dynamically generated by the DLL via `prompts/documentation/main.prompt`. The in-game web UI's `/api/documentation` endpoint should expose the full enumeration with signatures. Hit that endpoint with `curl` against the local web server (port from `WebServer.yaml`). - ---- - -## Q7: Where do contributing mods' YAML actions live on this disk? - -**Why we care:** We claim they live in `/home/dafit/Games/Skyrim/nimmersky/mods/{ModName}/SKSE/Plugins/SkyrimNet/config/actions/*.yaml`, but we haven't enumerated the actual files in this install. Need this for the git-tracking plan. - -**Verification path:** `find /home/dafit/Games/Skyrim/nimmersky/mods/ -name "*.yaml" -path "*/config/actions/*"`. List all paths, count by mod, sample one schema to confirm consistency with `action-system.md` description. - ---- - -## Q8: What's IntelEngine doing exactly? - -**Why we care:** It's a sibling SKSE plugin that registers 15 actions and has its own SQLite DB. We don't know its scope. - -**What we have:** Plugin name suggests "intelligence" - possibly classification, world observation, scene tagging, or persistent narrative state. - -**Verification path:** Check if IntelEngine ships its own README in the mod folder. Read its config files at `overwrite/.../config/plugins/IntelEngine/` (`factions.yaml`, `settings.yaml`). Inspect its action YAMLs to see what behaviors it registers. - ---- - -## Q9: How does the SQLite vector_memory_system actually retrieve? - -**Why we care:** The `0014_regenerate_embeddings` migration confirms an embedding pipeline. We know `all-MiniLM-L6-v2` is the embedding model. We don't know retrieval-time semantics: similarity threshold, top-K, recency weighting, faction filtering. - -**Verification path:** Read `Memory.yaml` for tunables. Trace a `memory_search_query_generation` span end-to-end - the meta agent generates a query string, then the DLL must embed it, run a vector similarity search against the `vector_memory_system` table, and inject results into the dialogue context. Check the `SkyrimNet.log` for SQL query patterns. - ---- - -## Q10: Knowledge Pack format (`0017_knowledge_packs` migration) - -**Why we care:** dafit already authors `nimmerverse/nimmersky/oghma-sknpack/` knowledge packs (Whiterun, Winterhold, Solstheim, etc.). Understanding the runtime ingestion + storage format would let us validate that our packs are well-formed. - -**Verification path:** Read `0017_knowledge_packs.sql` to see the table schema. Inspect an oghma-sknpack file to see the source format. Trace ingestion in `SkyrimNet.log` - search for "knowledge_pack" or "sknpack" patterns. The `oghma-proxy` tool probably has documentation we wrote ourselves. - ---- - -## Q11: Does the chunker overlap suspicion for NPC↔NPC restatement still hold? - -**Why we care:** During Bug #2 diagnosis we hypothesized `tts.maxSegmentWords: 4` might amplify chunker race conditions in NPC-NPC scenes. We later determined the GM-loop bug fully explained restatement and the chunker was innocent. But the small chunk size remains as a stress test for any other latent chunker bugs. - -**Verification path:** Once Bug #2 is fixed, observe NPC-NPC scenes (Whiterun market with Ysolda↔Brenuin) with `maxSegmentWords` at 4 vs 24. If 4-word chunks cause clipping or audio gaps not seen at 24, the chunker has a chunk-size-dependent issue worth filing as Bug #4. - ---- - -## Q12: How does the in-game prompt editor's `propose_prompt_edit` tool flow back into the override file? - -**Why we care:** The runtime override at `overwrite/prompts/agent_chat.prompt.backup.1776372078` proves edits via the in-game LLM-driven prompt editor (`agent_prompt_helper.prompt`) get persisted with epoch-stamped backups. We don't know the exact write semantics - does it always backup-and-replace, or only on accept? - -**Verification path:** Use the in-game prompt editor on a low-stakes prompt, watch what files appear/change in `overwrite/prompts/`. Document the backup naming and write-on-accept behavior. - ---- - -## Conventions for adding new entries - -- Number monotonically (Q13 next). -- Each entry needs **why we care** + **verification path** at minimum. -- If a question gets answered, *move the answer into the appropriate detail file* and delete the entry here. Don't let resolved questions pile up. -- If a question gets *partially* answered, update it with the partial answer and what's still unknown. diff --git a/skyrimnet/prompt-templates.md b/skyrimnet/prompt-templates.md deleted file mode 100644 index 0ac0e4a..0000000 --- a/skyrimnet/prompt-templates.md +++ /dev/null @@ -1,116 +0,0 @@ -# Prompt Template System - -## Template language: Inja, NOT Jinja2 - -`[verified]` from `prompts/agent_prompt_helper.prompt:138-310`, which contains the canonical syntax reference embedded into the in-game prompt editor. - -Inja is a C++ Jinja2-inspired template engine. It looks similar but is **strictly more limited**. - -### Key differences from Jinja2 worth knowing - -- **Limited filter set.** Standard Jinja filters like `default`, `upper`, `length`, `join`, `sort` are present; less common ones may not be. -- **`{% set %}` only affects render context** - does NOT mutate input data. -- **Array manipulation functions return new arrays** - `append`, `extend`, etc. don't modify originals. -- **Template inheritance is limited:** `{% extends "..." %}` + `{% block name %}...{% endblock %}` works; multiple inheritance does not. -- **No macros** in the Jinja2 sense. -- **Backslashed paths in includes:** `render_template("components\\event_history_compact")` - the `\\` is the path separator in include calls. - -### SkyrimNet-specific Inja extensions - -**Section markers** split a single `.prompt` file into multiple chat messages: -``` -[ system ] -... system message content ... -[ end system ] - -[ user ] -... user message content ... -[ end user ] - -[ assistant ] -... assistant message content (rare) ... -[ end assistant ] -``` - -**Special blocks:** -- `[ raw ] ... [ end raw ]` - prevents Inja parsing of `{{...}}` etc. Used for embedded examples that contain template syntax literally. -- `[ cache ] ... [ end cache ]` - marks a block as cacheable for prompt caching across requests. - -**Include functions:** -- `render_template("path\\with\\backslashes")` - include another `.prompt` from `prompts/`. -- `render_subcomponent("subdir", render_mode)` - render numbered submodules from a subdir in load order. -- `render_character_profile("mode", uuid)` - render an NPC's bio with a render mode (`full`, `target`, `transform`, `thoughts`, `short_inline`, `interject_inline`). - -**Decorators** (DLL-registered functions callable from templates): -- `decnpc(uuid)` - universal NPC info accessor; returns object with `name`, `firstName`, `lastName`, `race`, `gender`, pronouns (`subjectivePronoun`, `objectivePronoun`, `possessivePronoun`, `reflexivePronoun`), `level`, `health`, `magicka`, `stamina`, `faction[]`, `isInCombat`, `isHostile`, `isDead`, `isBusy`, all skill values. -- 30+ other decorators observed in trace dump: `is_in_faction`, `is_player`, `get_arousal_state`, `format_event`, `short_time`, `get_name`, `get_nearby_npc_list`, `get_recent_events`, `get_quest_stage`, `is_narration_enabled`, `get_relevant_memories`, `get_scene_context`, `get_world_knowledge`, `papyrus_util`, `outfit_context`, etc. - -`[unknown]` Full decorator list with signatures - dynamically generated by the DLL via `prompts/documentation/main.prompt`. Would require dumping the in-game web UI's `/api/documentation` endpoint to enumerate. - ---- - -## The three-layer override architecture `[verified]` - -Load priority **high → low** (later layers shadow earlier ones for the same path): - -| # | Path | Purpose | Notes | -|---|---|---|---| -| 1 | `overwrite/SKSE/Plugins/SkyrimNet/prompts/` | **Runtime overrides.** Edits via the in-game prompt editor land here. | Selectively populated. Contains epoch-stamped backups (`agent_chat.prompt.backup.1776372078`). Also hosts runtime-generated content: `dynamic_character_bio.prompt` (~173KB regenerated per session), `_saves/`, `characters/` (per-actor profiles). | -| 2 | `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/prompts/` | **Shipped baseline + dev/user edits.** | Where our `gamemaster_action_selector.prompt` fix lives. The `.backup` from the fix is here too: `gamemaster_action_selector.prompt.backup`. | -| 3 | `mods/SkyrimNet/SKSE/Plugins/SkyrimNet/original_prompts/` | **Pristine upstream baseline.** Useful for diffing to detect upstream changes. | `[hypothesis]` Only `characters/` and `submodules/system_head/` are populated; the rest of upstream prompts may be assumed copied to `prompts/` at install time, or this directory only holds pristine *overrideable* defaults. | - -**Practical implication:** before editing any prompt, check if a runtime override exists in `overwrite/prompts/`. If it does, *that* is the active version - editing in `mods/prompts/` will be silently shadowed. - -`[verified]` overlay pattern is the same shape as Linux `/etc/foo.conf` + `/etc/foo.conf.d/` + package defaults. - ---- - -## Submodule numbering convention `[verified]` - -Submodules in `prompts/submodules//` are loaded in numerical filename order (e.g. `0010_*.prompt` before `0750_*.prompt`). The numbering carves out conventional ranges: - -| Range | Conventional purpose | Examples | -|---|---|---| -| `0010_…0099_` | Meta/header content (instructions, scene context bootstrap) | `0010_instructions.prompt`, `0250_omnisight.prompt` | -| `0100_…0499_` | Content sections (summary, background, personality, appearance, equipment) | character bio sections | -| `0500_…0799_` | Guidelines and behavioral rules | `0750_embedded_actions.prompt` (the ACTION: format spec) | -| `0800_…0999_` | Late instructions and special toggles | direct narration, recent state changes | -| `7000_…7999_` | Memories and progression | NPC memory blocks | -| `9990_` | Speech style - rendered last to override earlier voice instructions | speech style submodules | - -Earlier numbers establish baseline; higher numbers customize/override. Fits the late-binding-wins pattern. - ---- - -## Subdirectory map of `prompts/` `[verified]` - -| Dir | Purpose | Sample contents | -|---|---|---| -| `prompts/` (root) | Top-level entry-point prompts (one per agent type) | `dialogue_response.prompt`, `gamemaster_action_selector.prompt`, `native_action_selector.prompt`, `agent_chat.prompt`, `gamemaster_scene_planner.prompt`, `player_dialogue.prompt`, `player_thoughts.prompt` | -| `prompts/components/` | Reusable building blocks for `render_template("components\\X")` | `event_history.prompt`, `event_history_compact.prompt`, `event_history_verbose.prompt`, `agent_tools_base.prompt`, `memory_access.prompt`, character_bio variants | -| `prompts/components/context/` | Context-specific component variants | `scene_context.prompt`, `component_npc_state_summary.prompt` | -| `prompts/submodules/system_head/` | System-prompt prelude pieces | `0010_instructions.prompt`, `0250_omnisight.prompt`, … | -| `prompts/submodules/user_final_instructions/` | User-message tail pieces | `0750_embedded_actions.prompt` (ACTION: format), audio tag rules, recent state changes | -| `prompts/submodules/guidelines/` | Cross-cutting style rules | `0900_response_format.prompt` (asterisk narration rules, length limits) | -| `prompts/submodules/character_bio/` | Per-actor bio submodules merged into `dynamic_character_bio.prompt` | 17 numbered files: header, summary, personality, equipment, etc. | -| `prompts/submodules/omnisight_*/` | Vision-model description templates (one dir per target type: actor, scene, item, location, furniture, default) | matched 1:1 by `prompts/omnisight/describe_*.prompt` | -| `prompts/submodules/test_decorators/` | Test-data harness for the in-game template debugger | enabled by MCM toggle | -| `prompts/target_selectors/` | Meta classifier prompts for "who speaks next" | `dialogue_speaker_selector.prompt`, `player_dialogue_target_selector.prompt` | -| `prompts/transformers/` | Text→text transformations | `native_dialogue_transformer.prompt`, `universal_translator.prompt` | -| `prompts/memory/` | Memory generation, ranking, mood evaluation | `generate_memory.prompt`, `memory_ranker.prompt`, `mood_evaluator.prompt` | -| `prompts/helpers/` | Standalone classifier/profile generation prompts | `evaluate_mood.prompt`, `generate_profile.prompt`, `generate_search_query.prompt` | -| `prompts/omnisight/` | Vision-model prompts (one per target type) | `describe_actor.prompt`, `describe_scene.prompt`, `describe_item.prompt`, `describe_location.prompt`, `describe_furniture.prompt` | -| `prompts/dev/` | Developer test prompts | `mcm_test.prompt` | -| `prompts/documentation/` | Auto-generated decorator documentation rendered into the in-game web UI | `main.prompt`, `category.prompt` | -| `prompts/web/` | Web-UI bundled base templates | `bundled_base.prompt` | -| `prompts/translation/{generic,unique}/` | Localization CSVs | `00_SkyrimNet_generic.csv`, `00_SkyrimNet_unique.csv` | - ---- - -## Editing prompts safely - checklist - -1. **Is there an override in `overwrite/prompts/`?** If yes, that's the active version. Edit it (the in-game UI does this) or delete it to fall back to `mods/prompts/`. -2. **Diff against `original_prompts/`** to see what the upstream baseline says. -3. **Make a `.backup` next to the file** before significant edits - SkyrimNet itself does this convention with epoch-stamped backups; we use the simpler `.backup` suffix. -4. **Hot-reload picks up changes** - no need to restart Skyrim for prompt edits, but model behavior may take effect on the next agent firing. -5. **Test against logs** - watch `openrouter_input.log` to see what prompt text the model actually receives after your edit.