reorg. done llm-stack copied over and edited to ip endpoint.
This commit is contained in:
48
guides-stack/llm-stack-architecture.md
Normal file
48
guides-stack/llm-stack-architecture.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# SkyrimNet LLM Stack — LIVE
|
||||
|
||||
All four GPUs live. llama.cpp on dioscuri/comfy-dev. vLLM on theia.
|
||||
Chat endpoint: `/v1/chat/completions` with `{"chat_template_kwargs": {"enable_thinking": false}}`
|
||||
|
||||
## Instances
|
||||
|
||||
| Instance | Node | GPU | Port | Model | Quant | VRAM | Vision |
|
||||
|----------|------|-----|------|-------|-------|------|--------|
|
||||
| **nyx** | theia | Blackwell 6000 | 31000 | self-trained ~41B | BF16 | ~84GB | ✅ |
|
||||
| **alpha1** | dioscuri | Ada #1 | 31001 | Qwen3.6-27B-Fable-711 | IQ4_XS | 16.6GB | ❌ |
|
||||
| **alpha2** | dioscuri | Ada #2 | 31002 | Qwen3.6-27B-Fable-711 | IQ4_XS | 16.6GB | ❌ |
|
||||
| **gamma** | comfy-dev | RTX 3090 | 31003 | Qwen3.6-27B-Fable-711-MTP | Q4_K_S MTP | 19.9GB | ✅ |
|
||||
|
||||
## Role Mapping
|
||||
|
||||
- **nyx (31000):** default dialogue, diary, agent_helper
|
||||
- **alpha1 (31001):** meta, combat, action_evaluation
|
||||
- **alpha2 (31002):** gamemaster, overflow
|
||||
- **gamma (31003):** memory, character_profile, vision, universal_translator
|
||||
|
||||
## Endpoints for SkyrimNet
|
||||
|
||||
```yaml
|
||||
default: 10.0.40.21:31000/v1
|
||||
diary: 10.0.40.21:31000/v1
|
||||
agent_helper: 10.0.40.21:31000/v1
|
||||
meta: 10.0.40.22:31001/v1
|
||||
combat: 10.0.40.22:31001/v1
|
||||
action_evaluation: 10.0.40.22:31001/v1
|
||||
gamemaster: 10.0.40.22:31002/v1
|
||||
memory: 10.0.30.124:31003/v1
|
||||
character_profile: 10.0.30.124:31003/v1
|
||||
vision: 10.0.30.124:31003/v1
|
||||
universal_translator: 10.0.30.124:31003/v1
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Throughput: gamma ~20 tok/s, alpha1 ~10 tok/s, alpha2 ~14 tok/s
|
||||
- alpha1/alpha2 identical models (SHA256 confirmed)
|
||||
- gamma has MTP quant; alpha1/alpha2 have non-MTP
|
||||
- All llama.cpp services run as nyx-organs:nimmerverse-agents
|
||||
- OStimNet: Fable-Fusion needs explicit x-rated vocabulary in system prompts
|
||||
|
||||
## Service Files
|
||||
|
||||
`~/tmp/skyrimnet-analysis/services/` — llama-gamma.service, llama-alpha1.service, llama-alpha2.service
|
||||
Reference in New Issue
Block a user