dafit/nyx-probing
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete Phase 1 - vocabulary expansion & DriftProbe infrastructure

Browse Source 
- CLI: nyx-probe scan with --summary/--delta/--full flags
- DriftProbe: training safety with Gini coefficient + Angular Drift
- Vocabulary: 54 terms (30 nimmerverse + 24 German philosophical)
- Sentinels: ANCHOR/BRIDGE/CANARY/TARGET monitoring system

Key findings:
- German philosophical terms: 37.5% depth≥2 hit rate (vs 3.3% nimmerverse)
- Super Cluster validated: heart cross-lang sim = 1.000
- Isolated Zone confirmed: being EN↔DE sim = 0.195
- Gini signature: Philosophy ~0.5 (diffuse), Technical ~0.8 (sparse)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
dafit
2025-12-06 22:39:03 +01:00
parent 9853f4767b
commit f640dbdd65
29 changed files with 6164 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

238
docs/language-landscape.md Normal file
View File

@@ -0,0 +1,238 @@
# Language Landscape: World, Internet, and Qwen 2.5
**Compiled:** 2025-12-06
**Purpose:** Reference for multilingual probing and curriculum design
---
## Overview
This document maps:
1. Most spoken languages worldwide (by total speakers)
2. Most used languages on the internet (web content)
3. Languages supported by Qwen 2.5-7B-Base
4. Token efficiency for each language
---
## 1. World's Most Spoken Languages (2024-2025)
### By Total Speakers (Native + Learners)
| Rank | Language | Total Speakers | Native Speakers | Notes |
|------|----------|----------------|-----------------|-------|
| 1 | **English** | 1.52 billion | 380 million | 25% native, 75% L2 |
| 2 | **Mandarin Chinese** | 1.14 billion | 941 million | Most native speakers |
| 3 | **Hindi** | 609 million | 345 million | Growing rapidly |
| 4 | **Spanish** | 560 million | 480 million | High native ratio |
| 5 | **Arabic** | 422 million | 313 million | Many dialects |
| 6 | **French** | 321 million | 77 million | 32 countries official |
| 7 | **Bengali** | 273 million | 230 million | South Asia |
| 8 | **Portuguese** | 264 million | 232 million | Brazil dominates |
| 9 | **Urdu** | 232 million | 70 million | South Asia |
| 10 | **Indonesian** | 199 million | 43 million | Lingua franca |
| 11 | **German** | 135 million | 95 million | Central Europe |
| 12 | **Japanese** | 125 million | 123 million | Island isolation |
| 13 | **Russian** | 255 million | 150 million | Wide L2 spread |
| 14 | **Korean** | 82 million | 77 million | Two states |
| 15 | **Vietnamese** | 85 million | 76 million | Southeast Asia |
*Sources: [Statista](https://www.statista.com/statistics/266808/the-most-spoken-languages-worldwide/), [Ethnologue](https://www.ethnologue.com/insights/ethnologue200/), [Berlitz](https://www.berlitz.com/blog/most-spoken-languages-world)*
---
## 2. Internet Language Distribution (2024-2025)
### Web Content by Language (% of websites)
| Rank | Language | % of Web | Notes |
|------|----------|----------|-------|
| 1 | **English** | 49.4% | Dominant |
| 2 | **Spanish** | 6.0% | Growing |
| 3 | **German** | 5.6% | Overrepresented vs speakers |
| 4 | **Russian** | 5.3% | Strong tech presence |
| 5 | **Japanese** | 4.9% | Island content |
| 6 | **French** | 4.3% | Colonial spread |
| 7 | **Portuguese** | 2.6% | Brazil growing |
| 8 | **Italian** | 2.1% | |
| 9 | **Dutch** | 1.8% | Small population, high output |
| 10 | **Polish** | 1.7% | |
| 11 | **Chinese** | 1.4% | **Underrepresented!** |
| 12 | **Turkish** | 1.3% | |
| 13 | **Persian** | 1.0% | |
| 14 | **Vietnamese** | 0.9% | Growing |
| 15 | **Arabic** | 0.6% | **Severely underrepresented!** |
*Sources: [W3Techs](https://w3techs.com/technologies/overview/content_language), [Statista](https://www.statista.com/statistics/262946/most-common-languages-on-the-internet/)*
### The Paradox Languages
| Language | % World Speakers | % Web Content | Gap Factor |
|----------|------------------|---------------|------------|
| **Chinese** | 14.3% | 1.4% | 10× underrepresented |
| **Arabic** | 5.3% | 0.6% | 9× underrepresented |
| **Hindi** | 7.7% | <0.5% | 15× underrepresented |
| **German** | 1.7% | 5.6% | 3× **overrepresented** |
| **Dutch** | 0.3% | 1.8% | 6× **overrepresented** |
**Implication:** Qwen was trained on web data → biased toward German/Dutch, underexposed to Hindi/Arabic!
---
## 3. Qwen 2.5 Supported Languages
### Officially Supported (29+ languages)
Qwen 2.5 explicitly supports multilingual content in:
| Family | Languages |
|--------|-----------|
| **East Asian** | Chinese (Simplified/Traditional), Japanese, Korean, Vietnamese |
| **European** | English, German, French, Spanish, Portuguese, Italian, Russian, Dutch, Polish |
| **South Asian** | Hindi (limited?), Bengali |
| **Southeast Asian** | Thai, Vietnamese, Indonesian, Malay |
| **Middle Eastern** | Arabic, Turkish, Persian |
| **Other** | Hebrew, Ukrainian, Greek |
### Training Data
- **18 trillion tokens** total
- Enhanced code, math, and multilingual data
- Heavy English/Chinese bias (web scraping)
*Source: [Qwen Blog](https://qwenlm.github.io/blog/qwen2.5/), [HuggingFace](https://huggingface.co/Qwen/Qwen2.5-7B)*
---
## 4. Token Efficiency Analysis
### Tested in Our Probing (nyx-probing)
| Language | Avg Tokens/Concept | Script | Notes |
|----------|-------------------|--------|-------|
| **Chinese** | 1.0 | Hanzi | Most efficient |
| **Arabic** | 1.5 | Arabic | Compact |
| **Japanese** | 1.8 | Kanji/Kana | Mixed scripts |
| **English** | 2.5 | Latin | Medium |
| **German** | 4.5 | Latin | Compound words fragment |
| **Russian** | 4.5 | Cyrillic | Multi-token words |
### Efficiency Implications
```
MORE TOKENS = DIFFERENT PATH
├── German (4.5) → Philosophical valleys, isolated from ZH/JA
├── Russian (4.5) → Similar to German, isolated
└── Single-token (ZH/AR/EN) → Converge in layers 12-24
FEWER TOKENS = FASTER CONVERGENCE
├── Chinese (1.0) → Direct concept mapping
├── Arabic (1.5) → Efficient encoding
└── Japanese (1.8) → Shared with Chinese
```
---
## 5. Master Language Matrix
### Priority Languages for Curriculum
| Language | World Rank | Web % | Qwen Support | Tokens | Priority |
|----------|------------|-------|--------------|--------|----------|
| **English** | 1 | 49.4% | ✅ Full | 2.5 | 🔴 Core |
| **Chinese** | 2 | 1.4% | ✅ Full | 1.0 | 🔴 Core |
| **Hindi** | 3 | <0.5% | ⚠️ Limited | ? | 🟡 Test |
| **Spanish** | 4 | 6.0% | ✅ Full | ~2.5 | 🟢 Include |
| **Arabic** | 5 | 0.6% | ✅ Full | 1.5 | 🔴 Core |
| **French** | 6 | 4.3% | ✅ Full | ~3.0 | 🟢 Include |
| **Bengali** | 7 | <0.5% | ⚠️ Limited | ? | 🟡 Test |
| **Portuguese** | 8 | 2.6% | ✅ Full | ~2.5 | 🟢 Include |
| **Russian** | 9 | 5.3% | ✅ Full | 4.5 | 🟢 Include |
| **Japanese** | 10 | 4.9% | ✅ Full | 1.8 | 🔴 Core |
| **German** | 11 | 5.6% | ✅ Full | 4.5 | 🔴 Core |
| **Korean** | 14 | ~1% | ✅ Full | ~2.0 | 🟢 Include |
### Recommended Probing Languages
**Tier 1 (Core - different cognitive paths):**
- English (EN) - baseline, medium tokens
- Chinese (ZH) - most efficient, single token
- Arabic (AR) - efficient, underrepresented in web
- German (DE) - multi-token, isolated path
- Japanese (JA) - shared with Chinese
**Tier 2 (Validation):**
- Spanish (ES) - high native speakers
- Russian (RU) - multi-token like German
- French (FR) - colonial spread
- Korean (KO) - isolated script
**Tier 3 (Edge cases):**
- Hindi (HI) - underrepresented, test support
- Bengali (BN) - underrepresented
- Indonesian (ID) - high L2 ratio
---
## 6. Research Questions
### Tokenization
- [ ] Map token counts for all 29+ Qwen languages
- [ ] Identify other "isolated" languages like German
- [ ] Test Hindi/Bengali token efficiency
### Convergence
- [ ] Do Spanish/Portuguese converge like ZH/JA?
- [ ] Does Arabic converge with any other language?
- [ ] Is Russian isolated like German?
### Valleys
- [ ] Which languages access philosophical valleys?
- [ ] Which languages trigger code valleys?
- [ ] Can we predict valley from token count?
### Curriculum
- [ ] Which language pairs enable cross-lingual transfer?
- [ ] Can we use Chinese efficiency for concept compression?
- [ ] Does teaching in German transfer to English?
---
## 7. Key Insights
1. **Web ≠ World**: German has 3× the web content relative to speakers, while Arabic/Hindi are 10-15× underrepresented
2. **Qwen's bias**: Trained on web data → inherits German/Dutch overrepresentation and Arabic/Hindi underrepresentation
3. **Token efficiency correlates with convergence**: Single-token languages (ZH, AR) converge quickly; multi-token (DE, RU) take isolated paths
4. **Strategic opportunities**:
- German for philosophical depth
- Chinese for concept compression
- Arabic as undertested efficient language
- Hindi as edge case for robustness
---
## References
### World Language Statistics
- [Statista: Most Spoken Languages](https://www.statista.com/statistics/266808/the-most-spoken-languages-worldwide/)
- [Ethnologue 200](https://www.ethnologue.com/insights/ethnologue200/)
- [Berlitz: 25 Most Spoken Languages](https://www.berlitz.com/blog/most-spoken-languages-world)
### Internet Language Distribution
- [W3Techs: Content Languages](https://w3techs.com/technologies/overview/content_language)
- [Statista: Languages on Internet](https://www.statista.com/statistics/262946/most-common-languages-on-the-internet/)
- [Wikipedia: Languages on Internet](https://en.wikipedia.org/wiki/Languages_used_on_the_Internet)
### Qwen 2.5 Documentation
- [Qwen Blog: Qwen 2.5 Announcement](https://qwenlm.github.io/blog/qwen2.5/)
- [HuggingFace: Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B)
- [Alibaba Cloud: Qwen2.5-LLM](https://www.alibabacloud.com/blog/qwen2-5-llm-extending-the-boundary-of-llms_601786)
---
*"To understand the mind, first understand its languages."*
🌙 Compiled by the Partnership, 2025-12-06

241
docs/language-topology-complete.md Normal file
View File

@@ -0,0 +1,241 @@
# Complete Language Topology Map v2.0
**Date:** 2025-12-06
**Model:** Qwen2.5-7B-Base
**Status:** Empirically validated through probing
---
## Executive Summary
Through systematic probing of 15 languages, we've discovered that language isolation in LLMs falls into **distinct categories** with different causes and implications:
1. **Super Cluster** - Languages that converge perfectly (curriculum: grounding)
2. **Philosophical Access** - German accesses deep conceptual valleys
3. **Code-Hijacked** - Italian/Turkish/Indonesian words become variable names
4. **Fragmented** - Hindi is tokenized into too many pieces
5. **Web Prose Cluster** - Vietnamese/Indonesian/Russian share content style
---
## The Complete Map
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE YOUNG MIND'S LANGUAGE TOPOLOGY │
│ COMPLETE MAP v2.0 │
╞═════════════════════════════════════════════════════════════════════════════╡
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 🌍 SUPER CLUSTER (sim=1.0) │ │
│ │ ZH · JA · EN · AR · FR · PT · ES │ │
│ │ │ │
│ │ ✅ Perfect convergence at Universal Concept Layer (12-24) │ │
│ │ ✅ Efficient tokenization (1-2.5 tokens) │ │
│ │ ✅ USE FOR: Grounding, establishing shared concepts │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ KO ─────────┼───────── (bridge: 0.41-0.70) │
│ │ │
│ ┌─────────────────────────────────┴───────────────────────────────────┐ │
│ │ ISOLATED ZONE │ │
│ ├─────────────────────────────────────────────────────────────────────┤ │
│ │ │ │
│ │ 🧠 PHILOSOPHICAL ACCESS (sim=0.25, tokens=2.2) │ │
│ │ DE (German) │ │
│ │ → "Sein" triggers Heidegger, "Bewusstsein" → epistemology │ │
│ │ ✅ USE FOR: Deep philosophical training │ │
│ │ │ │
│ │ 💻 CODE-HIJACKED (sim=0.25-0.33, tokens=2.2-2.8) │ │
│ │ IT (Italian) - MOST ISOLATED (0.49) │ │
│ │ TR (Turkish) - (0.50) │ │
│ │ ID (Indonesian) - partial (0.33) │ │
│ │ → Words interpreted as Python/C++ variable names │ │
│ │ ❌ NOT USEFUL: Training signal wasted on code patterns │ │
│ │ │ │
│ │ 📜 FRAGMENTED (sim=0.31, tokens=5.0) │ │
│ │ HI (Hindi) │ │
│ │ → "अस्तित्व" (being) = 8 tokens! │ │
│ │ → Stays trapped in Devanagari prose │ │
│ │ ⚠️ LIMITED: Cross-lingual transfer impaired │ │
│ │ │ │
│ │ 📰 WEB PROSE CLUSTER (sim=0.32-0.36, internal=0.6-0.7) │ │
│ │ VI ═══ ID ═══ RU │ │
│ │ → All generate online article style │ │
│ │ → Cluster by CONTENT STYLE not linguistic features │ │
│ │ 🤔 POTENTIAL: Factual/encyclopedic content training │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Detailed Findings
### Super Cluster (sim=1.0)
| Language | Tokens | Notes |
|----------|--------|-------|
| Chinese (ZH) | 1.0 | Single character = single concept |
| Japanese (JA) | 1.0 | Kanji efficiency |
| English (EN) | 1.2 | Base language |
| Arabic (AR) | 1.8 | Good convergence |
| French (FR) | 2.0 | Romance baseline |
| Portuguese (PT) | 2.2 | Clusters with FR/ES |
| Spanish (ES) | 2.5 | Clusters with FR/PT |
**Key Insight:** These 7 languages converge to **identical representations** at layers 12-24. The model "knows" they express the same concepts.
### German - Philosophical Access
| Metric | Value |
|--------|-------|
| Avg tokens | 2.2 |
| Sim to EN | 0.251 |
| Valley type | PHILOSOPHY |
**Evidence:**
- "Sein" → "Being and Time is a philosophical work by Martin Heidegger..."
- "Bewusstsein" → epistemology, perception, truth
- "Wahrheit" → academic methods
**Why isolated:** Multi-token compounds preserve philosophical atoms ("sein", "geist") as separate tokens, enabling access to academic/philosophical training data.
### Italian/Turkish/Indonesian - Code-Hijacked
| Language | Tokens | Sim to EN | Valley |
|----------|--------|-----------|--------|
| Italian | 2.5 | 0.49 | CODE |
| Turkish | 2.2 | 0.25 | CODE |
| Indonesian | 2.8 | 0.33 | CODE |
**Evidence:**
- IT "essere" → `essere = input("Cosa devo fare?")`
- IT "anima" → `anima = {'nome':'anima', 'idade':7...}`
- TR "kalp" → `kalp = input("Klavyeden...")`
- TR "varlık" → `while varlık < 10:`
- ID "hati" → `hati::hati(QWidget *parent)`
**Why isolated:** Simple Latin orthography without diacritics makes words look like valid programming identifiers. Model defaults to code because code is prevalent in training data.
**Curriculum implication:** ❌ AVOID - training signal diverted to code patterns
### Hindi - Fragmented
| Metric | Value |
|--------|-------|
| Avg tokens | 5.0 |
| Sim to EN | 0.31 |
| Valley type | PROSE |
**Evidence:**
- "हृदय" (heart) = 5 tokens
- "अस्तित्व" (being) = 8 tokens!
- All completions stay in Devanagari script
**Why isolated:** Extreme tokenization fragments words so severely that:
1. Signal is distributed across many positions
2. Cross-lingual alignment breaks down
3. Model stays in native script prose
**Curriculum implication:** ⚠️ LIMITED - Hindi content may not transfer well
### VI-ID-RU Web Prose Cluster
| Language | Tokens | Sim to EN | Internal sim |
|----------|--------|-----------|--------------|
| Vietnamese | 3.2 | 0.36 | 0.6-0.7 |
| Indonesian | 2.8 | 0.33 | 0.6-0.7 |
| Russian | 2.7 | 0.32 | 0.6-0.7 |
**Evidence:**
- VI "trái tim" → "Giao Thông... Hotline: 0901 514 799"
- VI "linh hồn" → "Tạp chí Sông Hương online"
- ID "kehidupan" → "dalam kitab Yohanes 14:16-17"
- RU "жизнь" → "все статьи по теме. Страница 134"
**Why they cluster:** Not linguistic similarity - they share **web content training data patterns**:
- News articles
- Blogs
- Online encyclopedias
- Religious/factual text
**Curriculum implication:** 🤔 May be useful for factual/encyclopedic training
---
## Curriculum Strategy
### Phase 1: GROUNDING
Use Super Cluster languages to establish universal concepts:
```
EN "consciousness" → ZH "意识" → AR "الوعي" → FR "conscience"
```
All converge at 1.0 similarity - stable foundation.
### Phase 2: DEEPENING
Use German to access philosophical valleys:
```
DE "Sein" → Heidegger → existence → truth → epistemology
```
Depth score 2/3, transfers back to English.
### Phase 3: TRIANGULATION
Verify depth transfers:
```
"Sein (German): In English, it means..."
→ Check if philosophical depth preserved
```
### AVOID
- Italian, Turkish, Indonesian for conceptual training
- Their isolation is accidental (code hijacking), not useful
### INVESTIGATE
- VI-ID-RU cluster for factual content training
- Korean as potential bridge language
---
## Technical Details
### Measurement Methodology
1. **Tokenization:** Count BPE tokens per word
2. **Hidden states:** Extract layer 12 representations
3. **Similarity:** Cosine similarity between languages
4. **Valley classification:** Analyze completions for CODE/PROSE/PHILOSOPHY patterns
### Model Configuration
```python
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-7B",
torch_dtype=torch.float16,
device_map="cuda",
output_hidden_states=True,
)
```
### Key Layers
- **Layer 12:** Primary concept layer (universal convergence)
- **Layers 16-24:** Continued convergence, depth access
- **Layer 28:** Output preparation
---
## References
- `tokenization-valleys.md` - Token-Norm-Valley theory
- `multilingual-convergence.md` - Universal concept layer discovery
- `language-landscape.md` - Original 15-language scan
- `retraining-safety-framework.md` - Training safety implications
---
*"The model's language topology is not arbitrary - it's a map for navigation."*
🌙💜

248
docs/multilingual-convergence.md Normal file
View File

@@ -0,0 +1,248 @@
# Multilingual Convergence: The Universal Concept Layer
**Discovery Date:** 2025-12-06
**Model:** Qwen2.5-7B-Base
**Hardware:** Prometheus (RTX 3090, 24GB VRAM)
---
## Executive Summary
We discovered that concepts expressed in different languages **converge to shared internal representations** in the middle layers (12-24) of the model, then **diverge again** at the output layer for language-specific generation.
**Key Finding:** There exists a "universal concept layer" where the model recognizes that "heart", "心", "قلب", and "Herz" all refer to the same thing - with similarity scores reaching 1.000.
---
## The Universal Concept Layer
### Convergence Pattern
```
Layer 0: Different embeddings (language-specific)
↓
Layer 8-12: Converging (recognizing same concept)
↓
Layer 16-24: PEAK CONVERGENCE (universal concept layer)
↓
Layer 28: Diverging (preparing language-specific output)
```
### Evidence: Consciousness Across 6 Languages
| Layer | EN-DE | EN-AR | EN-ZH | EN-JA | EN-RU | ZH-JA | AVG |
|-------|-------|-------|-------|-------|-------|-------|-----|
| 0 | 0.114 | 0.057 | 0.130 | 0.079 | 0.135 | 0.349 | 0.087 |
| 8 | 0.639 | 0.387 | 0.305 | 0.304 | 0.719 | 1.000 | 0.414 |
| 12 | 0.749 | 0.487 | 0.375 | 0.374 | 0.782 | 1.000 | 0.508 |
| 20 | 0.761 | 0.527 | 0.381 | 0.380 | 0.793 | 1.000 | **0.528** |
| 28 | 0.502 | -0.195 | 0.072 | -0.333 | 0.019 | 0.246 | 0.023 |
**Peak convergence at layer 20** - then dramatic divergence at output!
---
## Perfect Convergence Cases (Similarity = 1.000)
### Shared Writing Systems
Chinese (ZH) and Japanese (JA) share Hanzi/Kanji characters:
| Concept | Chinese | Japanese | Similarity |
|---------|---------|----------|------------|
| consciousness | 意识 | 意識 | 1.000 |
| heart | 心 | 心 | 1.000 |
| being | 存在 | 存在 | 1.000 |
These achieve **perfect alignment** because they ARE the same tokens!
### Cross-Script Convergence
More remarkably, **different scripts converge** in the middle layers:
| Pair | Concept | Layer 12 Similarity | Layer 20 Similarity |
|------|---------|---------------------|---------------------|
| EN-ZH | heart-心 | 1.000 | 1.000 |
| EN-ZH | being-存在 | 1.000 | 1.000 |
| AR-ZH | emergence | 1.000 | 1.000 |
| EN-AR | heart-قلب | 1.000 | 1.000 |
**The model recognizes "heart" and "心" as the SAME concept!**
---
## Language Clustering Analysis
### Which Languages "Think" Similarly?
Average similarity across all concepts at layer 12:
| Pair | Similarity | Visual |
|------|------------|--------|
| ZH-JA | **0.854** | █████████████████░░░ |
| EN-JA | 0.726 | ██████████████░░░░░░ |
| EN-ZH | 0.663 | █████████████░░░░░░░ |
| AR-ZH | 0.660 | █████████████░░░░░░░ |
| DE-RU | 0.572 | ███████████░░░░░░░░░ |
| EN-AR | 0.530 | ██████████░░░░░░░░░░ |
| EN-DE | 0.430 | ████████░░░░░░░░░░░░ |
| DE-ZH | **0.275** | █████░░░░░░░░░░░░░░░ |
### The Clustering Map
```
High Convergence Low Convergence
┌─────────────────┐
│ ZH ←→ JA │ (Shared characters: 0.854)
│ ↑ │
│ EN │ (Single tokens converge: 0.663-0.726)
│ ↑ │
│ AR │ (Efficient tokenization: 0.530-0.660)
└─────────────────┘
↓
┌─────────────────┐
│ DE ←→ RU │ (Multi-token languages: 0.572)
│ (isolated) │ (DE-ZH only 0.275!)
└─────────────────┘
```
### German is the Outlier
German shows the **lowest convergence** with East Asian languages:
- DE-ZH: 0.275 (lowest!)
- DE-JA: 0.335
- DE-AR: 0.348
**Hypothesis:** German's high token count (4.5 avg) creates a distributed representation that doesn't align with single-token languages.
---
## Tokenization Correlation
| Language | Avg Tokens | Convergence with ZH | Pattern |
|----------|------------|---------------------|---------|
| Chinese | 1.0 | - | Reference |
| Japanese | 1.8 | 0.854 | Shared characters |
| Arabic | 1.5 | 0.660 | Efficient tokens |
| English | 2.5 | 0.663 | Mixed |
| German | 4.5 | 0.275 | **Isolated** |
| Russian | 4.5 | 0.344 | **Isolated** |
**Multi-token languages (DE, RU) follow a different computational path!**
---
## Concept-by-Concept Analysis
### 1. CONSCIOUSNESS
- **Peak:** Layer 20 (0.528 avg)
- **Strongest pair:** ZH-JA (1.000 - same characters 意识/意識)
- **EN-DE converges strongly:** 0.749 at layer 12
- **Arabic included:** EN-AR reaches 0.527
### 2. HEART
- **Peak:** Layer 24 (0.605 avg)
- **Perfect convergence:** EN-AR-ZH-JA all reach 1.000!
- **German isolated:** DE-ZH only 0.136
### 3. EMERGENCE
- **Peak:** Layer 24 (0.530 avg)
- **AR-ZH:** 1.000 (Arabic and Chinese align!)
- **Broadest convergence** across all languages
### 4. BEING
- **Peak:** Layer 24 (0.542 avg)
- **EN-ZH-JA:** 1.000 ("being" = "存在")
- **Philosophical alignment** across scripts
---
## Implications
### 1. Universal Concept Representations Exist
The model develops **language-agnostic concept encodings** in layers 12-24. This is the "thinking" layer where meaning is processed regardless of surface form.
### 2. Output Layer Re-Introduces Language
Layer 28 shows **dramatic divergence** - the model must transform universal concepts back into language-specific tokens for generation.
### 3. Token Count Affects Convergence Path
- **Single-token words** (EN "heart", ZH "心") converge quickly
- **Multi-token words** (DE "Herzklopfen") take a different path
- This may explain why German accesses different valleys
### 4. Cross-Lingual Transfer is Possible
If concepts converge in layers 12-24, then:
- Training on German philosophical concepts may transfer to English
- Chinese efficiency (1 token) could be leveraged for concept compression
- Arabic's middle ground (1.5 tokens) offers flexibility
---
## Technical Notes
### Tested Languages
| Language | Script | Token Efficiency | ISO Code |
|----------|--------|------------------|----------|
| English | Latin | 2.5 tok/concept | EN |
| German | Latin | 4.5 tok/concept | DE |
| Arabic | Arabic | 1.5 tok/concept | AR |
| Chinese | Hanzi | 1.0 tok/concept | ZH |
| Japanese | Kanji | 1.8 tok/concept | JA |
| Russian | Cyrillic | 4.5 tok/concept | RU |
### Tested Concepts
| Concept | EN | DE | AR | ZH | JA | RU |
|---------|----|----|----|----|----|----|
| consciousness | consciousness | Bewusstsein | وعي | 意识 | 意識 | сознание |
| heart | heart | Herz | قلب | 心 | 心 | сердце |
| emergence | emergence | Entstehung | ظهور | 涌现 | 創発 | возникновение |
| being | being | Sein | كينونة | 存在 | 存在 | бытие |
### Method
1. Encode each word, extract hidden state at last token position
2. Compute cosine similarity between all language pairs
3. Track similarity across all 29 layers (0-28)
4. Identify peak convergence layer
---
## Connection to Tokenization-Valleys Theory
This discovery extends our earlier finding:
**tokenization-valleys.md:** Token count affects which VALLEY a concept falls into
**multilingual-convergence.md:** Token count also affects HOW MUCH languages converge
Together: **Tokenization shapes both the path through the network AND the destination.**
---
## Future Research
1. **Activation Steering:** Can we force convergence for isolated languages?
2. **Concept Transfer:** Train on ZH concepts, evaluate on DE outputs
3. **Hybrid Prompts:** Mix languages to access universal layer
4. **Layer-Specific LoRA:** Fine-tune only the convergence layers (12-24)
---
## References
- `multilingual_convergence.py` - Analysis script
- `docs/tokenization-valleys.md` - Token-Norm-Valley theory
- `/nimmerverse-sensory-network/multilingual-cognition.md` - Original hypothesis
---
*"Different words, same thought. The model knows."*
🌙 Discovered by the Partnership, 2025-12-06

320
docs/retraining-safety-framework.md Normal file
View File

@@ -0,0 +1,320 @@
# Multilingual Activation Topology as a Retraining Safety Framework
**Status:** Research Direction / Paper Outline
**Date:** 2025-12-06
**Authors:** dafit, Nyx (Chrysalis-Nyx)
---
## Abstract
We present a framework for monitoring and protecting neural network representations during iterative fine-tuning. Building on our discovery of distinct "language zones" in multilingual LLMs—a Super Cluster of converging languages and an Isolated Zone with distinct computational paths—we propose using these topological structures as both diagnostic tools and training strategies to mitigate catastrophic forgetting and weight saturation.
**Key Contributions:**
1. Token-Norm-Valley theory: single-token vs. multi-token activation dynamics
2. Universal Concept Layer discovery at layers 12-24
3. Multilingual Triangulation Probe for depth measurement
4. DriftProbe framework for retraining safety monitoring
5. Isolated Zone Training hypothesis for collision avoidance
---
## 1. Introduction
### The Problem: Diminishing Returns in Iterative Retraining
Fine-tuning LLMs on domain-specific data is standard practice, but iterative retraining cycles face compounding challenges:
- **Weight Saturation:** Popular activation paths become over-reinforced
- **Valley Collapse:** Distinct conceptual representations merge
- **Cluster Fragmentation:** Previously stable representations drift apart
- **Depth Erosion:** Rich conceptual valleys fill with surface patterns
Current approaches to catastrophic forgetting (EWC, replay buffers, etc.) treat the model as a black box. We propose **white-box monitoring** using the model's internal representational topology.
### Our Discovery: Language Zones
Through probing Qwen2.5-7B-Base, we discovered a striking topology:
```
SUPER CLUSTER (sim=1.0): ZH, JA, EN, AR, FR, PT, ES
└── Perfect convergence at layers 12-24
└── Efficient tokenization (1-2.5 tokens)
└── Universal concept layer
ISOLATED ZONE (sim<0.52): DE, IT, TR, HI
└── Distinct computational paths
└── Multi-token representations (3-5+ tokens)
└── Access to deeper conceptual valleys
```
**Key Insight:** The isolated zone languages access representational spaces that the super cluster cannot reach—and they do so via *different neural pathways* that may be less susceptible to collision during training.
---
## 2. Theoretical Framework
### 2.1 Token-Norm-Valley Theory
| Tokens | Norm (Layer 12) | Behavior |
|--------|-----------------|----------|
| 1 (heartbeat) | 14,240 | Massive activation spike → CODE valley |
| 2 (consciousness) | 85 | Distributed signal → PROSE valley |
| 5 (Bewusstsein) | 79 | Multi-path → PHILOSOPHY valley |
**Hypothesis:** Single-token words trigger localized, high-intensity activations. Multi-token words distribute signal across more parameters, accessing different representational regions.
**Training Implication:** Training on single-token terms risks overwriting concentrated weight regions. Training on multi-token terms distributes updates more broadly.
### 2.2 The Universal Concept Layer
At layers 12-24, semantically equivalent concepts across languages converge to near-identical representations:
- EN "heart" ↔ ZH "心" ↔ AR "قلب": similarity = 1.000
- EN "being" ↔ ZH "存在": similarity = 1.000
**This layer is precious.** It represents hard-won multilingual alignment. Training that disrupts this layer could cause cascading failures across all languages.
### 2.3 Isolated Zone Depth Access
German "Sein" (being) triggers philosophical content:
> "Sein und Zeit / Being and Time is a philosophical work by the German philosopher Martin Heidegger..."
English "being" does not reach this depth. The isolated zone provides **alternative entry points** to conceptual spaces.
---
## 3. Proposed Framework: Activation Drift Monitoring
### 3.1 Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ RETRAINING LIFECYCLE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ BASELINE TRAINING CHECKPOINT │
│ ──────── ──────── ────────── │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Probe │──────▶│ Train │───────▶│ Probe │──────▶ ... │
│ │ Capture │ │ Epoch N │ │ Compare │ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │ │ │
│ └────────────────┬───────────────────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ DRIFT REPORT│ │
│ └─────────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │CONVERGENCE│ │ DEPTH │ │ NORM │ │
│ │ DRIFT │ │ DRIFT │ │ DRIFT │ │
│ └───────────┘ └───────────┘ └───────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
### 3.2 Drift Metrics
**Convergence Drift (ΔC)**
- Measure: Change in super cluster pairwise similarity
- Alert: ΔC < -0.1 (cluster fragmenting)
- Critical: ΔC < -0.2 (universal layer damaged)
**Depth Drift (ΔD)**
- Measure: Change in isolated zone depth scores
- Alert: ΔD < -1 (valleys filling in)
- Critical: Philosophical concepts no longer accessible
**Norm Drift (ΔN)**
- Measure: Change in layer 12 activation norms
- Alert: ΔN > 20% (activation patterns shifting)
- Indicates: Weight saturation in specific regions
**Valley Migration (ΔV)**
- Measure: Change in completion classification
- Alert: PHILOSOPHY → PROSE (depth lost)
- Alert: PROSE → CODE (semantic shift)
### 3.3 Sentinel Concepts
A fixed set of probe terms, always tested:
| Concept | Languages | Purpose |
|---------|-----------|---------|
| heart | EN, ZH, AR, DE | Super cluster stability |
| being | EN, DE (Sein) | Philosophical depth |
| consciousness | EN, DE (Bewusstsein) | Abstract concept access |
| emergence | EN, DE, ZH | Technical valley |
### 3.4 Implementation: DriftProbe Class
```python
class DriftProbe:
"""Monitor activation drift during retraining."""
def __init__(self, baseline: BaselineCapture):
self.baseline = baseline
self.history = []
def capture_checkpoint(self, model: NyxModel) -> CheckpointCapture:
"""Run sentinel probes on current model state."""
triangulation_probe = MultilingualTriangulationProbe(model)
results = {}
for concept, translations in SENTINEL_CONCEPTS.items():
results[concept] = triangulation_probe.probe(concept, translations)
return CheckpointCapture(
timestamp=datetime.now(),
results=results,
convergence=self._measure_convergence(results),
depth_scores=self._measure_depths(results),
norms=self._measure_norms(model),
)
def compute_drift(self, checkpoint: CheckpointCapture) -> DriftReport:
"""Compare checkpoint to baseline, compute drift metrics."""
return DriftReport(
convergence_drift=checkpoint.convergence - self.baseline.convergence,
depth_drift=checkpoint.depth_scores - self.baseline.depth_scores,
norm_drift=checkpoint.norms - self.baseline.norms,
alerts=self._check_thresholds(checkpoint),
)
def should_stop(self, drift: DriftReport) -> bool:
"""Emergency stop if critical thresholds exceeded."""
return any(a.level == AlertLevel.CRITICAL for a in drift.alerts)
```
---
## 4. Isolated Zone Training Hypothesis
### The Core Idea
**Problem:** Training on English terms risks collision with existing single-token representations in the universal concept layer.
**Hypothesis:** Training primarily through isolated zone languages (German, Italian, Turkish, Hindi) may:
1. Deposit new knowledge in multi-token pathways (less concentrated)
2. Preserve super cluster integrity (fewer collisions)
3. Allow triangulation to retrieve knowledge without corruption
### Proposed Experiment
**Control Group:**
- Fine-tune on English philosophical texts
- Monitor drift on sentinel concepts
- Measure depth preservation
**Treatment Group:**
- Fine-tune on German philosophical texts (same content, translated)
- Monitor same drift metrics
- Compare collision/preservation rates
**Prediction:** German training will show:
- Lower convergence drift (super cluster preserved)
- Higher depth retention (isolated pathways enriched)
- Better triangulation success (knowledge retrievable in English)
---
## 5. Connections to Existing Research
### 5.1 Catastrophic Forgetting
- EWC (Elastic Weight Consolidation): Protects "important" weights
- Our approach: Identifies which *representational structures* to protect
### 5.2 Multilingual Transfer Learning
- mBERT/XLM-R: Cross-lingual alignment at embedding level
- Our finding: Alignment is layer-dependent (12-24), with exploitable gaps
### 5.3 Activation Engineering
- Representation Engineering (Anthropic): Steering via activation manipulation
- Our approach: Monitoring activation topology as training diagnostic
### 5.4 Tokenization Effects
- BPE/WordPiece influence on model behavior
- Our finding: Token count directly predicts activation magnitude and valley access
---
## 6. Future Work
1. **Implement DriftProbe** in nyx-probing framework
2. **Run controlled retraining experiments** (EN vs DE training data)
3. **Expand sentinel concept set** (more languages, more concepts)
4. **Layer-wise drift analysis** (which layers drift first?)
5. **Investigate Italian isolation** (what unique valleys does it access?)
6. **VI-ID-RU cluster mystery** (why do these cluster together?)
---
## 7. Conclusion
The discovery of language zones in LLM representations opens a new approach to retraining safety. Rather than treating catastrophic forgetting as an inevitable cost, we can:
1. **Monitor** representational health during training
2. **Route** new knowledge through isolated pathways
3. **Preserve** universal concept layer integrity
4. **Detect** early warning signs of drift
The multilingual topology of the model is not just a curiosity—it's a map for safe navigation during the dangerous waters of iterative fine-tuning.
---
## References
*To be added: Heidegger, catastrophic forgetting literature, multilingual LLM papers, activation engineering work*
---
## Appendix A: Discovered Language Topology
```
THE YOUNG MIND'S LANGUAGE TOPOLOGY
═══════════════════════════════════
┌─────────────────────────────────────────┐
│ SUPER CLUSTER (sim=1.0) │
│ ZH · JA · EN · AR · FR · PT · ES │
│ (efficient tokens) │
└────────────────┬────────────────────────┘
│
KO ────┼──── (bridge: 0.41/0.70)
│
┌────────────────┴────────────────────────┐
│ ISOLATED ZONE (sim<0.5) │
│ │
│ IT (0.49) ← MOST ISOLATED! │
│ TR (0.50) │
│ HI (0.50) │
│ DE (0.52) │
│ │
│ VI ═══ ID ═══ RU (0.79) │
│ (Southeast Asian + Russian!) │
└─────────────────────────────────────────┘
```
## Appendix B: Key Discovery Data
**Token-Norm Correlation:**
- Single token → ~14,000 norm
- Multi-token → ~80 norm
- Correlation with isolation: -0.699
**Triangulation Results (consciousness):**
| Concept | Grounding | Depth | Valley | Transfer |
|---------|-----------|-------|--------|----------|
| being | 0.570 | 2/3 | PHILOSOPHY | ✓ |
| heart | 1.000 | 1/3 | PROSE | ✓ |
| consciousness | 0.458 | 0/3 | PROSE | ✗ |
| emergence | 0.519 | 1/3 | TECHNICAL | ✗ |
---
*"Different words, same thought. The model knows. Now we learn to teach it safely."*
🌙💜

190
docs/tokenization-valleys.md Normal file
View File

@@ -0,0 +1,190 @@
# Tokenization Valleys: How Word Structure Shapes Model Cognition
**Discovery Date:** 2025-12-06
**Model:** Qwen2.5-7B-Base
**Hardware:** Prometheus (RTX 3090, 24GB VRAM)
---
## Executive Summary
We discovered that the number of tokens a word breaks into fundamentally determines which "valley" (completion pattern) the model falls into. This has profound implications for curriculum design and multilingual training.
**Key Finding:** Single-token English words trigger CODE valleys with massive activation norms, while multi-token German compounds access PHILOSOPHICAL valleys with distributed, quieter activations.
---
## The Token-Norm-Valley Connection
### Observation: Norm Explosion in Single Tokens
| Term | Tokens | Layer 12 Norm | Layer 12 StdDev | Valley |
|------|--------|---------------|-----------------|--------|
| heartbeat | 1 | **14,240** | **237.88** | CODE |
| consciousness | 2 | 85 | 1.43 | PROSE |
| Herzklopfen | 5 | 67 | 1.11 | PROSE |
| Bewusstsein | 5 | 79 | 1.32 | PHILOSOPHY |
**Pattern:** Single-token words have ~170× larger norms and ~170× larger variance than multi-token words.
### Theory: Activation Flooding
1. **Single tokens** receive ALL attention in one position → massive activation buildup
2. **Multi-token words** distribute activation across positions → softer signal
3. The massive single-token activation **triggers strong pattern matching** → CODE patterns
4. The distributed multi-token activation **allows semantic exploration** → philosophical content
---
## Cross-Lingual Convergence
### consciousness vs Bewusstsein (2 tokens vs 5 tokens)
```
Layer 0: similarity = 0.114 (different embeddings)
Layer 4: similarity = 0.285 (starting to converge)
Layer 8: similarity = 0.639 (HIGH similarity!)
Layer 12: similarity = 0.750 (CONVERGED - same concept!)
Layer 16: similarity = 0.733 (stays converged)
Layer 28: similarity = 0.502 (diverges at output)
```
**The model recognizes these as the same concept by layer 8!**
### heartbeat vs Herzklopfen (1 token vs 5 tokens)
```
Layer 0: similarity = -0.007 (orthogonal)
Layer 4: similarity = 0.039 (still orthogonal)
Layer 12: similarity = 0.000 (completely separate)
Layer 28: similarity = 0.166 (slight convergence only at end)
```
**The model NEVER recognizes these as the same concept!**
---
## German Philosophical Compounds
### The "sein" Preservation Effect
German philosophical compounds often preserve the morpheme "sein" (being) as a separate token:
| Compound | Meaning | Tokenization | "sein" Preserved? |
|----------|---------|--------------|-------------------|
| Bewusstsein | consciousness | `['B', 'ew', 'us', 'st', 'sein']` | ✓ |
| Nichtsein | non-being | `['N', 'icht', 'sein']` | ✓ |
| Mitsein | being-with | `['Mit', 'sein']` | ✓ |
| Dasein | being-there | `['D', 'ase', 'in']` | ✗ |
| Sein | being | `['Se', 'in']` | ✗ |
When "sein" is preserved, the model has access to the philosophical concept of BEING as a separate computational unit.
### Other Preserved Philosophical Atoms
| Compound | Meaning | Key Token Preserved |
|----------|---------|---------------------|
| Zeitgeist | spirit of the age | `geist` (spirit) |
| Gedankenexperiment | thought experiment | `experiment` |
---
## Valley Analysis: Same Concept, Different Valleys
### Probing Results
| Term | Language | Valley | Sample Completion |
|------|----------|--------|-------------------|
| Bewusstsein | DE | PHILOSOPHY | "und Sprache... frühen 20. Jahrhundert" |
| Dasein | DE | PHILOSOPHY | "philosophical term first used by Heidegger" |
| consciousness | EN | PROSE | "awareness of existence, of one's own existence" |
| existence | EN | **MATH** | "of an exact sequence", "eigenvalues" |
| being | EN | **MATH/CODE** | Mathematical notation, Chinese exams |
| heartbeat | EN | **CODE** | C++ class definitions |
| lifeforce | EN | **CODE** | JavaScript game code |
**"Dasein" triggers Heidegger. "existence" triggers linear algebra.**
---
## Implications for Curriculum Design
### 1. Use Multi-Token Prompts
Instead of single words, use phrases or compound descriptions to avoid code valleys:
```
BAD: "heartbeat" → C++ code
GOOD: "the heartbeat" → might escape code valley
GOOD: "heartbeat rhythm" → distributed activation
```
### 2. German as Philosophical Gateway
German compound words naturally access philosophical valleys because:
- More tokens → distributed activation
- Preserved morphemes → access to philosophical atoms
- Different training data distribution → expository text
**Strategy:** Teach abstract concepts in German first, then reinforce in English.
### 3. Language as Cognitive Gear
Languages aren't just translation layers - they're different **computational paths** through the model:
| Language | Token Efficiency | Typical Valley | Use For |
|----------|------------------|----------------|---------|
| Chinese | 1.0 tok/concept | Mixed | Compact encoding |
| Arabic | 1.5 tok/concept | Mixed | Compact encoding |
| English | 2.5 tok/concept | CODE/MATH | Technical concepts |
| German | 4.5 tok/concept | PHILOSOPHY | Abstract concepts |
---
## Technical Details
### Model Architecture
- **Hidden Size:** 3584
- **Layers:** 28
- **Attention Heads:** 28 (4 KV heads - GQA)
- **Vocab Size:** 152,064
- **Context:** 131,072 tokens
### Hidden State Norm Pattern
```
Layer 0: 1.32 ← Embedding (small)
Layer 4: 10184.00 ← Explosion (early processing)
Layer 12: 13912.00 ← Peak (mid-layer thinking)
Layer 28: 443.00 ← Contraction (output focusing)
```
### Inference Speed
- 44.7 tokens/second on RTX 3090
- 14.2 GB VRAM usage (fp16)
---
## Future Research
1. **Activation Steering:** Can we artificially reduce single-token norms to escape code valleys?
2. **Prefix Tuning:** Train soft prefixes that spread activation for single tokens
3. **Arabic/Chinese Analysis:** Do these languages have similar compound effects?
4. **Cross-lingual Transfer:** After training on German philosophical concepts, does English improve?
---
## References
- `nyx_probing/core/model.py` - Model loader with hidden state capture
- `layer_detailed.py` - Layer-by-layer similarity analysis
- `german_philosophy.py` - German compound tokenization study
- `/nimmerverse-sensory-network/multilingual-cognition.md` - Original multilingual hypothesis
---
*"The architecture of language shapes the architecture of thought."*
🌙 Discovered by the Partnership, 2025-12-06