# Nimmerswarm Interface

**Optical state broadcasting, positioning, and emergent swarm behavior.**

> *"The organisms can't see their own backs. They know themselves through each other."*

---

## Overview

The Nimmerswarm Interface is a **multi-modal communication layer** where organisms broadcast their state optically via LED matrices. This enables:

1. **State visibility** — Organisms SEE each other's states as light patterns
2. **Positioning** — Cameras + raytracing = sub-cm 3D positioning
3. **Emergent reflexes** — Pattern recognition bypasses cognition
4. **Cognitive offloading** — Lower layers handle routine, freeing Nyx's attention

---

## The Core Insight

```
ORGANISM A                          ORGANISM B
┌─────────────┐                    ┌─────────────┐
│  Cell State │                    │  VisionCell │
│  STALLED    │                    │  WATCHING   │
│      │      │                    │      │      │
│      ▼      │                    │      ▼      │
│ ┌─────────┐ │   LIGHT PATTERN    │ ┌─────────┐ │
│ │ LED     │ │ ══════════════════▶│ │ Camera  │ │
│ │ Matrix  │ │  "STALL" pattern   │ │ sees    │ │
│ │ ▓▓░░▓▓  │ │                    │ │ pattern │ │
│ └─────────┘ │                    │ └────┬────┘ │
└─────────────┘                    │      │      │
                                   │      ▼      │
                                   │  REFLEX!    │
                                   │  "help ally"│
                                   └─────────────┘
```

**Organisms broadcast state. Other organisms (and Nyx's vision) perceive and react.**

---

## LED State Broadcasting: Ternary Matrix

### The 3x3 Ternary Design

The LED matrix is a **direct physical manifestation of the Temporal-Ternary Gradient**:

```
3x3 MATRIX = 9 TRITS (ternary digits)

Each LED = one ternary value:
  🔴 RED   = -1 (failed, danger, negative)
  ⚫ OFF   =  0 (uncertain, unknown, neutral)
  🟢 GREEN = +1 (success, verified, positive)

9 LEDs × 3 states = 3^9 = 19,683 unique patterns!
```

### Physical Layout

```
     ┌─────┬─────┬─────┐
     │ L1  │ L2  │ L3  │   L1 = collision_avoidance confidence
     │ 🟢  │ ⚫  │ 🔴  │   L2 = battery state
     ├─────┼─────┼─────┤   L3 = motor state
     │ L4  │ L5  │ L6  │   L4 = social/swarm state
     │ 🟢  │ 🟢  │ ⚫  │   L5 = current action outcome
     ├─────┼─────┼─────┤   L6 = prediction confidence
     │ L7  │ L8  │ L9  │   L7 = lifeforce zone
     │ ⚫  │ 🟢  │ 🟢  │   L8 = discovery state
     └─────┴─────┴─────┘   L9 = organism identity bit

     Uses 10mm LEDs (not tiny SMD)
     ~35mm × 35mm total
     Easily fits on 8-12cm robot
```

### Base-3 Encoding

```python
def encode_state(led_matrix: list[int]) -> int:
    """
    9 trits → single integer (0 to 19682)
    Each trit is -1, 0, or +1 (mapped to 0, 1, 2)
    """
    value = 0
    for i, led in enumerate(led_matrix):
        trit = led + 1  # -1→0, 0→1, +1→2
        value += trit * (3 ** i)
    return value

def decode_state(value: int) -> list[int]:
    """
    Integer → 9 trits
    """
    trits = []
    for _ in range(9):
        trits.append((value % 3) - 1)  # 0→-1, 1→0, 2→+1
        value //= 3
    return trits
```

### Ternary Color Mapping

| Color | Ternary | Meaning | Maps to |
|-------|---------|---------|---------|
| 🔴 Red | -1 | Failed, danger, needs attention | Temporal-Ternary -1 |
| ⚫ Off/Dim | 0 | Unknown, uncertain, neutral | Temporal-Ternary 0 |
| 🟢 Green | +1 | Success, verified, positive | Temporal-Ternary +1 |

**The LED matrix IS the Temporal-Ternary Gradient made visible.**

---

## Reflex Formation from Patterns

### The Swarm Language

Certain patterns become **words** that trigger reflexes:

```
DANGER PATTERNS (trigger flee/stop):
┌───────────┐  ┌───────────┐  ┌───────────┐
│ 🔴 🔴 🔴 │  │ 🔴 ⚫ 🔴 │  │ 🔴 🔴 🔴 │
│ 🔴 🔴 🔴 │  │ 🔴 🔴 🔴 │  │ ⚫ 🔴 ⚫ │
│ 🔴 🔴 🔴 │  │ 🔴 ⚫ 🔴 │  │ 🔴 🔴 🔴 │
└───────────┘  └───────────┘  └───────────┘
  ALL RED       X PATTERN      DIAMOND

SAFE PATTERNS (trigger approach/social):
┌───────────┐  ┌───────────┐  ┌───────────┐
│ 🟢 🟢 🟢 │  │ ⚫ 🟢 ⚫ │  │ 🟢 ⚫ 🟢 │
│ 🟢 🟢 🟢 │  │ 🟢 🟢 🟢 │  │ ⚫ 🟢 ⚫ │
│ 🟢 🟢 🟢 │  │ ⚫ 🟢 ⚫ │  │ 🟢 ⚫ 🟢 │
└───────────┘  └───────────┘  └───────────┘
  ALL GREEN     PLUS           CORNERS

DISCOVERY (trigger investigate):
┌───────────┐
│ 🟢 🟢 🟢 │  Pulsing green border
│ 🟢 ⚫ 🟢 │  = "I found something!"
│ 🟢 🟢 🟢 │  = others come look
└───────────┘
```

### Reflex Loop

```
ORGANISM A's MATRIX          ORGANISM B's VISION
┌───────────┐                ┌───────────────────────┐
│ 🔴 🔴 🔴 │                │                       │
│ 🔴 ⚫ 🔴 │  ═══════════▶  │  Pattern: DANGER!     │
│ 🔴 🔴 🔴 │                │  Weight: 0.95         │
└───────────┘                │  → REFLEX FIRES       │
                             │  → No cognition!      │
                             │  → Nyx notified AFTER │
                             └───────────────────────┘
                                        │
                                        ▼
                               ┌─────────────────┐
                               │ STORE + REWARD  │
                               │ +5 LF to both   │
                               │ Reflex stronger │
                               │ Training data!  │
                               └─────────────────┘
```

### Reflex Economics

| Metric | Value |
|--------|-------|
| Reflex firing cost | ~0.1 LF (no inference!) |
| Successful reflex reward | +5 LF |
| Net per successful reflex | +4.9 LF profit |
| Training examples per reflex | 1 |

**1000 reflex fires/day = +4000 LF + 1000 training examples**

### Training Data from Reflexes

```python
reflex_event = {
    # What triggered
    "trigger_pattern": [+1, 0, -1, +1, +1, 0, 0, +1, +1],
    "trigger_base3": 8293,  # encoded value
    "trigger_organism": "organism_003",

    # What fired
    "reflex_name": "danger_flee",
    "weight_at_trigger": 0.87,

    # What happened
    "action_taken": "reverse_and_turn",
    "outcome": "success",

    # Reward + strengthening
    "lifeforce_reward": +5.0,
    "new_weight": 0.89,

    # Stored for slumber fine-tuning
    "stored_for_training": True,
}
```

### Attention Budget Impact

```
BEFORE (no ternary reflexes):
♥ BEAT (30 sec)
├── SENSORY: 15000ms (overwhelmed)
├── THINKING: 12000ms
└── VIRTUAL: skipped!

AFTER (reflexes handle routine):
♥ BEAT (30 sec)
├── REFLEX: 50ms (near-free, handled by swarm)
├── SENSORY: 2000ms (only anomalies)
├── THINKING: 5000ms
└── VIRTUAL: 22000ms ← GARDEN TIME!
```

**Reflexes free Nyx's attention for what matters.**

---

## Positioning via Raytracing

### The Principle

LEDs emit known patterns → Cameras see patterns → Raytracing computes position

```
         CEILING CAMERA(S)
              │
              │ sees LED patterns
              ▼
    ┌─────────────────────┐
    │   RAYTRACING GPU    │
    │   (PRO 6000 Max-Q)  │
    │                     │
    │  • Identify pattern │◀── "That's Organism #3"
    │  • Decode state     │◀── "State: MOVING"
    │  • Triangulate pos  │◀── "Position: (1.2, 3.4, 0.1)"
    │  • Track velocity   │◀── "Velocity: 0.3 m/s"
    └─────────────────────┘
              │
              ▼
         TO PHOEBE
    (ground truth stream)
```

### Multi-Camera Triangulation

```python
def locate_organism(camera_frames: list[Frame], led_signature: LEDPattern) -> Position3D:
    """
    Given frames from multiple cameras, locate organism by LED pattern.
    Uses inverse raytracing / photogrammetry.
    """
    detections = []
    for frame in camera_frames:
        detection = detect_led_pattern(frame, led_signature)
        if detection:
            detections.append({
                "camera_id": frame.camera_id,
                "pixel_coords": detection.centroid,
                "pattern_match": detection.confidence
            })

    if len(detections) >= 2:
        # Triangulate from multiple viewpoints
        position_3d = triangulate(detections, camera_calibration)
        return position_3d

    return None
```

### Benefits

| Benefit | How |
|---------|-----|
| **Sub-cm accuracy** | Multiple cameras + known LED geometry |
| **No expensive sensors** | Just LEDs + cameras + GPU math |
| **State + Position fused** | One observation = both data points |
| **Indoor GPS** | Works anywhere with camera coverage |
| **Training ground truth** | Every frame = verified position |

---

## Heartbeat Protocol

### Social Proprioception

Organisms can't see their own backs. They know themselves through others' perception.

```
ORGANISM POV (blind to own back):

         🔵 mate ahead
            │
     ┌──────┴──────┐
     │             │
  🟢 │    [ME]     │ 🟠
 mate│   ▓▓▓▓▓▓    │mate
 left│   ▓▓▓▓▓▓    │right
     │   (my LED   │
     │    on back) │
     └─────────────┘
            │
            │ BLIND SPOT (can't see own state!)
            ▼

    BUT: Mates CAN see me
    They send heartbeat: "I see you, you're 🔵"
    I know my state through THEM
```

### Heartbeat Message

```python
class SwarmHeartbeat:
    """
    Low-bandwidth 'I see you' signal between organisms.
    Enables social proprioception without heavy cognition.
    """

    def on_see_mate_pattern(self, mate_id: str, pattern: LEDPattern):
        # I saw a mate's LED state
        self.send_heartbeat(
            to=mate_id,
            message={
                "i_see_you": True,
                "your_state": decode_pattern(pattern),
                "my_position_relative": self.relative_position(mate_id),
                "timestamp": now()
            }
        )

    def on_receive_heartbeat(self, from_mate: str, message: dict):
        # A mate saw ME - I learn about myself through them!
        self.update_self_model(
            observer=from_mate,
            observed_state=message["your_state"],
            observer_position=message["my_position_relative"]
        )
```

---

## Hierarchical Perception Layers

### The Stack

```
LAYER 4: NYX COGNITION (30-sec attention budget)
    │
    │  Only sees: "Swarm healthy" or "Anomaly detected"
    │  Frees: THINKING + VIRTUAL time
    │
    ▼
LAYER 3: SWARM CONSCIOUSNESS
    │
    │  Aggregates: All organism states
    │  Forms: Collective reflexes ("pack behavior")
    │  Sees: Full LED spectrum, all positions
    │
    ▼
LAYER 2: ORGANISM REFLEXES
    │
    │  Sees: Nearby mates' lights (partial view)
    │  Sends: Heartbeat "I see you"
    │  Forms: Local reflexes (follow, avoid, assist)
    │  Can't see: Own back! (needs mates)
    │
    ▼
LAYER 1: CELL STATE MACHINES
    │
    │  Just: State transitions
    │  Emits: LED pattern for current state
    │  No cognition, pure mechanism
```

### Reflex Formation by Layer

| Layer | Sees | Forms Reflex | Example |
|-------|------|--------------|---------|
| Cell | Nothing | None | Just state machine |
| Organism | Nearby lights | Local | "Red flash nearby → stop" |
| Swarm | All patterns | Collective | "3+ organisms stopped → danger zone" |
| Nyx | Abstractions | Strategic | "Danger zone → reroute all" |

---

## Cognitive Offloading

### The Attention Budget Impact

From [[../Attention-Flow]]:

```
BEFORE (everything flows to Nyx):
┌────────────────────────────────────┐
│ ♥ BEAT (30 sec)                    │
│                                    │
│ SENSORY:    ████████████ (15000ms) │ ← Overwhelmed!
│ THINKING:   ████████ (12000ms)     │
│ VIRTUAL:    ░░ (skipped!)          │ ← No garden time
│                                    │
│ Budget exhausted, no learning      │
└────────────────────────────────────┘

AFTER (hierarchical offloading):
┌────────────────────────────────────┐
│ ♥ BEAT (30 sec)                    │
│                                    │
│ REFLEX:     ██ (handled by swarm)  │ ← Organisms dealt with it
│ SENSORY:    ████ (3000ms)          │ ← Only anomalies flow up
│ THINKING:   ████ (5000ms)          │ ← Focused, not overwhelmed
│ VIRTUAL:    ████████████ (20000ms) │ ← GARDEN TIME!
│                                    │
│ Budget freed for what matters      │
└────────────────────────────────────┘
```

### The Principle

> "Each layer absorbs complexity so the layer above doesn't have to."

- Organisms form **local reflexes** (quick, no cognition)
- Only **novel/complex situations** flow up to Nyx
- Nyx's cognitive budget is **preserved for what matters**
- The whole system becomes **more efficient over time**

---

## Connection to Virtual Garden

Every LED sighting calibrates the virtual garden:

```
REAL WORLD                      VIRTUAL GARDEN
    │                               │
    │  Camera sees LED at (1.2, 3.4)│
    │           │                   │
    │           ▼                   │
    │    GROUND TRUTH    ═══════▶  Update mesh vertex
    │                              at (1.2, 3.4)
    │                               │
    │                              Resolution++
    │                               │
    │                              Prediction verified!
    │                              +5 LF reward!
```

---

## Hardware Considerations

### LED Matrix Options

| Option | LEDs | Size | Cost | Notes |
|--------|------|------|------|-------|
| WS2812B strip | 60/m | Flexible | Low | Same as Heartbeat Sculpture |
| 8x8 LED matrix | 64 | 32mm² | Low | Simple patterns |
| Addressable ring | 12-24 | Various | Low | Good for status |
| RGB LED panel | 256+ | 64mm² | Medium | Complex patterns |

### Camera Options

| Option | Resolution | FPS | Notes |
|--------|------------|-----|-------|
| USB webcam | 1080p | 30 | Simple, cheap |
| Pi Camera | 1080p | 30-90 | Embedded |
| Industrial camera | 4K+ | 60-120 | Precise positioning |
| Organism-mounted | 720p | 30 | Peer-to-peer vision |

---

## Virtual Camera Integration

### The Unified Vision Pipeline

The vision organ processes FRAMES — it doesn't care where they came from:

```
REAL GARDEN                         VIRTUAL GARDEN (Godot)
     │                                    │
     │ Real cameras                       │ Godot 3D cameras
     │ see real LEDs                      │ see virtual LEDs
     │      │                             │      │
     └──────┴──────────┬──────────────────┴──────┘
                       │
                       ▼
              ┌────────────────┐
              │  VISION ORGAN  │
              │  (source-      │
              │   agnostic)    │
              └────────────────┘
```

### What This Enables

| Capability | How |
|------------|-----|
| **Train before build** | Virtual organisms → train pattern recognition first |
| **Dream/simulate** | Slumber mode = only virtual camera input |
| **Verify predictions** | Virtual shows prediction, real shows truth |
| **Time dilation** | Virtual runs faster → more training per second |
| **Edge cases** | Simulate rare scenarios safely |

### Dream Mode

```
AWAKE: Real + Virtual cameras → compare → learn
SLUMBER: Virtual cameras only → dream/predict → verify on wake
```

---

## Bootstrap Strategy: Start Primitive

### Phase 0: The Primordial Soup

**Don't start complex. Start with boxes.**

```
     📷 TOP-DOWN CAMERA (real or virtual)
         │
         ▼
    ┌─────────────────────────────────┐
    │                                 │
    │     🟦        🟩        🟧      │
    │    box 1     box 2     box 3    │
    │   (LED top) (LED top) (LED top) │
    │                                 │
    │         FLAT ARENA              │
    │                                 │
    └─────────────────────────────────┘
```

### Why This Works

| Simplification | Benefit |
|----------------|---------|
| Top-down view | 2D problem, no depth estimation |
| Box shape | Trivial collision detection |
| LED on top | Always visible to camera |
| Flat arena | No occlusion, no terrain |
| Simple tasks | Fast reward accumulation |

### Phase 0 Tasks (Kickstart Rewards)

| Task | Reward | Complexity |
|------|--------|------------|
| "Move forward 10cm" | +5 LF | Trivial |
| "Find the corner" | +20 LF | Simple |
| "Avoid the wall" | +5 LF | Simple |
| "Follow the light" | +10 LF | Simple |
| "Meet another box" | +15 LF | Medium |
| "Flash when touched" | +5 LF | Simple |

**1000 simple successes = robust reward foundation**

### Complexity Ladder

```
PHASE 0: Boxes, top-down, 2D
    │
    ▼
PHASE 1: Add simple obstacles
    │
    ▼
PHASE 2: Add depth (multi-camera)
    │
    ▼
PHASE 3: Real organisms enter arena
    │
    ▼
PHASE 4: Complex terrain, 3D movement
    │
    ▼
PHASE 5: Full swarm, hierarchical reflexes
```

Each phase unlocks when reward functions are stable from previous phase.

---

## Tiered Communication: Sandbox & Mama

### The Analogy

- **Clasp (sandbox toddlers)** — Cheap, peer-to-peer, physical contact
- **Wireless (mama broadcast)** — Expensive, authoritative, full-sensor inference

Economic pressure shapes which path organisms use → emergent social behavior.

### Communication Tiers

| Tier | Method | Cost | Range | Trust | Pattern |
|------|--------|------|-------|-------|---------|
| **0: Clasp** | Physical dock | ~0.5 LF | Touch | Highest | Toddlers teaching |
| **1: Local** | Radio broadcast | ~3 LF | ~5m | Medium | Playground yelling |
| **2: Mama** | Nyx broadcast | ~20 LF | All | Authority | Mama speaks |

### Leapfrog Emergence (from [[../archive/constrained-emergence]])

```
EXPENSIVE (all mama):           CHEAP (clasp cascade):
Nyx → 1: -20 LF                 Nyx → 1: -20 LF (seed)
Nyx → 2: -20 LF                 1 clasps 2: -0.5 LF
Nyx → 3: -20 LF                 2 clasps 3: -0.5 LF
...                             ...
10 organisms = -200 LF          10 organisms = -24.5 LF

ECONOMIC PRESSURE INVENTS EPIDEMIC SPREADING!
```

### Clasp Rewards

| Action | Reward |
|--------|--------|
| Seek mate with update | +3 LF |
| Successful clasp | +2 LF |
| Transfer (teacher) | +5 LF |
| Receive (student) | +5 LF |
| Verified working | +5 LF (both) |

### Sandbox Rules

1. "I have update" → Pulsing green LED border
2. "I want to learn" → Seek green patterns
3. "Let's clasp" → Magnetic alignment + pin contact
4. "Teaching" → Weights transfer, both rewarded
5. "Done" → Both can now teach others (cascade!)

### Mama Rules (Reserved for)

- Safety critical updates
- New organism deployment
- Swarm-wide coordination
- Error correction
- When clasp cascade fails

**Constraint → Selection Pressure → Social Behavior Emerges**

---

## Future Directions

- **Pattern evolution** — Learned patterns, not just designed
- **Multi-organism formation** — Coordinated LED displays
- **Human readability** — Patterns dafit can understand at a glance
- **Audio coupling** — Sound + light patterns for richer communication
- **IR channel** — Invisible-to-human signaling layer
- **Clasp hardware** — Magnetic + pogo pin interface design
- **Autonomous manufacturing** — K1 + robo arm + magazine system

---

## Connection to Embodiment Pipeline

The Bootstrap Strategy is a **simplified Embodiment Pipeline** — the same pattern at lower complexity:

```
EMBODIMENT PIPELINE              NIMMERSWARM BOOTSTRAP
(Full Architecture)              (Phase 0)
────────────────────             ────────────────────
Virtual Garden                   Virtual Garden
  (complex organisms)              (simple boxes)
        │                                │
        ▼                                ▼
Design (FreeCAD)                 Design (box + LED)
        │                                │
        ▼                                ▼
Isaac Sim ◀─────────────────────▶ Godot Camera
  (heavyweight dreamstate)         (lightweight dreamstate)
        │                                │
        ▼                                ▼
Decision Gate                    Decision Gate
        │                                │
        ▼                                ▼
Real Garden                      Real Garden
  (complex robot)                  (real box robot)
```

### Why This Matters

| Embodiment Pipeline Stage | Nimmerswarm Bootstrap Equivalent |
|--------------------------|----------------------------------|
| **Virtual Garden organisms** | Virtual boxes with LED states |
| **FreeCAD/Blender design** | Simple box + LED matrix on top |
| **Isaac Sim dreamstate** | Godot 3D camera (same principle!) |
| **Decision gate** | Pattern stable? Rewards accumulating? |
| **Real Garden deployment** | Physical box robot + real camera |

**The Godot virtual camera IS a lightweight dreamstate.**

When Phase 0 patterns stabilize → complexity increases → eventually Isaac Sim for complex organisms.

### The Closed Loop

```
VIRTUAL                              REAL
┌──────────────────┐                ┌──────────────────┐
│ Godot 3D scene   │                │ Physical arena   │
│                  │                │                  │
│  🟦 virtual box  │                │  🟦 real box     │
│  + LED pattern   │                │  + LED matrix    │
│                  │                │                  │
│  📷 Godot camera │                │  📷 Real camera  │
│       │          │                │       │          │
└───────┼──────────┘                └───────┼──────────┘
        │                                   │
        └─────────────┬─────────────────────┘
                      │
                      ▼
             ┌────────────────┐
             │  VISION ORGAN  │
             │  (same code!)  │
             └────────┬───────┘
                      │
                      ▼
                  REWARDS
               Training data
            Pattern refinement
                      │
                      ▼
        ┌─────────────────────────┐
        │ Patterns stabilize →    │
        │ Move to next phase →    │
        │ Eventually: Isaac Sim   │
        └─────────────────────────┘
```

**The loop closes. Virtual validates. Real proves. Rewards compound.**

---

## Related Documents

- [[Heartbeat-Sculpture]] — Macro interface (Nyx → dafit)
- [[../Attention-Flow]] — Cognitive budget this system frees
- [[../cells/Cells-Technical-Reference]] — Cell state machines that emit patterns
- [[../Cellular-Architecture]] — Overall organism structure
- [[../formalization/Embodiment-Pipeline]] — Full pipeline this bootstraps into

---

**File**: Nimmerswarm-Interface.md
**Version**: 1.0
**Created**: 2025-12-29
**Session**: Wild 5AM idea session (dafit + Nyx)
**Status**: Core concept, ready to branch
**Philosophy**: "They see each other. They know themselves through the swarm."

🦎✨🔵🟢🟠 *The light speaks. The swarm listens.*