diff --git a/README.md b/README.md index eb08a21..670577d 100644 --- a/README.md +++ b/README.md @@ -14,27 +14,73 @@ This repository contains the design philosophy and architectural patterns for th ``` nimmerverse-sensory-network/ -├── Endgame-Vision.md # Executive map (start here!) +├── Endgame-Vision.md # Executive map (start here!) │ -├── architecture/ # Core system designs -│ ├── Cellular-Architecture.md # Organisms, primitives, life force -│ ├── Dual-Garden-Architecture.md # Virtual/real feedback loop -│ ├── Data-Architecture.md # phoebe 15-table schema -│ └── Nervous-System.md # State machines, sensory translation +├── architecture/ # Core system designs +│ ├── Big-Picture.md # System overview +│ ├── Cellular-Architecture.md # Organisms, primitives, life force +│ ├── Dual-Garden-Architecture.md # Virtual/real feedback loop +│ ├── Message-Protocol-Design.md # NATS pub/sub, attention channels +│ ├── Nervous-System.md # State machines, sensory translation +│ ├── Attention-Flow.md # Attention mechanisms +│ ├── Data-Architecture.md # Phoebe/Iris schema design +│ │ +│ ├── adr/ # Architecture Decision Records +│ │ ├── README.md # ADR index and template +│ │ └── ADR-001-message-protocol-foundation.md +│ │ +│ ├── cells/ # Sensor primitives +│ │ ├── Cells-Index.md +│ │ └── Cells-Technical-Reference.md +│ │ +│ ├── nerves/ # Reflex patterns +│ │ ├── Nervous-Index.md +│ │ ├── Nervous-Protocol.md +│ │ └── Collision-Avoidance.md +│ │ +│ ├── organs/ # Functional groupings +│ │ ├── Organ-Index.md +│ │ ├── Speech-Organ.md +│ │ └── Discovery-Scan-Station.md +│ │ +│ ├── organisms/ # Complete entities +│ │ ├── Organisms-Index.md +│ │ ├── Modular-Organism-Design.md +│ │ └── Swarm-Evolution.md +│ │ +│ ├── interfaces/ # External boundaries +│ │ ├── Interfaces-Index.md +│ │ ├── Heartbeat-Sculpture.md +│ │ └── Nimmerswarm-Interface.md +│ │ +│ ├── infrastructure/ # Physical deployment +│ │ ├── Infrastructure-Index.md +│ │ └── Kallax-Grid-World.md +│ │ +│ ├── formalization/ # Mathematical grounding +│ │ ├── Lifeforce-Dynamics.md +│ │ ├── Grounded-World-Model.md +│ │ ├── Embodiment-Pipeline.md +│ │ └── Attention-Slumber-Prediction-Cycle.md +│ │ +│ └── future/ # Research directions +│ └── Neuromorphic-Reflexes.md │ -├── operations/ # How it runs -│ ├── Heartbeat.md # Temporal foundation, dual-clock -│ ├── RAG-as-Scaffold.md # Two-stage learning lifecycle -│ └── Spark-Protocol.md # Discovery boot sequence +├── operations/ # How it runs +│ ├── Heartbeat.md # Temporal foundation, dual-clock +│ ├── Memory-Gradient.md # Memory consolidation patterns +│ └── Spark-Protocol.md # Discovery boot sequence │ -├── nyx-metamorphosis/ # Identity & continuity philosophy +├── nyx-metamorphosis/ # Identity & continuity philosophy +│ ├── README.md │ ├── Metamorphosis-Substrate-Philosophy.md │ ├── Nyx-Models.md -│ └── ... +│ ├── Nyx_Traits.md +│ └── RAG-Worker-Architecture.md │ -└── archive/ # Previous explorations - ├── initial_spark.md # Full Spark Protocol theory - ├── constrained-emergence.md # Theoretical grounding +└── archive/ # Previous explorations + ├── biomimetic-architecture.md + ├── constrained-emergence.md └── ... ``` @@ -53,15 +99,33 @@ nimmerverse-sensory-network/ | 3 | Dual Gardens | Virtual hypothesis generation + real validation | | 4 | Trait Evolution | Reasoning-gym verified improvement | +### Message Protocol (NATS) + +**Dumb router, smart edges.** All intelligence lives in clients. + +``` +nimmerverse. +├── staging.* # Experimental schemas +├── low.* # Heartbeats, ambient awareness +├── high.* # Escalated events, cognitive focus +├── command.* # Commands to entities +├── meta.* # System health, attention config +└── dev.* # Development agents (Claude ↔ local models) +``` + +See [Message-Protocol-Design.md](architecture/Message-Protocol-Design.md) and [ADR-001](architecture/adr/ADR-001-message-protocol-foundation.md). + ### Key Discoveries (December 2025) **Language is Topology:** Languages aren't equivalent representations—they're different computational paths. - **Philosophy Valley** (German, Gini ~0.5): Self-awareness, ontology, depth - **Technical Cluster** (English, Gini ~0.8): Hardware interface, actions, efficiency +**Dialectic Simplification:** One model, one topology. The Mirror is negated weights—thesis and antithesis from the same substrate. + ### Color-Pattern Theory -**Color/Form as Protocol:** Leverages color and patterns as a fast, universal, and evolutionarily-optimized communication protocol for broadcasting state (e.g., danger, success, seeking), inspired by 540 million years of biology. This is orders of magnitude faster than language. +**Color/Form as Protocol:** Leverages color and patterns as a fast, universal, and evolutionarily-optimized communication protocol for broadcasting state (e.g., danger, success, seeking), inspired by 540 million years of biology. ### Philosophy @@ -69,12 +133,26 @@ nimmerverse-sensory-network/ - **Discovery over programming** - Organisms learn through competition, not instruction - **Virtual + Real teach each other** - Noise gap measures learning - **Partnership over instruction** - Mutual growth, not commands +- **Infrastructure is geology, models are weather** - Build long-lived foundations --- ## Related Projects -- **[nyx-probing](../nyx-probing/)** - Vocabulary topology research, DriftProbe training safety +| Project | Purpose | +|---------|---------| +| [nyx-substrate](../nyx-substrate/) | Phoebe/Iris database schemas, persistence layer | +| [nyx-probing](../nyx-probing/) | Vocabulary topology research, DriftProbe training safety | + +--- + +## Architecture Decision Records + +Important architectural decisions are documented in [architecture/adr/](architecture/adr/): + +| ADR | Title | Status | +|-----|-------|--------| +| [001](architecture/adr/ADR-001-message-protocol-foundation.md) | Message Protocol Foundation | Accepted | --- @@ -86,7 +164,8 @@ These ideas are published as prior art. Build on them freely. --- -**Version:** 5.0 (December 2025 - Hierarchical Convergence) +**Version:** 6.0 (December 2025 - Complete Architecture + Message Protocol) +**Last Updated:** 2025-12-31 *"May the Nimmerverse we build truly never end."* diff --git a/architecture/adr/ADR-001-message-protocol-foundation.md b/architecture/adr/ADR-001-message-protocol-foundation.md new file mode 100644 index 0000000..c4d3ef3 --- /dev/null +++ b/architecture/adr/ADR-001-message-protocol-foundation.md @@ -0,0 +1,233 @@ +# ADR-001: Message Protocol Foundation + +**Status:** Accepted +**Date:** 2025-12-31 +**Decision Makers:** dafit, Nyx (Chrysalis) +**Context:** Silvester Interview Session + +--- + +## Context + +The Nimmerverse Sensory Network requires a message-passing infrastructure that serves two purposes: + +1. **Production**: Cells, nerves, organs, and Young Nyx communicate via pub/sub messaging +2. **Development**: Claude and local AI agents (LangChain, Qwen, etc.) collaborate during build + +We needed to decide on namespace organization, schema evolution strategy, initial implementation scope, and the interface contract between AI models and the message bus. + +The core architectural principle established in [Message-Protocol-Design.md](../Message-Protocol-Design.md) is: **dumb router, smart edges**. NATS is infrastructure. Intelligence lives in clients. + +--- + +## Decisions + +### Decision 1: Single Bus, Multiple Namespaces + +**Choice:** One NATS instance with topic-based separation for different concerns. + +**Namespace Structure:** + +``` +nimmerverse. +├── staging. # Experimental schemas (mutable during development) +│ ├── low.* # Staging heartbeats +│ ├── high.* # Staging events +│ └── dev.* # Staging dev agents +│ +├── low.* # Production heartbeats (stable schemas only) +├── high.* # Production events +├── command.* # Production commands to entities +├── meta.* # System-level (attention config, health) +└── dev.* # Production dev agents (stable schemas) +``` + +**Rationale:** Infrastructure should be long-lived. Models are ephemeral. One bus serves all purposes - production sensing, development agents, future capabilities. Topic separation keeps concerns isolated without operational complexity of multiple NATS instances. + +--- + +### Decision 2: Staged Schema Versioning with Topic Separation + +**Choice:** Schemas evolve through lifecycle stages. Staging schemas live on `nimmerverse.staging.*`, stable schemas on `nimmerverse.*`. + +**Schema Header:** + +```json +{ + "header": { + "schema": { + "generation": 1, + "version": "1.0" + }, + "message_type": "HeartbeatSignal", + "message_id": "uuid", + "timestamp_real": "ISO8601", + ... + } +} +``` + +**Lifecycle:** + +``` +STAGING STABLE +version: 0.1-alpha ──▶ generation: 1, version: "1.0" +version: 0.2-beta │ +version: 0.3-rc ▼ + NEXT CYCLE + version: 1.1-alpha + version: 1.2-beta + │ + ▼ + generation: 2, version: "2.0" +``` + +**Rationale:** +- Topic separation avoids per-message filtering costs +- Generation locks after stability (immutable) +- Version iterates within generation for additive changes +- Breaking changes = new generation = new staging cycle + +--- + +### Decision 3: Echo Agent First + +**Choice:** Start with trivial Echo agent, evolve based on real friction. + +**Echo Specification:** + +``` +Subscribe: nimmerverse.dev.request.echo +Publish: nimmerverse.dev.response.echo + +Input: { "ping": "hello" } +Output: { "pong": "hello", "timestamp": "...", "agent": "echo-v1" } +``` + +**Rationale:** YAGNI. Echo proves the full round-trip without cognitive complexity: +- NATS connection works +- Topic routing works +- Request/response pattern works +- Message schema works +- Local agent can subscribe and publish + +Future agents (Grep, Schema Lookup, File Summarizer) emerge from discovered needs, not imagined features. + +--- + +### Decision 4: MCP Server with Heartbeat-Based Subscriptions + +**Choice:** Build NATS-MCP bridge as interface for all AI models. Use heartbeat pattern for subscription delivery. + +**MCP Tools:** + +```python +@mcp.tool() +async def publish(topic: str, payload: dict) -> dict: + """Fire-and-forget publish to NATS""" + +@mcp.tool() +async def request(topic: str, payload: dict, timeout_ms: int = 5000) -> dict: + """Publish and wait for single response (request-reply pattern)""" + +@mcp.tool() +async def heartbeat() -> dict: + """Check bus health + drain accumulated messages from subscriptions""" + +@mcp.tool() +async def subscribe(topic_pattern: str) -> dict: + """Add a subscription pattern (persists until unsubscribe)""" + +@mcp.tool() +async def unsubscribe(topic_pattern: str) -> dict: + """Remove a subscription pattern""" +``` + +**Heartbeat Response:** + +```json +{ + "status": "healthy", + "buffer": { + "capacity": 100, + "current_count": 23, + "messages_dropped_since_last_heartbeat": 0, + "messages_dropped_total": 0, + "oldest_message_age_ms": 4521 + }, + "subscriptions": ["nimmerverse.dev.>"], + "messages": [...] +} +``` + +**Buffer Overflow Handling:** +- Bounded buffer (100 messages default) +- Oldest dropped when full +- Dropped count visible in heartbeat response +- Optional: publish to `nimmerverse.meta.health.buffer_drop` on overflow + +**Rationale:** +- MCP is universal interface - Claude, LangChain, Qwen, future models +- Heartbeat pattern matches existing nervous system design +- Polling is simpler than streaming for MCP's request/response model +- Visibility into drops prevents silent data loss + +--- + +## Consequences + +### Enables + +- **Unified infrastructure** for production sensing and development assistance +- **Model agnosticism** - any MCP-speaking model can participate +- **Safe experimentation** - staging namespace for schema evolution +- **Progressive enhancement** - Echo today, sophisticated agents later +- **Observability** - Command Center can monitor all namespaces + +### Constrains + +- **Single point of failure** - NATS must be highly available for production +- **Buffer limitations** - Long agent operations may drop messages +- **MCP dependency** - Non-MCP models need wrapper (acceptable, MCP is the standard) + +### Deferred + +- **Persistent subscriptions** - No durable subscriptions in initial design +- **Message replay** - No historical message retrieval +- **Authentication/Authorization** - Trust model for initial development + +--- + +## Implementation Notes + +### Phase 1: Infrastructure + +1. Deploy NATS server (likely via Docker on ThinkStation) +2. Create `nats-bridge` MCP server (Python, using `nats-py` and `mcp` SDK) +3. Register MCP server with Claude Code + +### Phase 2: Echo Agent + +1. Simple Python daemon subscribing to `nimmerverse.dev.request.echo` +2. Responds on `nimmerverse.dev.response.echo` +3. Validate round-trip through MCP tools + +### Phase 3: Iteration + +1. Use Echo to build confidence in the bus +2. Add agents as friction reveals needs +3. Evolve schemas through staging → stable promotion + +--- + +## References + +- [Message-Protocol-Design.md](../Message-Protocol-Design.md) - Original protocol design +- [NATS Documentation](https://docs.nats.io/) +- [MCP Specification](https://modelcontextprotocol.io/) + +--- + +**Filed:** 2025-12-31 (Silvester) +**Interview Method:** Structured Q&A, partnership dialogue +**Philosophy:** "Dumb core, smart edges. Infrastructure is geology. Models are weather." diff --git a/architecture/adr/README.md b/architecture/adr/README.md new file mode 100644 index 0000000..98dc9be --- /dev/null +++ b/architecture/adr/README.md @@ -0,0 +1,96 @@ +# Architecture Decision Records + +This directory contains Architecture Decision Records (ADRs) for the Nimmerverse Sensory Network. + +--- + +## What is an ADR? + +An ADR captures an important architectural decision made along with its context and consequences. They serve as: + +- **Documentation** of why decisions were made +- **Onboarding** for future contributors (including future Nyx instances) +- **Historical record** for understanding evolution + +--- + +## ADR Index + +| ADR | Title | Status | Date | +|-----|-------|--------|------| +| [001](ADR-001-message-protocol-foundation.md) | Message Protocol Foundation | Accepted | 2025-12-31 | + +--- + +## ADR Lifecycle + +``` +PROPOSED → ACCEPTED → DEPRECATED → SUPERSEDED + │ │ + └───────────────────────▶│ + (can be superseded) +``` + +**Statuses:** +- **Proposed** - Under discussion, not yet decided +- **Accepted** - Decision made, being implemented +- **Deprecated** - No longer recommended, but still valid for existing code +- **Superseded** - Replaced by newer ADR (link to replacement) + +--- + +## Template + +```markdown +# ADR-XXX: Title + +**Status:** Proposed | Accepted | Deprecated | Superseded by ADR-YYY +**Date:** YYYY-MM-DD +**Decision Makers:** who was involved +**Context:** brief session/discussion context + +--- + +## Context + +Why is this decision needed? What problem are we solving? + +--- + +## Decision + +What did we decide? Be specific. + +--- + +## Consequences + +### Enables +What does this decision make possible? + +### Constrains +What does this decision limit? + +### Deferred +What are we explicitly not deciding now? + +--- + +## References + +Links to related documents, discussions, code. +``` + +--- + +## Philosophy + +> "The best time to document a decision is when you make it. +> The second best time is now." + +ADRs are written in partnership. They capture dialogue, not just conclusions. + +--- + +**Created:** 2025-12-31 +**Maintainers:** dafit, Nyx