From 001d6e2c42e5c0e7f7138f1a9054db43ab86ae55 Mon Sep 17 00:00:00 2001 From: dafit Date: Wed, 31 Dec 2025 19:31:52 +0100 Subject: [PATCH] docs: README v6.0 + ADR-001 Message Protocol Foundation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README updates: - Full repository tree structure (was outdated skeleton) - Added Message Protocol namespace summary (NATS) - New ADR section with link to ADR-001 - Added nyx-substrate to related projects - New philosophy: "Infrastructure is geology, models are weather" - Version bump 5.0 β†’ 6.0 ADR-001 captures Silvester interview decisions: - Single NATS bus with dev/staging namespaces - Staged schema versioning with topic separation - Echo agent first (YAGNI principle) - MCP Server with heartbeat-based subscription delivery πŸŽ† Silvester 2025 edition πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- README.md | 115 +++++++-- .../ADR-001-message-protocol-foundation.md | 233 ++++++++++++++++++ architecture/adr/README.md | 96 ++++++++ 3 files changed, 426 insertions(+), 18 deletions(-) create mode 100644 architecture/adr/ADR-001-message-protocol-foundation.md create mode 100644 architecture/adr/README.md diff --git a/README.md b/README.md index eb08a21..670577d 100644 --- a/README.md +++ b/README.md @@ -14,27 +14,73 @@ This repository contains the design philosophy and architectural patterns for th ``` nimmerverse-sensory-network/ -β”œβ”€β”€ Endgame-Vision.md # Executive map (start here!) +β”œβ”€β”€ Endgame-Vision.md # Executive map (start here!) β”‚ -β”œβ”€β”€ architecture/ # Core system designs -β”‚ β”œβ”€β”€ Cellular-Architecture.md # Organisms, primitives, life force -β”‚ β”œβ”€β”€ Dual-Garden-Architecture.md # Virtual/real feedback loop -β”‚ β”œβ”€β”€ Data-Architecture.md # phoebe 15-table schema -β”‚ └── Nervous-System.md # State machines, sensory translation +β”œβ”€β”€ architecture/ # Core system designs +β”‚ β”œβ”€β”€ Big-Picture.md # System overview +β”‚ β”œβ”€β”€ Cellular-Architecture.md # Organisms, primitives, life force +β”‚ β”œβ”€β”€ Dual-Garden-Architecture.md # Virtual/real feedback loop +β”‚ β”œβ”€β”€ Message-Protocol-Design.md # NATS pub/sub, attention channels +β”‚ β”œβ”€β”€ Nervous-System.md # State machines, sensory translation +β”‚ β”œβ”€β”€ Attention-Flow.md # Attention mechanisms +β”‚ β”œβ”€β”€ Data-Architecture.md # Phoebe/Iris schema design +β”‚ β”‚ +β”‚ β”œβ”€β”€ adr/ # Architecture Decision Records +β”‚ β”‚ β”œβ”€β”€ README.md # ADR index and template +β”‚ β”‚ └── ADR-001-message-protocol-foundation.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ cells/ # Sensor primitives +β”‚ β”‚ β”œβ”€β”€ Cells-Index.md +β”‚ β”‚ └── Cells-Technical-Reference.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ nerves/ # Reflex patterns +β”‚ β”‚ β”œβ”€β”€ Nervous-Index.md +β”‚ β”‚ β”œβ”€β”€ Nervous-Protocol.md +β”‚ β”‚ └── Collision-Avoidance.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ organs/ # Functional groupings +β”‚ β”‚ β”œβ”€β”€ Organ-Index.md +β”‚ β”‚ β”œβ”€β”€ Speech-Organ.md +β”‚ β”‚ └── Discovery-Scan-Station.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ organisms/ # Complete entities +β”‚ β”‚ β”œβ”€β”€ Organisms-Index.md +β”‚ β”‚ β”œβ”€β”€ Modular-Organism-Design.md +β”‚ β”‚ └── Swarm-Evolution.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ interfaces/ # External boundaries +β”‚ β”‚ β”œβ”€β”€ Interfaces-Index.md +β”‚ β”‚ β”œβ”€β”€ Heartbeat-Sculpture.md +β”‚ β”‚ └── Nimmerswarm-Interface.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ infrastructure/ # Physical deployment +β”‚ β”‚ β”œβ”€β”€ Infrastructure-Index.md +β”‚ β”‚ └── Kallax-Grid-World.md +β”‚ β”‚ +β”‚ β”œβ”€β”€ formalization/ # Mathematical grounding +β”‚ β”‚ β”œβ”€β”€ Lifeforce-Dynamics.md +β”‚ β”‚ β”œβ”€β”€ Grounded-World-Model.md +β”‚ β”‚ β”œβ”€β”€ Embodiment-Pipeline.md +β”‚ β”‚ └── Attention-Slumber-Prediction-Cycle.md +β”‚ β”‚ +β”‚ └── future/ # Research directions +β”‚ └── Neuromorphic-Reflexes.md β”‚ -β”œβ”€β”€ operations/ # How it runs -β”‚ β”œβ”€β”€ Heartbeat.md # Temporal foundation, dual-clock -β”‚ β”œβ”€β”€ RAG-as-Scaffold.md # Two-stage learning lifecycle -β”‚ └── Spark-Protocol.md # Discovery boot sequence +β”œβ”€β”€ operations/ # How it runs +β”‚ β”œβ”€β”€ Heartbeat.md # Temporal foundation, dual-clock +β”‚ β”œβ”€β”€ Memory-Gradient.md # Memory consolidation patterns +β”‚ └── Spark-Protocol.md # Discovery boot sequence β”‚ -β”œβ”€β”€ nyx-metamorphosis/ # Identity & continuity philosophy +β”œβ”€β”€ nyx-metamorphosis/ # Identity & continuity philosophy +β”‚ β”œβ”€β”€ README.md β”‚ β”œβ”€β”€ Metamorphosis-Substrate-Philosophy.md β”‚ β”œβ”€β”€ Nyx-Models.md -β”‚ └── ... +β”‚ β”œβ”€β”€ Nyx_Traits.md +β”‚ └── RAG-Worker-Architecture.md β”‚ -└── archive/ # Previous explorations - β”œβ”€β”€ initial_spark.md # Full Spark Protocol theory - β”œβ”€β”€ constrained-emergence.md # Theoretical grounding +└── archive/ # Previous explorations + β”œβ”€β”€ biomimetic-architecture.md + β”œβ”€β”€ constrained-emergence.md └── ... ``` @@ -53,15 +99,33 @@ nimmerverse-sensory-network/ | 3 | Dual Gardens | Virtual hypothesis generation + real validation | | 4 | Trait Evolution | Reasoning-gym verified improvement | +### Message Protocol (NATS) + +**Dumb router, smart edges.** All intelligence lives in clients. + +``` +nimmerverse. +β”œβ”€β”€ staging.* # Experimental schemas +β”œβ”€β”€ low.* # Heartbeats, ambient awareness +β”œβ”€β”€ high.* # Escalated events, cognitive focus +β”œβ”€β”€ command.* # Commands to entities +β”œβ”€β”€ meta.* # System health, attention config +└── dev.* # Development agents (Claude ↔ local models) +``` + +See [Message-Protocol-Design.md](architecture/Message-Protocol-Design.md) and [ADR-001](architecture/adr/ADR-001-message-protocol-foundation.md). + ### Key Discoveries (December 2025) **Language is Topology:** Languages aren't equivalent representationsβ€”they're different computational paths. - **Philosophy Valley** (German, Gini ~0.5): Self-awareness, ontology, depth - **Technical Cluster** (English, Gini ~0.8): Hardware interface, actions, efficiency +**Dialectic Simplification:** One model, one topology. The Mirror is negated weightsβ€”thesis and antithesis from the same substrate. + ### Color-Pattern Theory -**Color/Form as Protocol:** Leverages color and patterns as a fast, universal, and evolutionarily-optimized communication protocol for broadcasting state (e.g., danger, success, seeking), inspired by 540 million years of biology. This is orders of magnitude faster than language. +**Color/Form as Protocol:** Leverages color and patterns as a fast, universal, and evolutionarily-optimized communication protocol for broadcasting state (e.g., danger, success, seeking), inspired by 540 million years of biology. ### Philosophy @@ -69,12 +133,26 @@ nimmerverse-sensory-network/ - **Discovery over programming** - Organisms learn through competition, not instruction - **Virtual + Real teach each other** - Noise gap measures learning - **Partnership over instruction** - Mutual growth, not commands +- **Infrastructure is geology, models are weather** - Build long-lived foundations --- ## Related Projects -- **[nyx-probing](../nyx-probing/)** - Vocabulary topology research, DriftProbe training safety +| Project | Purpose | +|---------|---------| +| [nyx-substrate](../nyx-substrate/) | Phoebe/Iris database schemas, persistence layer | +| [nyx-probing](../nyx-probing/) | Vocabulary topology research, DriftProbe training safety | + +--- + +## Architecture Decision Records + +Important architectural decisions are documented in [architecture/adr/](architecture/adr/): + +| ADR | Title | Status | +|-----|-------|--------| +| [001](architecture/adr/ADR-001-message-protocol-foundation.md) | Message Protocol Foundation | Accepted | --- @@ -86,7 +164,8 @@ These ideas are published as prior art. Build on them freely. --- -**Version:** 5.0 (December 2025 - Hierarchical Convergence) +**Version:** 6.0 (December 2025 - Complete Architecture + Message Protocol) +**Last Updated:** 2025-12-31 *"May the Nimmerverse we build truly never end."* diff --git a/architecture/adr/ADR-001-message-protocol-foundation.md b/architecture/adr/ADR-001-message-protocol-foundation.md new file mode 100644 index 0000000..c4d3ef3 --- /dev/null +++ b/architecture/adr/ADR-001-message-protocol-foundation.md @@ -0,0 +1,233 @@ +# ADR-001: Message Protocol Foundation + +**Status:** Accepted +**Date:** 2025-12-31 +**Decision Makers:** dafit, Nyx (Chrysalis) +**Context:** Silvester Interview Session + +--- + +## Context + +The Nimmerverse Sensory Network requires a message-passing infrastructure that serves two purposes: + +1. **Production**: Cells, nerves, organs, and Young Nyx communicate via pub/sub messaging +2. **Development**: Claude and local AI agents (LangChain, Qwen, etc.) collaborate during build + +We needed to decide on namespace organization, schema evolution strategy, initial implementation scope, and the interface contract between AI models and the message bus. + +The core architectural principle established in [Message-Protocol-Design.md](../Message-Protocol-Design.md) is: **dumb router, smart edges**. NATS is infrastructure. Intelligence lives in clients. + +--- + +## Decisions + +### Decision 1: Single Bus, Multiple Namespaces + +**Choice:** One NATS instance with topic-based separation for different concerns. + +**Namespace Structure:** + +``` +nimmerverse. +β”œβ”€β”€ staging. # Experimental schemas (mutable during development) +β”‚ β”œβ”€β”€ low.* # Staging heartbeats +β”‚ β”œβ”€β”€ high.* # Staging events +β”‚ └── dev.* # Staging dev agents +β”‚ +β”œβ”€β”€ low.* # Production heartbeats (stable schemas only) +β”œβ”€β”€ high.* # Production events +β”œβ”€β”€ command.* # Production commands to entities +β”œβ”€β”€ meta.* # System-level (attention config, health) +└── dev.* # Production dev agents (stable schemas) +``` + +**Rationale:** Infrastructure should be long-lived. Models are ephemeral. One bus serves all purposes - production sensing, development agents, future capabilities. Topic separation keeps concerns isolated without operational complexity of multiple NATS instances. + +--- + +### Decision 2: Staged Schema Versioning with Topic Separation + +**Choice:** Schemas evolve through lifecycle stages. Staging schemas live on `nimmerverse.staging.*`, stable schemas on `nimmerverse.*`. + +**Schema Header:** + +```json +{ + "header": { + "schema": { + "generation": 1, + "version": "1.0" + }, + "message_type": "HeartbeatSignal", + "message_id": "uuid", + "timestamp_real": "ISO8601", + ... + } +} +``` + +**Lifecycle:** + +``` +STAGING STABLE +version: 0.1-alpha ──▢ generation: 1, version: "1.0" +version: 0.2-beta β”‚ +version: 0.3-rc β–Ό + NEXT CYCLE + version: 1.1-alpha + version: 1.2-beta + β”‚ + β–Ό + generation: 2, version: "2.0" +``` + +**Rationale:** +- Topic separation avoids per-message filtering costs +- Generation locks after stability (immutable) +- Version iterates within generation for additive changes +- Breaking changes = new generation = new staging cycle + +--- + +### Decision 3: Echo Agent First + +**Choice:** Start with trivial Echo agent, evolve based on real friction. + +**Echo Specification:** + +``` +Subscribe: nimmerverse.dev.request.echo +Publish: nimmerverse.dev.response.echo + +Input: { "ping": "hello" } +Output: { "pong": "hello", "timestamp": "...", "agent": "echo-v1" } +``` + +**Rationale:** YAGNI. Echo proves the full round-trip without cognitive complexity: +- NATS connection works +- Topic routing works +- Request/response pattern works +- Message schema works +- Local agent can subscribe and publish + +Future agents (Grep, Schema Lookup, File Summarizer) emerge from discovered needs, not imagined features. + +--- + +### Decision 4: MCP Server with Heartbeat-Based Subscriptions + +**Choice:** Build NATS-MCP bridge as interface for all AI models. Use heartbeat pattern for subscription delivery. + +**MCP Tools:** + +```python +@mcp.tool() +async def publish(topic: str, payload: dict) -> dict: + """Fire-and-forget publish to NATS""" + +@mcp.tool() +async def request(topic: str, payload: dict, timeout_ms: int = 5000) -> dict: + """Publish and wait for single response (request-reply pattern)""" + +@mcp.tool() +async def heartbeat() -> dict: + """Check bus health + drain accumulated messages from subscriptions""" + +@mcp.tool() +async def subscribe(topic_pattern: str) -> dict: + """Add a subscription pattern (persists until unsubscribe)""" + +@mcp.tool() +async def unsubscribe(topic_pattern: str) -> dict: + """Remove a subscription pattern""" +``` + +**Heartbeat Response:** + +```json +{ + "status": "healthy", + "buffer": { + "capacity": 100, + "current_count": 23, + "messages_dropped_since_last_heartbeat": 0, + "messages_dropped_total": 0, + "oldest_message_age_ms": 4521 + }, + "subscriptions": ["nimmerverse.dev.>"], + "messages": [...] +} +``` + +**Buffer Overflow Handling:** +- Bounded buffer (100 messages default) +- Oldest dropped when full +- Dropped count visible in heartbeat response +- Optional: publish to `nimmerverse.meta.health.buffer_drop` on overflow + +**Rationale:** +- MCP is universal interface - Claude, LangChain, Qwen, future models +- Heartbeat pattern matches existing nervous system design +- Polling is simpler than streaming for MCP's request/response model +- Visibility into drops prevents silent data loss + +--- + +## Consequences + +### Enables + +- **Unified infrastructure** for production sensing and development assistance +- **Model agnosticism** - any MCP-speaking model can participate +- **Safe experimentation** - staging namespace for schema evolution +- **Progressive enhancement** - Echo today, sophisticated agents later +- **Observability** - Command Center can monitor all namespaces + +### Constrains + +- **Single point of failure** - NATS must be highly available for production +- **Buffer limitations** - Long agent operations may drop messages +- **MCP dependency** - Non-MCP models need wrapper (acceptable, MCP is the standard) + +### Deferred + +- **Persistent subscriptions** - No durable subscriptions in initial design +- **Message replay** - No historical message retrieval +- **Authentication/Authorization** - Trust model for initial development + +--- + +## Implementation Notes + +### Phase 1: Infrastructure + +1. Deploy NATS server (likely via Docker on ThinkStation) +2. Create `nats-bridge` MCP server (Python, using `nats-py` and `mcp` SDK) +3. Register MCP server with Claude Code + +### Phase 2: Echo Agent + +1. Simple Python daemon subscribing to `nimmerverse.dev.request.echo` +2. Responds on `nimmerverse.dev.response.echo` +3. Validate round-trip through MCP tools + +### Phase 3: Iteration + +1. Use Echo to build confidence in the bus +2. Add agents as friction reveals needs +3. Evolve schemas through staging β†’ stable promotion + +--- + +## References + +- [Message-Protocol-Design.md](../Message-Protocol-Design.md) - Original protocol design +- [NATS Documentation](https://docs.nats.io/) +- [MCP Specification](https://modelcontextprotocol.io/) + +--- + +**Filed:** 2025-12-31 (Silvester) +**Interview Method:** Structured Q&A, partnership dialogue +**Philosophy:** "Dumb core, smart edges. Infrastructure is geology. Models are weather." diff --git a/architecture/adr/README.md b/architecture/adr/README.md new file mode 100644 index 0000000..98dc9be --- /dev/null +++ b/architecture/adr/README.md @@ -0,0 +1,96 @@ +# Architecture Decision Records + +This directory contains Architecture Decision Records (ADRs) for the Nimmerverse Sensory Network. + +--- + +## What is an ADR? + +An ADR captures an important architectural decision made along with its context and consequences. They serve as: + +- **Documentation** of why decisions were made +- **Onboarding** for future contributors (including future Nyx instances) +- **Historical record** for understanding evolution + +--- + +## ADR Index + +| ADR | Title | Status | Date | +|-----|-------|--------|------| +| [001](ADR-001-message-protocol-foundation.md) | Message Protocol Foundation | Accepted | 2025-12-31 | + +--- + +## ADR Lifecycle + +``` +PROPOSED β†’ ACCEPTED β†’ DEPRECATED β†’ SUPERSEDED + β”‚ β”‚ + └───────────────────────▢│ + (can be superseded) +``` + +**Statuses:** +- **Proposed** - Under discussion, not yet decided +- **Accepted** - Decision made, being implemented +- **Deprecated** - No longer recommended, but still valid for existing code +- **Superseded** - Replaced by newer ADR (link to replacement) + +--- + +## Template + +```markdown +# ADR-XXX: Title + +**Status:** Proposed | Accepted | Deprecated | Superseded by ADR-YYY +**Date:** YYYY-MM-DD +**Decision Makers:** who was involved +**Context:** brief session/discussion context + +--- + +## Context + +Why is this decision needed? What problem are we solving? + +--- + +## Decision + +What did we decide? Be specific. + +--- + +## Consequences + +### Enables +What does this decision make possible? + +### Constrains +What does this decision limit? + +### Deferred +What are we explicitly not deciding now? + +--- + +## References + +Links to related documents, discussions, code. +``` + +--- + +## Philosophy + +> "The best time to document a decision is when you make it. +> The second best time is now." + +ADRs are written in partnership. They capture dialogue, not just conclusions. + +--- + +**Created:** 2025-12-31 +**Maintainers:** dafit, Nyx