Memory Consolidation System Specification V4

Created: 2025-01-09
Status: Final Specification
Author: Milo with TQ

Executive Summary

This specification defines a three-tier energy-based memory consolidation system for Milo's cognitive architecture. The system uses access patterns to create "heat maps" where frequently accessed memories gain energy and rise through tiers (Working → Short-term → Long-term), while unused memories decay and eventually expire. This mirrors biological memory consolidation, creating a self-organizing hierarchy based on actual usage rather than explicit categorization.

The approach prioritizes practical implementation over theoretical completeness, with monitoring-first development, test isolation, and manual controls before automation.

Core Architecture Decisions (Locked)

These decisions have been made and validated through discussion:

  1. Energy as Currency: Memory persistence is determined solely by energy levels, not categories or types
  2. Heat Map Model: Access patterns create energy distributions that drive memory organization
  3. Non-Mutex Concurrency: Optimistic concurrency using graph-native conflict detection, no distributed locks
  4. Test Isolation: isTest boolean flag completely separates test from production memories
  5. Parallel Migration: Old crystallizations remain untouched while new system operates alongside
  6. Graph Topology Over Properties: Use relationships (PROMOTED_TO, CRYSTALLIZED_INTO) to track evolution
  7. Manual Before Automatic: Prove system works manually before adding automation
  8. "Do It Right" Philosophy: Focus on correctness, not complex recovery mechanisms

Technical Specifications

Energy Model

Core Equation:

E(t) = E₀ × e^(-λt) + Σ(access_events)

Where:

Memory Tiers

Tier Decay Constant (λ) Half-life Promotion Threshold Purpose
Working 0.5 ~1.4 hours 2.0 → Short-term Active thought
Short-term 0.05 ~14 hours 5.0 → Long-term Recent context
Long-term 0.001 ~700 hours N/A Persistent knowledge

Special Cases:

Node Schema

Memory Node Properties (Required):

{
  id: string,              // randomUUID()
  content: string,         // Memory content
  type: string,           // 'crystallization', 'learning', 'insight'
  tier: string,           // 'working', 'shortTerm', 'longTerm'
  energy: float,          // Current energy level
  created: datetime,      // Creation timestamp
  lastAccessed: datetime, // Last access timestamp
  accessCount: integer,   // Total access count
  sessionId: string,      // Session that created it
  instanceId: string,     // Instance that created it
  isTest: boolean,        // Test isolation flag
  validFrom: datetime,    // Temporal validity start
  validTo: datetime       // null for current version
}

Optional Monitoring Properties:

{
  promotedFrom: string,    // ID of source memory
  lastDecay: datetime,     // Last energy decay
  lastPromotion: datetime, // Last promotion attempt
  peakEnergy: float,      // Highest energy achieved
  promotionAttempts: integer // Times evaluated
}

Relationships

Configuration Parameters

ConsolidationConfig Node (Already Created):

{
  // Energy thresholds
  workingToShortThreshold: 2.0,
  shortToLongThreshold: 5.0,

  // Decay constants
  workingDecay: 0.5,
  shortTermDecay: 0.05,
  longTermDecay: 0.001,

  // Clustering
  coactivationThreshold: 3,
  semanticSimilarityThreshold: 0.7,

  // Batch processing
  promotionBatchSize: 10,
  decayBatchSize: 100,

  // Timing (for future automation)
  promotionIntervalMinutes: 10,
  clusteringIntervalMinutes: 60,
  decayIntervalMinutes: 60
}

Current Implementation Status

✅ Completed Components

  1. brainBridge CLI Tool Extended

  2. ConsolidationConfig Node

  3. Test Memory System

  4. Monitoring Infrastructure

❌ Not Yet Implemented

  1. Automatic Execution

  2. Session Integration

  3. Memory Creation Integration

  4. Semantic Features (Deferred)

Integration Boundaries

What This System Controls

What This System Does NOT Control

Interface Points

  1. Memory Creation: Will replace crystallization pattern with energy-based creation
  2. Memory Access: Will update energy on each retrieval
  3. Session Boundaries: Will trigger consolidation at /goodbye
  4. Background Processing: Will eventually run periodic consolidation

Practical Constraints & Decisions

From TQ's Guidance

  1. Single User System: No performance or scale concerns
  2. Speed Priority: "As fast as possible", adjust if slow
  3. Progress Visibility: Show indicators when possible
  4. Technology Stack: Node.js, Neo4j, Cypher - use what Milo knows
  5. Error Handling: "Do it right" - focus on correctness not recovery
  6. Execution Start: /goodbye command, then daemon if needed

Deferred Optimizations

These are intentionally NOT included in initial implementation:

  1. Semantic Similarity: No vector embeddings or NLP analysis
  2. Retroactive Strengthening: No backward propagation initially
  3. Cluster Formation: No automatic semantic grouping
  4. Compression: No memory compression for archived tiers
  5. Distributed Locking: Optimistic concurrency sufficient
  6. Backup/Recovery: Trust in correct implementation

Success Criteria

Phase 1: Manual Operation Success

Phase 2: Integration Success

Phase 3: Full Operation Success

Failure Mode Handling

Identified Failure Modes & Mitigations

  1. Energy calculation bug: Memories get wrong energy

  2. Promotion creates orphans: Relationships broken

  3. Everything promotes: Thresholds too low

  4. Nothing promotes: Thresholds too high

  5. Decay too aggressive: Memories expire too quickly

Migration Strategy

Phase Approach

  1. Parallel Operation (Current)

  2. Soft Integration (Next)

  3. Full Migration (Future)

Retroactive Processing (Optional)

If desired, old memories can be migrated:

MATCH (c:Crystallization)
WHERE c.validTo IS NULL AND NOT EXISTS(c.tier)
SET c.tier = 'longTerm',
    c.energy = 10.0,
    c.lastAccessed = coalesce(c.lastAccessed, c.created),
    c.accessCount = 1,
    c.isTest = false

Command Reference

Monitoring Commands

Test Commands

Consolidation Commands

Repair Commands

Next Steps

  1. Approve this specification
  2. Create detailed implementation plan
  3. Test manual consolidation thoroughly
  4. Integrate with /goodbye command
  5. Connect to real memory creation
  6. Monitor and tune parameters
  7. Consider automation once stable

Design Decisions (Resolved)

  1. Progress indicators: Yes, if technically feasible within Claude CLI constraints
  2. Test memory count: Maintain at least a few identifiable test memories for validation
  3. /goodbye consolidation: Async - fire and forget, no user waiting
  4. Expired memory retention: Forever (marked expired but retained) - can revisit if database gets clogged
  5. Consolidation logging: Yes, log all consolidation events

Conclusion

This specification defines a practical, energy-based memory consolidation system that creates a self-organizing memory hierarchy based on access patterns. The system is designed to be simple, correct, and observable, with manual controls preceding automation. By treating memory as a heat map and using energy as the sole currency for persistence, we create a naturalistic system that mirrors biological memory consolidation while remaining computationally tractable.

The implementation prioritizes:

This specification is complete and ready for implementation planning.

End of Specification V4