HyperAI Plugin System Architecture

Table of Contents

• Overview
• Design Goals
• Architecture Decision: Hybrid Approach
• Plugin Manifest Structure
• Communication Protocol
• Plugin Manager Implementation
• Uniform UI System
• Settings Storage and Isolation
• Development Workflow
• Language Examples
• Security Considerations
• Installation Process
• Benefits
• Implementation Roadmap
• Potential Issues and Challenges

Overview

The HyperAI Plugin System is designed to allow users to extend the application's functionality through downloadable, self-contained plugins. The system prioritizes uniform UI, settings isolation, runtime independence, and security.

Key Requirements

• Uniform UI: All plugins must have consistent appearance and behavior
• Settings Isolation: Plugins can only access their own settings
• Self-Contained: No dependency on user's local environment
• Downloadable: Plugins distributed as packages, not compiled with app
• Multi-Language: Support any programming language
• Secure: Sandboxed execution with controlled permissions

Architectural Constraints

• On-Demand Loading: Plugins only start when user accesses their functionality
• Single Plugin UI: Settings displayed for one plugin at a time
• Resource Efficiency: Maximum 1-2 plugins active simultaneously
• Fast Startup: HyperAI starts immediately, plugins load as needed

Backward Compatibility Strategy

This new executable-based plugin system is designed to coexist with the existing plugin infrastructure. The implementation follows a dual-plugin approach:

Current Plugin System (Preserved)

• Existing functionality remains unchanged: All current plugins continue to work exactly as they do today
• No breaking changes: Current plugin API, interfaces, and behaviors are maintained
• Gradual migration path: Existing plugins can be migrated over time, not all at once

New Executable Plugin System (Added)

• Separate plugin category: New plugins are clearly distinguished from existing ones
• Independent management: New plugins have their own installation, configuration, and lifecycle
• Enhanced capabilities: Supports complex scenarios like MAME integration that weren't possible before

Hybrid Architecture Benefits

interface PluginRegistry {
  // Existing plugin system
  legacyPlugins: LegacyPlugin[];

  // New executable plugin system
  executablePlugins: ExecutablePlugin[];

  // Unified management interface
  getAllPlugins(): Plugin[];
  getPluginsByCategory(category: string): Plugin[];
}

Migration Timeline

1. Phase 1: Implement new executable plugin system alongside existing system
2. Phase 2: New complex plugins (like MAME) use executable system exclusively
3. Phase 3: Gradually migrate existing plugins to new system (optional, based on need)
4. Phase 4: Eventually deprecate legacy system (long-term, only if beneficial)

This approach ensures:

• ✅ Zero disruption to current users and workflows
• ✅ Immediate benefits from new plugin capabilities
• ✅ Flexible migration at your own pace
• ✅ Future-ready architecture for complex integrations

Design Goals

1. Zero Dependencies: Plugins must work on any machine regardless of installed software
2. Language Agnostic: Support Python, C#, Go, Rust, JavaScript, etc.
3. Consistent UX: All plugins use the same UI templates and patterns
4. Security: Process isolation and controlled API access
5. Easy Distribution: Simple zip-based plugin packages
6. Settings Safety: Plugins cannot interfere with each other's settings
7. Real-time Communication: WebSocket support for always-running applications and event-driven responses

Architecture Decision: Hybrid Approach

After evaluating multiple approaches (JavaScript-only, executable-only, runtime detection), we chose the hybrid approach that provides the best of both worlds:

Why Hybrid Architecture?

• ✅ 80/20 Rule: Most plugins (80%) are simple and benefit from JavaScript simplicity
• ✅ Complex Capabilities: Complex plugins (20%) get full executable power
• ✅ Developer Friendly: Simple plugins have zero build complexity
• ✅ Maximum Flexibility: Any language support for complex scenarios
• ✅ Gradual Complexity: Start simple, upgrade to executable when needed
• ✅ Better Security: JavaScript plugins run in secure V8 contexts

Hybrid Plugin Architecture Details

Plugin Type Decision Matrix

Plugin Complexity	Recommended Type	Communication	Examples	Benefits
Simple Utilities	JavaScript	Direct API	LED control, ROM organizer, theme manager	Easy development, fast startup, better security
System Integration	Executable	Socket.IO	MAME setup, emulator configuration	Full system access, any language, maximum flexibility
API Integrations	JavaScript	Direct API	EmuMovies sync, metadata fetcher, cloud backup	Network access, JSON handling, async operations
Complex Processing	Executable	Socket.IO	ROM analysis, image processing, file conversion	Performance critical, native libraries
Always-Running Services	JavaScript/Executable	Socket.IO	Hardware monitoring, real-time LED control, background sync	Real-time bidirectional communication, persistent connection

JavaScript Plugin Benefits

✅ No Cross-Platform Compilation - Single .js file works everywhere ✅ Zero Build Complexity - Edit and test immediately ✅ Better Security - V8 context isolation ✅ Instant Startup - <1ms load time vs 10-50ms for executables ✅ Hot Reload - Update plugins without restart

Executable Plugin Benefits

✅ Full System Integration - Direct hardware/OS access ✅ Any Programming Language - C++, Rust, Go, Python, etc. ✅ Maximum Performance - Native code execution ✅ Existing Libraries - Use any native library

WebSocket Communication Benefits

✅ Real-time Bidirectional - Instant communication both ways ✅ Event-Driven Architecture - React to HyperAI events immediately ✅ Always-Running Services - Persistent background applications ✅ Low Latency - <10ms response time for critical events ✅ Connection Recovery - Automatic reconnection on disconnect ✅ Standard Protocol - Works with any language/framework

Development Guidelines

Choose JavaScript When:

• Simple data processing, API calls, UI manipulation
• Rapid development and iteration needed
• Cross-platform compatibility is priority
• Maximum security/sandboxing desired
• Examples: Theme managers, ROM organizers, metadata fetchers, simple LED control

Choose Executable When:

• Direct hardware/OS access required
• Performance-critical operations
• Need to use specific native libraries
• Complex system setup/configuration needed
• Examples: MAME integration, hardware drivers, image/video processing, emulator configuration

Communication Method Guidelines

Use Direct API (JavaScript plugins):

✅ Best for: Simple, on-demand operations ✅ Lifecycle: Plugin starts when settings opened, stops when closed ✅ Examples: ROM organizers, theme managers, one-time data processing

Use Socket.IO (Executable plugins):

✅ Best for: All executable plugins - real-time, bidirectional communication ✅ Lifecycle: Plugin stays running, continuously communicates with HyperHQ ✅ Examples: MAME setup wizards, file conversion tools, system configuration, hardware monitoring, real-time LED control, background sync services

// WebSocket use cases
interface WebSocketUseCases {
  hardwareMonitoring: {
    description: "Monitor CPU, GPU, temperature in real-time";
    events: ["system_load", "temperature_alert"];
    bidirectional: true;
  };

  realTimeLEDControl: {
    description: "Instant LED responses to game events";
    events: ["game:launched", "game:exited", "game:selected"];
    latency: "<10ms";
  };

  backgroundServices: {
    description: "Continuous sync with external services";
    events: ["rom_added", "metadata_updated"];
    persistent: true;
  };
}

Plugin Manifest Structure

Every plugin includes a plugin.json manifest file supporting both JavaScript and executable plugins:

interface PluginManifest {
  // Core Identity
  id: string;                    // Unique plugin identifier
  name: string;                  // Display name
  version: string;               // Semantic version
  description: string;           // Brief description
  author: string;                // Plugin author
  type: 'javascript' | 'executable';  // Plugin type

  // JavaScript Plugin Specific (when type === 'javascript')
  main?: string;                 // "plugin.js" - main JavaScript file
  sandbox?: {
    networkAccess?: boolean;     // Can make network requests
    fileSystemAccess?: string[]; // File access permissions
    apis?: string[];             // Allowed HyperAI APIs
  };

  // Executable Plugin Specific (when type === 'executable')
  executable?: string;           // "plugin.exe", "plugin", "plugin.bat"
  args?: string[];              // Optional startup arguments
  permissions?: {
    filesystem: string[];        // Allowed file paths
    processLaunching: boolean;   // Can spawn processes
    networkAccess: boolean;      // Can access network
  };

  // WebSocket Configuration (optional for both types)
  websocket?: {
    enabled: boolean;            // Enable WebSocket communication
    autoReconnect?: boolean;     // Automatic reconnection on disconnect
    heartbeat?: number;          // Heartbeat interval in milliseconds
    events?: string[];           // Events plugin wants to subscribe to
  };

  // UI Definition (common for both types)
  ui: {
    displayName: string;         // User-facing name
    icon?: string;              // Icon filename in plugin package
    category: 'arcade' | 'emulator' | 'utility' | 'media';
    settings: PluginSettingDefinition[];
  };

  // Optional Metadata
  requiresRestart?: boolean;     // Restart needed after settings change
  autoStart?: boolean;          // Start with HyperAI
}

interface PluginSettingDefinition {
  key: string;                  // Setting identifier
  name: string;                 // Display name
  type: 'text' | 'boolean' | 'select' | 'file' | 'directory' | 'number';
  defaultValue: any;            // Default value
  description?: string;         // Help text
  required?: boolean;           // Is required
  options?: string[];           // For select type
  validation?: {
    pattern?: string;           // Regex pattern
    min?: number;              // Min value/length
    max?: number;              // Max value/length
  };
}

Example Manifests

JavaScript Plugin Example (Recommended for simple plugins)

{
  "id": "ledblinky-control",
  "name": "LEDBlinky Control",
  "version": "1.0.0",
  "description": "Control LEDBlinky LED lighting for arcade cabinets",
  "author": "HyperAI Team",
  "type": "javascript",
  "main": "plugin.js",
  "sandbox": {
    "networkAccess": false,
    "fileSystemAccess": ["read-only"],
    "apis": ["hyperai.data", "hyperai.settings"]
  },
  "ui": {
    "displayName": "LEDBlinky LED Control",
    "icon": "led-icon.png",
    "category": "arcade",
    "settings": [
      {
        "key": "ledBlinkyPath",
        "name": "LEDBlinky Executable Path",
        "type": "file",
        "defaultValue": "",
        "description": "Path to LEDBlinky.exe",
        "required": true,
        "validation": {
          "pattern": ".*\\.exe$"
        }
      },
      {
        "key": "enableLed",
        "name": "Enable LED Control",
        "type": "boolean",
        "defaultValue": false
      }
    ]
  }
}

Executable Plugin Example (For complex system integration)

{
  "id": "mame-integration",
  "name": "MAME Integration Helper",
  "version": "1.0.0",
  "description": "Complete MAME setup and configuration assistant",
  "author": "HyperAI Team",
  "type": "executable",
  "executable": "plugin.exe",
  "permissions": {
    "filesystem": ["%PLUGIN_DATA%", "%TEMP%", "%MAME_PATH%"],
    "processLaunching": true,
    "networkAccess": true
  },
  "ui": {
    "displayName": "MAME Integration",
    "icon": "mame-icon.png",
    "category": "emulator",
    "settings": [
      {
        "key": "mamePath",
        "name": "MAME Installation Path",
        "type": "directory",
        "required": true
      },
      {
        "key": "autoDownloadRoms",
        "name": "Auto-download ROM List",
        "type": "boolean",
        "defaultValue": true
      }
    ]
  }
}

WebSocket Plugin Example (For always-running applications)

{
  "id": "hardware-monitor",
  "name": "Hardware Monitor",
  "version": "1.0.0",
  "description": "Real-time hardware monitoring and control",
  "author": "HyperAI Team",
  "type": "executable",
  "executable": "hardware-monitor.exe",
  "websocket": {
    "enabled": true,
    "autoReconnect": true,
    "heartbeat": 30000,
    "events": [
      "game:launched",
      "game:exited",
      "system:startup",
      "system:shutdown",
      "game:selected"
    ]
  },
  "permissions": {
    "filesystem": ["%PLUGIN_DATA%"],
    "processLaunching": false,
    "networkAccess": true
  },
  "ui": {
    "displayName": "Hardware Monitor",
    "icon": "hardware-icon.png",
    "category": "utility",
    "settings": [
      {
        "key": "monitoringEnabled",
        "name": "Enable Real-time Monitoring",
        "type": "boolean",
        "defaultValue": true
      },
      {
        "key": "updateInterval",
        "name": "Update Interval (ms)",
        "type": "number",
        "defaultValue": 1000,
        "validation": { "min": 100, "max": 10000 }
      }
    ]
  }
}

Communication Protocol

JavaScript Plugins

JavaScript plugins communicate with HyperAI through direct API calls in the secure V8 context:

// JavaScript plugin communication - direct API access
class MyPlugin {
  async initialize(data) {
    // Direct API access
    this.settings = await hyperai.setting.getAll();
    return "ok";
  }

  async execute(data) {
    // Request data from HyperAI
    const systems = await hyperai.data.request('getSystems', {});

    // Update UI component
    hyperai.ui.updateComponent('system-list', { systems });

    return true;
  }
}

Executable Plugins (Socket.IO Communication)

All executable plugins communicate with HyperHQ via Socket.IO for real-time, bidirectional communication:

// Socket.IO plugin communication - for all executable plugins
const io = require('socket.io-client');

class ExecutablePlugin {
  constructor() {
    this.socket = io('http://localhost:52789/plugin', {
      reconnection: true,
      reconnectionAttempts: 10
    });
    this.setupSocketIO();
  }

  setupSocketIO() {
    this.socket.on('connect', () => {
      // Authenticate with HyperHQ
      this.socket.emit('authenticate', {
        pluginId: process.env.HYPERHQ_PLUGIN_ID,
        challenge: process.env.HYPERHQ_AUTH_CHALLENGE
      });
    });

    this.socket.on('authenticated', (response) => {
      if (response.success) {
        this.sessionToken = response.sessionToken;
        this.register();
      }
    });

    this.socket.on('request', async (message) => {
      const result = await this.handleRequest(message);
      this.socket.emit('plugin:response', {
        id: message.id,
        type: 'response',
        data: result,
        sessionToken: this.sessionToken
      });
    });

    this.socket.on('game:launched', (data) => {
      this.onGameLaunched(data);
    });

    this.socket.on('disconnect', () => {
      // Automatic reconnection is handled by Socket.IO
      console.log('Disconnected, will attempt to reconnect...');
    });
  }

  register() {
    this.socket.emit('plugin:register', {
      id: process.env.HYPERHQ_PLUGIN_ID,
      version: '1.0.0',
      capabilities: ['socketio']
    });
  }
}

Socket.IO Server Implementation

HyperHQ runs a local Socket.IO server (localhost:52789) for real-time plugin communication:

// Socket.IO server in HyperHQ
import { Server } from 'socket.io';

class PluginSocketIOServer {
  private io: Server;
  private authenticatedPlugins = new Map<string, any>();

  constructor() {
    this.io = new Server(52789, {
      cors: {
        origin: 'http://localhost',
        methods: ['GET', 'POST']
      }
    });

    const pluginNamespace = this.io.of('/plugin');
    pluginNamespace.on('connection', this.handleConnection.bind(this));
  }

  private handleConnection(socket: any) {
    console.log('Plugin attempting to connect');

    socket.on('authenticate', (data: any) => {
      this.handleAuthentication(socket, data);
    });

    socket.on('disconnect', () => {
      const pluginId = this.getPluginIdFromSocket(socket);
      this.authenticatedPlugins.delete(pluginId);
      console.log(`Plugin ${pluginId} disconnected`);
    });
  }

  private handleAuthentication(socket: any, data: any) {
    // Validate authentication challenge
    const pluginId = data.pluginId;
    const challenge = data.challenge;

    if (this.validateChallenge(pluginId, challenge)) {
      const sessionToken = this.generateSessionToken(pluginId);
      this.authenticatedPlugins.set(pluginId, socket);

      socket.emit('authenticated', {
        success: true,
        sessionToken: sessionToken
      });
    } else {
      socket.emit('authenticated', {
        success: false,
        error: 'Invalid authentication challenge'
      });
    }
  }

  // Broadcast events to subscribed plugins
  public broadcastEvent(eventType: string, data: any) {
    this.authenticatedPlugins.forEach((socket, pluginId) => {
      if (this.isPluginSubscribedToEvent(pluginId, eventType)) {
        socket.emit(eventType, data);
      }
    });
  }
}

Socket.IO Security & Authentication

// Plugin authentication for Socket.IO connections
interface SocketIOSecurity {
  // Authentication challenge generated when plugin starts
  authChallenge: string;      // One-time challenge for authentication
  sessionToken: string;       // Session token after successful authentication

  // Local-only connections
  allowedOrigins: ['localhost', '127.0.0.1'];

  // Message validation
  messageSchema: JSONSchema;  // Validate all incoming messages

  // Rate limiting
  maxMessagesPerSecond: 100;  // Prevent spam

  // Connection limits
  maxConnectionsPerPlugin: 1; // One connection per plugin
}

Message Format

interface PluginMessage {
  id: string;                   // Request correlation ID
  type: 'request' | 'response' | 'event' | 'error';
  method?: string;              // For requests: 'initialize', 'execute', etc.
  data?: any;                   // Message payload
  error?: string;               // Error description if type is 'error'
}

Required Plugin Methods

All plugins must implement these methods:

// Initialize plugin
{"id": "1", "type": "request", "method": "initialize", "data": {
  "pluginId": "ledblinky-control",
  "settings": {"enableLed": true, "ledBlinkyPath": "C:\\LEDBlinky\\LEDBlinky.exe"},
  "hyperaiVersion": "1.0.0",
  "pluginDataPath": "C:\\Users\\...\\HyperAI\\plugins\\ledblinky-control\\data"
}}
// Response: {"id": "1", "type": "response", "data": "ok"}

// Execute action
{"id": "2", "type": "request", "method": "execute", "data": {
  "action": "game_start",
  "params": {"rom": {"name": "pacman"}, "emulator": {"name": "mame"}}
}}
// Response: {"id": "2", "type": "response", "data": true}

// Test plugin functionality
{"id": "3", "type": "request", "method": "test", "data": {}}
// Response: {"id": "3", "type": "response", "data": true}

// Shutdown plugin
{"id": "4", "type": "request", "method": "shutdown", "data": {}}
// Response: {"id": "4", "type": "response", "data": "ok"}

Complete Communication Flow Example

Here's a detailed example showing how the plugin-HyperAI communication works in practice, demonstrating the data mediation architecture:

// 1. UI Component needs data (e.g., MAME versions for a dropdown)
// HyperAI automatically sends data request to plugin when component loads

// 2. Plugin receives data request from HyperAI
const dataRequestHandler = (message: PluginMessage) => {
  if (message.type === 'data-request' && message.method === 'getMameVersions') {
    // Plugin doesn't call APIs directly - asks HyperAI for the data
    return {
      id: message.id,
      type: 'data-response',
      data: {
        needsData: true,
        hyperaiMethod: 'getMameVersions',  // Tell HyperAI what method to call
        params: { limit: 10, includePrerelease: false }
      }
    };
  }
};

// 3. HyperAI receives the request and decides how to handle it
class PluginDataRequestHandler {
  async handleRequest(pluginId: string, request: any) {
    if (request.hyperaiMethod === 'getMameVersions') {
      // HyperAI decides: check cache first, then external API if needed
      let versions = this.cacheService.get('mame-versions');

      if (!versions) {
        // HyperAI makes the external API call (plugin doesn't need network permissions)
        versions = await this.githubService.getMameReleases();
        this.cacheService.set('mame-versions', versions, 3600); // Cache for 1 hour
      }

      // Send data back to plugin
      this.sendToPlugin(pluginId, {
        id: request.id,
        type: 'data-response',
        data: versions
      });
    } else if (request.hyperaiMethod === 'scanRomDirectory') {
      // Local data - serve directly from filesystem
      const romFiles = await this.fileSystemService.scanDirectory(request.params.path);

      this.sendToPlugin(pluginId, {
        id: request.id,
        type: 'data-response',
        data: {
          files: romFiles,
          count: romFiles.length,
          scannedAt: new Date().toISOString()
        }
      });
    } else if (request.hyperaiMethod === 'createEmulator') {
      // Database operation - check permissions first
      this.validatePluginPermissions(pluginId, 'database.write.emulators');

      const emulatorId = await this.emulatorService.create(request.params.emulator);

      this.sendToPlugin(pluginId, {
        id: request.id,
        type: 'data-response',
        data: { emulatorId, success: true }
      });
    }
  }
}

// 4. Plugin receives data and forwards to UI or processes it
const dataResponseHandler = (message: PluginMessage) => {
  if (message.type === 'data-response') {
    const data = message.data;

    // Update UI component with new data
    this.sendToHyperAI({
      type: 'ui-data-update',
      componentId: 'mame-version-selector',
      data: {
        options: data.map(version => ({
          value: version.tag_name,
          label: `${version.name} (${version.tag_name})`,
          description: version.body?.substring(0, 100)
        }))
      }
    });

    // Or process the data for plugin logic
    this.availableVersions = data;
    this.updateInstallationOptions();
  }
};

// 5. Real-world example: MAME Plugin requesting multiple data types
class MamePlugin {
  async setupInstallation() {
    // Request MAME versions from GitHub (external)
    const versionsRequest = {
      id: 'mame-versions-1',
      type: 'data-request',
      hyperaiMethod: 'getMameVersions',
      params: { includePrerelease: false }
    };

    // Request existing systems (local database)
    const systemsRequest = {
      id: 'systems-1',
      type: 'data-request',
      hyperaiMethod: 'getSystems',
      params: { category: 'arcade' }
    };

    // Request user's ROM directories (local filesystem)
    const romDirsRequest = {
      id: 'rom-dirs-1',
      type: 'data-request',
      hyperaiMethod: 'getRomDirectories',
      params: {}
    };

    // Send all requests - HyperAI handles them efficiently
    this.sendRequests([versionsRequest, systemsRequest, romDirsRequest]);
  }

  onDataReceived(message: PluginMessage) {
    switch (message.id) {
      case 'mame-versions-1':
        this.updateVersionSelector(message.data);
        break;
      case 'systems-1':
        this.checkExistingMameSystems(message.data);
        break;
      case 'rom-dirs-1':
        this.suggestRomPath(message.data);
        break;
    }
  }
}

Benefits of This Communication Flow

✅ Local Data Optimization: HyperAI can serve local data (systems, ROMs, settings) instantly without any network calls

✅ Smart Caching: HyperAI handles caching decisions - external data like MAME versions cached locally, local data served directly

✅ Security: Plugin doesn't need network permissions - HyperAI handles all external communication

✅ Performance: No unnecessary API layers - direct access to local databases and filesystem

✅ Consistent Error Handling: HyperAI can provide consistent error handling and retry logic

✅ Permission Control: HyperAI validates all data requests against plugin permissions before serving data

This architecture ensures plugins get the data they need efficiently while maintaining security and performance, regardless of whether the data is local or requires external API calls.

Data Request Limitations and Safeguards

Potential Problems with Large Data Requests

While the data mediation architecture provides many benefits, plugins requesting excessive amounts of data can cause several issues:

1. Memory Exhaustion

// ❌ Problematic: Plugin requests all ROMs at once
const badRequest = {
  hyperaiMethod: 'getAllRoms',
  params: {} // Could return 50,000+ ROMs = 500MB+ of data
};

// ✅ Better: Plugin requests paginated data
const goodRequest = {
  hyperaiMethod: 'getRoms',
  params: {
    limit: 100,
    offset: 0,
    system: 'arcade'
  }
};

2. UI Freezing and Performance Issues

• Large datasets can freeze UI components during loading
• Database queries for massive datasets can be slow
• JSON serialization/deserialization becomes expensive

3. Resource Exhaustion Attacks

• Malicious plugins could overwhelm HyperAI with massive data requests
• Poorly written plugins might request the same large dataset repeatedly

Implemented Safeguards

1. Request Size Limits

class PluginDataRequestValidator {
  private readonly MAX_RECORDS_PER_REQUEST = 1000;
  private readonly MAX_RESPONSE_SIZE_MB = 50;

  validateRequest(pluginId: string, request: PluginDataRequest): ValidationResult {
    // Enforce record limits
    if (request.params?.limit > this.MAX_RECORDS_PER_REQUEST) {
      return {
        valid: false,
        error: `Request limit exceeds maximum (${this.MAX_RECORDS_PER_REQUEST} records)`
      };
    }

    // Estimate response size for known methods
    const estimatedSize = this.estimateResponseSize(request);
    if (estimatedSize > this.MAX_RESPONSE_SIZE_MB * 1024 * 1024) {
      return {
        valid: false,
        error: `Estimated response size (${estimatedSize}MB) exceeds limit`
      };
    }

    return { valid: true };
  }

  private estimateResponseSize(request: PluginDataRequest): number {
    // Estimate based on method and parameters
    switch (request.hyperaiMethod) {
      case 'getRoms':
        const limit = request.params?.limit || 100;
        return limit * 2048; // ~2KB per ROM record
      case 'getSystems':
        return 50 * 1024; // ~1KB per system, max 50 systems
      default:
        return 1024; // 1KB default estimate
    }
  }
}

2. Automatic Pagination

class PluginDataService {
  async handleRomRequest(pluginId: string, request: any): Promise<any> {
    const limit = Math.min(request.params?.limit || 100, 1000);
    const offset = request.params?.offset || 0;

    // Always use pagination internally
    const roms = await this.romService.getRoms({
      limit,
      offset,
      system: request.params.system,
      filters: request.params.filters
    });

    // Include pagination metadata
    const totalCount = await this.romService.getRomCount(request.params);

    return {
      data: roms,
      pagination: {
        limit,
        offset,
        total: totalCount,
        hasMore: (offset + limit) < totalCount,
        nextOffset: (offset + limit) < totalCount ? offset + limit : null
      }
    };
  }
}

3. Rate Limiting

class PluginRequestRateLimiter {
  private requestCounts = new Map<string, number[]>();
  private readonly MAX_REQUESTS_PER_MINUTE = 60;
  private readonly MAX_LARGE_REQUESTS_PER_MINUTE = 10;

  checkRateLimit(pluginId: string, requestSize: 'small' | 'large'): boolean {
    const now = Date.now();
    const oneMinuteAgo = now - 60000;

    // Clean old requests
    const requests = this.requestCounts.get(pluginId) || [];
    const recentRequests = requests.filter(time => time > oneMinuteAgo);

    // Check limits based on request size
    const limit = requestSize === 'large'
      ? this.MAX_LARGE_REQUESTS_PER_MINUTE
      : this.MAX_REQUESTS_PER_MINUTE;

    if (recentRequests.length >= limit) {
      return false; // Rate limit exceeded
    }

    // Record this request
    recentRequests.push(now);
    this.requestCounts.set(pluginId, recentRequests);
    return true;
  }
}

4. Request Timeouts and Cancellation

class PluginCommunicationService {
  private readonly REQUEST_TIMEOUT_MS = 30000; // 30 seconds

  async sendDataRequest(pluginId: string, request: PluginDataRequest): Promise<any> {
    const timeoutPromise = new Promise((_, reject) => {
      setTimeout(() => reject(new Error('Request timeout')), this.REQUEST_TIMEOUT_MS);
    });

    const requestPromise = this.processDataRequest(pluginId, request);

    try {
      return await Promise.race([requestPromise, timeoutPromise]);
    } catch (error) {
      // Cancel the request if it's still running
      this.cancelRequest(pluginId, request.id);
      throw error;
    }
  }
}

Plugin-Side Best Practices

1. Implement Pagination in Plugin UI

// Plugin manifest with paginated component
{
  "id": "rom-browser",
  "type": "data-table",
  "label": "ROM Browser",
  "dataSource": {
    "type": "hyperai-data",
    "method": "getRoms",
    "params": {
      "limit": 100,
      "system": "{{selectedSystem}}"
    },
    "pagination": true
  },
  "props": {
    "columns": ["name", "system", "year"],
    "pageSize": 100,
    "virtualScrolling": true
  }
}

2. Stream Large Operations

# Plugin Python example with progress reporting
def handle_large_operation(self, data):
    total_items = data.get('totalItems', 0)
    batch_size = 100

    for offset in range(0, total_items, batch_size):
        # Request data in batches
        batch_request = {
            'hyperaiMethod': 'getRoms',
            'params': {
                'limit': batch_size,
                'offset': offset,
                'system': data.get('system')
            }
        }

        batch_data = self.request_data(batch_request)
        self.process_batch(batch_data)

        # Report progress
        progress = min(100, int(((offset + batch_size) / total_items) * 100))
        self.send_progress_update(progress, f"Processed {offset + batch_size}/{total_items} items")

User Experience Improvements

1. Progressive Loading

// UI shows data as it loads, not all at once
class ProgressiveDataLoader {
  async loadLargeDataset(componentId: string, dataRequest: any) {
    const pageSize = 100;
    let offset = 0;
    let hasMore = true;

    // Show loading indicator
    this.updateComponent(componentId, { loading: true, data: [] });

    while (hasMore) {
      const batch = await this.requestBatch(dataRequest, offset, pageSize);

      // Append new data to existing
      this.updateComponent(componentId, {
        loading: false,
        data: [...existingData, ...batch.data],
        hasMore: batch.pagination.hasMore
      });

      offset += pageSize;
      hasMore = batch.pagination.hasMore;

      // Small delay to prevent UI freezing
      await this.delay(10);
    }
  }
}

2. Virtual Scrolling for Large Lists

// Large ROM lists use virtual scrolling
{
  "id": "large-rom-list",
  "type": "virtual-list",
  "label": "All ROMs",
  "props": {
    "itemHeight": 40,
    "visibleItems": 20,
    "totalItems": "{{romCount}}",
    "loadMoreThreshold": 5
  },
  "dataSource": {
    "type": "hyperai-data",
    "method": "getRoms",
    "pagination": true,
    "pageSize": 100
  }
}

Monitoring and Alerting

class PluginResourceMonitor {
  monitorDataRequests(pluginId: string, requestSize: number, responseTime: number) {
    // Log large requests
    if (requestSize > 10 * 1024 * 1024) { // 10MB
      this.logger.warn(`Plugin ${pluginId} requested large dataset: ${requestSize}MB`);
    }

    // Log slow requests
    if (responseTime > 5000) { // 5 seconds
      this.logger.warn(`Plugin ${pluginId} slow data request: ${responseTime}ms`);
    }

    // Update plugin resource usage stats
    this.updatePluginStats(pluginId, {
      dataRequested: requestSize,
      avgResponseTime: responseTime
    });
  }
}

These safeguards ensure that plugins can access the data they need while preventing resource exhaustion, maintaining good performance, and providing a smooth user experience even with large datasets.

Specific Example: Plugin Displaying All Systems in Dropdown

Here's a step-by-step flow showing how a plugin gets systems data for a dropdown component:

// 1. Plugin defines UI with systems dropdown in plugin.json
{
  "ui": {
    "type": "simple",
    "layout": {
      "components": [
        {
          "id": "system-selector",
          "type": "select-dropdown",
          "label": "Select System",
          "props": {
            "placeholder": "Choose a system...",
            "searchable": true
          },
          "dataSource": {
            "type": "hyperai-data",
            "method": "getAllSystems",
            "params": {},
            "cache": true
          },
          "validation": "required"
        }
      ]
    }
  }
}

// 2. HyperAI loads plugin UI and sees dropdown needs system data
// HyperAI automatically sends data request to plugin
→ HyperAI sends to Plugin: {
  "id": "ui-data-req-1",
  "type": "data-request",
  "method": "getAllSystems",
  "componentId": "system-selector",
  "params": {}
}

// 3. Plugin receives request and asks HyperAI for the actual data
← Plugin responds to HyperAI: {
  "id": "ui-data-req-1",
  "type": "data-response",
  "data": {
    "needsData": true,
    "hyperaiMethod": "getSystems",  // Tell HyperAI what data to fetch
    "params": {
      "includeHidden": false,
      "sortBy": "name"
    }
  }
}

// 4. HyperAI receives request and serves data from local database
// No external API call needed - systems are stored locally
class HyperAIDataService {
  async handlePluginDataRequest(pluginId: string, request: any) {
    if (request.hyperaiMethod === 'getSystems') {
      // Get systems directly from local database
      const systems = await this.systemService.getAllSystems({
        includeHidden: request.params.includeHidden,
        sortBy: request.params.sortBy
      });

      // Transform for dropdown use
      const dropdownOptions = systems.map(system => ({
        value: system.id,
        label: system.name,
        description: system.description,
        icon: system.iconPath,
        metadata: {
          romCount: system.romCount,
          emulator: system.defaultEmulator
        }
      }));

      // Send data back to plugin
      this.sendToPlugin(pluginId, {
        id: request.id,
        type: 'data-response',
        data: dropdownOptions
      });
    }
  }
}

// 5. Plugin receives system data and forwards to UI
← HyperAI sends to Plugin: {
  "id": "ui-data-req-1",
  "type": "data-response",
  "data": [
    {
      "value": "arcade",
      "label": "Arcade",
      "description": "MAME Arcade Games",
      "icon": "/systems/arcade.png",
      "metadata": {
        "romCount": 2847,
        "emulator": "mame"
      }
    },
    {
      "value": "genesis",
      "label": "Sega Genesis",
      "description": "16-bit Sega console",
      "icon": "/systems/genesis.png",
      "metadata": {
        "romCount": 156,
        "emulator": "retroarch"
      }
    }
    // ... more systems
  ]
}

// 6. Plugin forwards data to HyperAI UI system
→ Plugin sends to HyperAI: {
  "type": "ui-data-update",
  "componentId": "system-selector",
  "data": {
    "options": message.data,  // The dropdown options
    "loaded": true,
    "error": null
  }
}

// 7. HyperAI updates the dropdown component in real-time
// User sees populated dropdown with all systems, icons, and metadata
// Dropdown shows: "Arcade (2847 ROMs)", "Genesis (156 ROMs)", etc.

What Happens Behind the Scenes

HyperAI Side:

class PluginUIRenderer {
  async renderDropdownComponent(component: PluginUIComponent) {
    if (component.dataSource?.type === 'hyperai-data') {
      // Automatically request data when component loads
      await this.requestComponentData(component);
    }
  }

  private async requestComponentData(component: PluginUIComponent) {
    const request = {
      id: uuid(),
      type: 'data-request',
      method: component.dataSource.method,
      componentId: component.id,
      params: component.dataSource.params || {}
    };

    // Send to plugin and wait for response
    const response = await this.pluginCommunication.sendMessage(
      this.pluginId,
      request
    );

    // Update component when data arrives
    this.updateComponentData(component.id, response.data);
  }
}

Plugin Side (Python example):

def handle_data_request(self, message):
    """Handle data requests from HyperAI UI components"""
    method = message.get('method')
    component_id = message.get('componentId')

    if method == 'getAllSystems':
        # Plugin doesn't have direct access to systems
        # Ask HyperAI to provide the data
        return {
            'id': message['id'],
            'type': 'data-response',
            'data': {
                'needsData': True,
                'hyperaiMethod': 'getSystems',
                'params': {
                    'includeHidden': False,
                    'sortBy': 'name'
                }
            }
        }

def handle_data_response(self, message):
    """Process data received from HyperAI"""
    if message.get('hyperaiMethod') == 'getSystems':
        systems_data = message['data']

        # Plugin can process/filter the data if needed
        filtered_systems = self.filter_systems_for_plugin(systems_data)

        # Send to UI component
        self.send_ui_update('system-selector', {
            'options': filtered_systems,
            'loaded': True
        })

Key Benefits Demonstrated

✅ Instant Local Data: Systems served immediately from HyperAI's database - no loading delays

✅ Automatic UI Population: Developer just defines the dropdown, HyperAI handles all the data loading

✅ Rich Metadata: Each option includes ROM counts, emulator info, icons automatically

✅ No Database Access: Plugin doesn't need database permissions - HyperAI mediates safely

✅ Consistent Formatting: All system dropdowns across all plugins look and behave the same

✅ Real-time Updates: If systems are added/removed in HyperAI, dropdown updates automatically

This flow shows why the data mediation approach is so powerful - the plugin gets rich, up-to-date system data without any direct database access or complex API calls!

Systems Dropdown Interaction Diagram

Key Points from the Diagram:

1. Developer Simplicity: Plugin developer only defines the dropdown in JSON - no data loading code
2. Automatic Detection: HyperAI UI automatically detects when components need data
3. Data Mediation: Plugin never accesses database directly - HyperAI serves all data
4. Rich Metadata: Database query includes ROM counts, emulator info, icons automatically
5. Instant Updates: Local data served immediately, dropdown populates in real-time

The diagram shows why this architecture is so powerful - complex data operations happen automatically with minimal developer effort!

Plugin Manager Implementation

@Injectable({ providedIn: 'root' })
export class ExecutablePluginManager {
  private runningPlugins = new Map<string, PluginProcess>();

  async loadPlugin(pluginId: string): Promise<LoadedPlugin> {
    const manifest = await this.loadPluginManifest(pluginId);
    const process = await this.startPluginExecutable(manifest);

    // Initialize plugin
    await this.sendMessage(process, {
      id: uuid(),
      type: 'request',
      method: 'initialize',
      data: {
        pluginId,
        settings: await this.getPluginSettings(pluginId),
        hyperaiVersion: await this.getAppVersion(),
        pluginDataPath: this.getPluginDataPath(pluginId)
      }
    });

    const plugin = new LoadedPlugin(manifest, process);
    this.runningPlugins.set(pluginId, process);
    return plugin;
  }

  private async startPluginExecutable(manifest: PluginManifest): Promise<PluginProcess> {
    const pluginPath = this.getPluginPath(manifest.id);
    const executablePath = path.join(pluginPath, manifest.executable);

    // Generate authentication challenge for this plugin instance
    const authChallenge = this.generateAuthChallenge(manifest.id);

    const args = manifest.args || [];
    const childProcess = spawn(executablePath, args, {
      cwd: pluginPath,
      env: {
        ...process.env,
        HYPERHQ_PLUGIN_ID: manifest.id,
        HYPERHQ_PLUGIN_NAME: manifest.name,
        HYPERHQ_PLUGIN_VERSION: manifest.version,
        HYPERHQ_SOCKET_PORT: this.socketIOPort.toString(),
        HYPERHQ_AUTH_CHALLENGE: authChallenge,
        PLUGIN_DATA_PATH: this.getPluginDataPath(manifest.id)
      }
    });

    return new PluginProcess(childProcess, manifest.id);
  }
}

Dynamic UI System with Component Library

The plugin system generates UI dynamically from JSON manifests using a library of reusable components that maintain HyperAI's design language:

Component Library Architecture

// Base component interface for all plugin UI components
interface PluginUIComponent {
  id: string;
  type: PluginComponentType;
  label: string;
  validation?: PluginValidation;
  dataSource?: PluginDataSource;  // For dynamic data loading
  props: Record<string, any>;
  events?: PluginEventHandler[];
}

// Available component types in the library
type PluginComponentType =
  | 'text-input'          // Basic text input
  | 'file-picker'         // File selection with browse button
  | 'directory-picker'    // Folder selection
  | 'dropdown'            // Single selection dropdown
  | 'multi-select'        // Multiple selection
  | 'toggle'              // Boolean switch
  | 'slider'              // Numeric range selector
  | 'progress-bar'        // Progress indication
  | 'version-selector'    // Specialized for version selection
  | 'system-selector'     // Choose from available systems
  | 'rom-path-selector'   // ROM directory with validation
  | 'filter-config'       // Complex filter configuration
  | 'step-wizard'         // Multi-step wizard container
  | 'data-table'          // Tabular data display
  | 'status-display';     // Real-time status information

// Dynamic data sources for populating components
interface PluginDataSource {
  type: 'hyperai-data' | 'static' | 'computed';
  method: string;              // HyperAI method to call (e.g., 'getMameVersions', 'scanRomDirectory')
  params?: Record<string, any>; // Parameters to pass to HyperAI
  transform?: string;          // Data transformation expression
  cache?: boolean;             // Cache results in HyperAI
  refreshInterval?: number;    // Auto-refresh in seconds
}

// UI layout definitions
interface PluginUIDefinition {
  type: 'simple' | 'wizard' | 'dashboard';
  theme: 'default' | 'dark' | 'light';
  layout: PluginLayout;
  components: PluginUIComponent[];
  dataBindings?: PluginDataBinding[];
}

Plugin-HyperAI Communication Layer

Instead of plugins making API calls, HyperAI provides data services directly through the plugin communication channel:

// HyperAI Data Service - handles all data requests for plugins
@Injectable()
export class PluginDataService {

  // Local data (no external calls needed)
  async getSystems(): Promise<SystemInfo[]> {
    return await window.api.mainDb.systems.getAll();
  }

  async getEmulators(): Promise<EmulatorInfo[]> {
    return await window.api.mainDb.emulators.getAll();
  }

  async scanRomDirectory(path: string, filters?: ScanFilters): Promise<RomFile[]> {
    // Direct filesystem access - no API needed
    return await window.api.fileSystem.scanDirectory(path, filters);
  }

  async validateRomPath(path: string): Promise<ValidationResult> {
    // Local validation logic
    const exists = await window.api.fileSystem.exists(path);
    const hasRoms = exists ? await this.countRomFiles(path) : 0;
    return { valid: exists && hasRoms > 0, romCount: hasRoms };
  }

  // External data (HyperAI handles the external calls)
  async getMameVersions(): Promise<MameVersion[]> {
    // HyperAI makes the GitHub API call, handles caching, rate limiting
    return await this.externalDataService.fetchMameVersions();
  }

  // Database operations (controlled by permissions)
  async createSystem(pluginId: string, system: SystemData): Promise<string> {
    // Validate plugin permissions first
    this.validatePluginPermissions(pluginId, 'database.write.systems');

    // Create system using existing HyperAI logic
    return await window.api.mainDb.systems.add(system);
  }

  async importRoms(pluginId: string, roms: RomData[], systemId: string): Promise<ImportResult> {
    // Use existing ROM import service
    return await this.romImportService.bulkImport(roms, systemId);
  }
}

// Plugin Communication Service - handles message passing
@Injectable()
export class PluginCommunicationService {

  // Send data request to plugin via Socket.IO
  async requestData<T>(pluginId: string, method: string, params?: any): Promise<T> {
    const requestId = uuid();
    const message = {
      id: requestId,
      type: 'data-request',
      method,
      params
    };

    // Send to plugin via Socket.IO
    const pluginSocket = this.getPluginSocket(pluginId);
    pluginSocket.emit('request', message);

    // Wait for response
    return new Promise((resolve, reject) => {
      const timeout = setTimeout(() => reject(new Error('Request timeout')), 30000);

      const messageHandler = (response: any) => {
        if (response.id === requestId) {
          clearTimeout(timeout);
          pluginSocket.off('plugin:response', messageHandler);

          if (response.type === 'response') {
            resolve(response.data);
          } else if (response.type === 'error') {
            reject(new Error(response.error));
          }
        }
      };

      pluginSocket.on('plugin:response', messageHandler);
    });
  }

  // Handle data requests FROM plugins
  async handlePluginDataRequest(pluginId: string, request: PluginDataRequest): Promise<any> {
    try {
      // Route to appropriate data service based on method
      switch (request.method) {
        case 'getSystems':
          return await this.pluginDataService.getSystems();
        case 'getMameVersions':
          return await this.pluginDataService.getMameVersions();
        case 'scanRomDirectory':
          return await this.pluginDataService.scanRomDirectory(request.params.path, request.params.filters);
        case 'createSystem':
          return await this.pluginDataService.createSystem(pluginId, request.params.system);
        case 'importRoms':
          return await this.pluginDataService.importRoms(pluginId, request.params.roms, request.params.systemId);
        default:
          throw new Error(`Unknown data method: ${request.method}`);
      }
    } catch (error) {
      throw new Error(`Plugin data request failed: ${error.message}`);
    }
  }
}

Dynamic Component Examples

// MAME Version Selector - requests data from HyperAI
const mameVersionSelector: PluginUIComponent = {
  id: 'mame-version',
  type: 'version-selector',
  label: 'MAME Version',
  dataSource: {
    type: 'hyperai-data',
    method: 'getMameVersions',  // HyperAI method to call
    cache: true,
    transform: 'versions.map(v => ({label: v.name, value: v.tag, size: v.size}))'
  },
  props: {
    placeholder: 'Select MAME version...',
    searchable: true,
    showSize: true  // Show download size
  },
  validation: {
    required: true,
    errorMessage: 'Please select a MAME version'
  },
  events: [
    {
      type: 'change',
      action: 'hyperai-request',
      method: 'getMameDownloadInfo',
      data: '{{selectedValue}}',
      updateComponent: 'download-info-display'
    }
  ]
};

// ROM Path Selector with live validation
const romPathSelector: PluginUIComponent = {
  id: 'rom-path',
  type: 'rom-path-selector',
  label: 'ROM Directory',
  validation: {
    required: true,
    async: {
      method: 'validateRomPath',  // HyperAI validation method
      errorMessage: 'Directory must exist and contain ROM files'
    }
  },
  events: [
    {
      type: 'change',
      action: 'hyperai-request',
      method: 'scanRomDirectory',
      data: '{{value}}',
      updateComponent: 'rom-count-display'
    }
  ]
};

// Progress display with real-time updates via HyperAI
const progressDisplay: PluginUIComponent = {
  id: 'installation-progress',
  type: 'progress-bar',
  label: 'Installation Progress',
  dataSource: {
    type: 'hyperai-data',
    method: 'getInstallationProgress',
    refreshInterval: 1000
  },
  props: {
    showPercentage: true,
    showETA: true,
    animated: true
  }
};

Communication Flow Example

Here's how the plugin-HyperAI communication works:

// 1. UI Component needs data (e.g., MAME versions)
// HyperAI automatically sends data request to plugin when component loads

// 2. Plugin receives data request from HyperAI
const dataRequestHandler = (message: PluginMessage) => {
  if (message.type === 'data-request' && message.method === 'getMameVersions') {
    // Plugin asks HyperAI for the data
    return {
      id: message.id,
      type: 'data-response',
      data: {
        needsData: true,
        hyperaiMethod: 'getMameVersions',  // Tell HyperAI what method to call
        params: {}
      }
    };
  }
};

// 3. HyperAI receives the request and fetches data
class PluginDataRequestHandler {
  async handleRequest(pluginId: string, request: any) {
    if (request.hyperaiMethod === 'getMameVersions') {
      // HyperAI decides: local cache, external API call, etc.
      const versions = await this.getMameVersionsFromGitHub();

      // Send data back to plugin
      this.sendToPlugin(pluginId, {
        id: request.id,
        type: 'data-response',
        data: versions
      });
    }
  }
}

// 4. Plugin receives data and forwards to UI
const dataResponseHandler = (message: PluginMessage) => {
  if (message.type === 'data-response') {
    // Forward to HyperAI UI system
    this.sendToHyperAI({
      type: 'ui-data-update',
      componentId: 'mame-version',
      data: message.data
    });
  }
};

Benefits of This Approach

✅ Local Data Optimization: HyperAI can serve local data (systems, ROMs, settings) instantly without any network calls

✅ Smart Caching: HyperAI handles caching decisions - external data like MAME versions cached locally, local data served directly

✅ Security: Plugin doesn't need network permissions - HyperAI handles all external communication

✅ Performance: No unnecessary API layers - direct access to local databases and filesystem

✅ Consistent Error Handling: HyperAI can provide consistent error handling and retry logic

✅ Permission Control: HyperAI validates all data requests against plugin permissions before serving data

Multi-Step Wizard Definition

{
  "ui": {
    "type": "wizard",
    "theme": "default",
    "layout": {
      "steps": [
        {
          "id": "version-selection",
          "title": "Select MAME Version",
          "description": "Choose which version of MAME to install",
          "components": ["mame-version", "install-options"],
          "validation": "required"
        },
        {
          "id": "rom-configuration",
          "title": "ROM Setup",
          "description": "Configure ROM directory and import filters",
          "components": ["rom-path", "rom-filters", "rom-preview"],
          "validation": "rom-path-valid"
        },
        {
          "id": "installation",
          "title": "Installing MAME",
          "description": "Please wait while MAME is downloaded and configured",
          "components": ["installation-progress", "installation-log"],
          "canGoBack": false
        }
      ]
    }
  }
}

Dynamic UI Renderer Service

@Injectable()
export class PluginUIRenderer {
  constructor(
    private componentFactory: ComponentFactoryResolver,
    private pluginAPI: PluginAPI,
    private pluginComm: PluginCommunicationService,
    private componentLibrary: PluginComponentLibrary
  ) {}

  async renderPluginUI(
    container: ViewContainerRef,
    manifest: PluginManifest,
    pluginId: string
  ): Promise<PluginUIInstance> {

    const uiDefinition = manifest.ui;

    // Apply theme
    this.applyTheme(container, uiDefinition.theme);

    // Render based on UI type
    switch (uiDefinition.type) {
      case 'wizard':
        return this.renderWizard(container, uiDefinition, pluginId);
      case 'simple':
        return this.renderSimpleForm(container, uiDefinition, pluginId);
      case 'dashboard':
        return this.renderDashboard(container, uiDefinition, pluginId);
    }
  }

  private async renderComponent(
    component: PluginUIComponent,
    context: PluginUIContext
  ): Promise<ComponentRef<any>> {

    // Load dynamic data if component has data source
    if (component.dataSource) {
      await this.loadComponentData(component, context);
    }

    // Get component from library
    const componentClass = this.componentLibrary.getComponent(component.type);
    if (!componentClass) {
      throw new Error(`Unknown component type: ${component.type}`);
    }

    // Create component instance
    const factory = this.componentFactory.resolveComponentFactory(componentClass);
    const componentRef = factory.create(context.injector);

    // Apply properties and bind data
    this.bindComponentProperties(componentRef, component, context);

    // Wire up events (API calls, SignalR, component updates)
    this.wireComponentEvents(componentRef, component.events, context);

    return componentRef;
  }
}

This approach gives you:

• ✅ Truly dynamic UI generated from JSON manifests
• ✅ Consistent design language across all plugins
• ✅ Rich component library for complex scenarios
• ✅ Real-time data binding via API + SignalR
• ✅ Complex workflows like multi-step wizards
• ✅ Live validation and updates as users interact

Settings Storage and Isolation

Plugin settings are stored in HyperAI's main database for optimal performance and accessibility, using a namespaced approach for complete isolation.

Database Storage Benefits

Why Database Over Local Files:

• Performance: Instant access for HyperAI when processing plugin requests
• Query Capabilities: Can filter/search plugin settings for admin features
• Consistency: All HyperAI data in one place, same backup/restore process
• Reliability: Database transactions ensure setting updates are atomic
• Cross-Plugin Features: Enable future features that work across plugins

Settings Storage Architecture

// Plugin settings stored in HyperAI's main database
// Using existing settings table with namespaced keys for isolation

@Injectable()
export class PluginSettingsService {
  private settingsCache = new Map<string, Map<string, any>>();

  async getPluginSettings(pluginId: string): Promise<{ [key: string]: any }> {
    // Check cache first for performance
    if (this.settingsCache.has(pluginId)) {
      return Object.fromEntries(this.settingsCache.get(pluginId)!);
    }

    // Load from database with plugin namespace
    const settingPrefix = `Plugin_${pluginId}_`;
    const allSettings = await window.api.mainDb.settings.getAllSettings();

    const pluginSettings = allSettings
      .filter(s => s.name.startsWith(settingPrefix))
      .reduce((acc, setting) => {
        const key = setting.name.substring(settingPrefix.length);
        acc[key] = this.deserializeValue(setting.value, setting.type);
        return acc;
      }, {} as { [key: string]: any });

    // Cache for subsequent requests
    this.settingsCache.set(pluginId, new Map(Object.entries(pluginSettings)));

    return pluginSettings;
  }

    async savePluginSetting(pluginId: string, key: string, value: any): Promise<void> {
    // SECURITY: Always prefix with plugin ID - no exceptions
    const settingName = `Plugin_${pluginId}_${key}`;

    // SECURITY: Double-check plugin isolation
    if (!this.validatePluginAccess(pluginId, settingName)) {
      throw new Error(`Security violation: Plugin '${pluginId}' attempted to access unauthorized setting '${settingName}'`);
    }

    // SECURITY: Sanitize the key to prevent injection
    if (!this.isValidSettingKey(key)) {
      throw new Error(`Invalid setting key: '${key}'. Keys must be alphanumeric with underscores/dashes only.`);
    }

    const setting: ISettingEntity = {
      name: settingName,
      value: this.serializeValue(value),
      type: typeof value,
      default: '',
      id: undefined,
      createdDate: new Date(),
      modifiedDate: new Date()
    };

    await window.api.mainDb.settings.updateSetting(setting);

    // Update cache
    this.invalidatePluginCache(pluginId);

    // SECURITY: Log setting access for audit trail
    this.logSettingAccess(pluginId, key, 'write');
  }

  async getPluginSetting(pluginId: string, key: string): Promise<any> {
    // SECURITY: Validate key format before database access
    if (!this.isValidSettingKey(key)) {
      throw new Error(`Invalid setting key: '${key}'`);
    }

    // SECURITY: Always use namespaced access - never direct key access
    const settingName = `Plugin_${pluginId}_${key}`;

    // SECURITY: Verify plugin can only access its own namespace
    if (!this.validatePluginAccess(pluginId, settingName)) {
      this.logSecurityViolation(pluginId, key, 'unauthorized_read_attempt');
      return null; // Return null instead of throwing to prevent plugin crashes
    }

    const settings = await this.getPluginSettings(pluginId);
    const value = settings[key] ?? null;

    // SECURITY: Log setting access for audit trail
    this.logSettingAccess(pluginId, key, 'read');

    return value;
  }

  // SECURITY: Strict validation - only the exact plugin ID prefix is allowed
  private validatePluginAccess(pluginId: string, settingName: string): boolean {
    const expectedPrefix = `Plugin_${pluginId}_`;
    return settingName === expectedPrefix + settingName.substring(expectedPrefix.length) &&
           settingName.startsWith(expectedPrefix);
  }

  // SECURITY: Validate setting key format to prevent injection attacks
  private isValidSettingKey(key: string): boolean {
    // Only alphanumeric, underscores, and dashes allowed
    const validKeyPattern = /^[a-zA-Z0-9_-]+$/;
    return validKeyPattern.test(key) && key.length > 0 && key.length <= 100;
  }

  // SECURITY: Comprehensive access logging for audit purposes
  private logSettingAccess(pluginId: string, key: string, operation: 'read' | 'write'): void {
    const logEntry = {
      timestamp: new Date().toISOString(),
      pluginId,
      settingKey: key,
      operation,
      fullSettingName: `Plugin_${pluginId}_${key}`
    };

    // Log to console for development, database for production
    console.log(`Plugin Setting Access: ${JSON.stringify(logEntry)}`);
  }

  // SECURITY: Log security violations for monitoring
  private logSecurityViolation(pluginId: string, attemptedKey: string, violationType: string): void {
    const violation = {
      timestamp: new Date().toISOString(),
      pluginId,
      attemptedKey,
      violationType,
      severity: 'HIGH'
    };

    console.error(`SECURITY VIOLATION: ${JSON.stringify(violation)}`);
    // In production, this should also alert administrators
  }

  private serializeValue(value: any): string {
    return typeof value === 'string' ? value : JSON.stringify(value);
  }

  private deserializeValue(value: string, type?: string): any {
    if (type === 'boolean') return value === 'true';
    if (type === 'number') return parseFloat(value);
    if (type === 'object') {
      try { return JSON.parse(value); }
      catch { return value; }
    }
    return value;
  }

  private invalidatePluginCache(pluginId: string): void {
    this.settingsCache.delete(pluginId);
  }
}

Settings Performance Optimization

class PluginSettingsOptimizer {
  // Batch load settings for multiple plugins
  async preloadPluginSettings(pluginIds: string[]): Promise<void> {
    const prefixes = pluginIds.map(id => `Plugin_${id}_`);
    const allSettings = await window.api.mainDb.settings.getAllSettings();

    // Group settings by plugin
    const settingsByPlugin = new Map<string, Map<string, any>>();

    allSettings.forEach(setting => {
      const matchingPrefix = prefixes.find(prefix => setting.name.startsWith(prefix));
      if (matchingPrefix) {
        const pluginId = matchingPrefix.substring(7, matchingPrefix.length - 1); // Remove "Plugin_" and "_"
        const key = setting.name.substring(matchingPrefix.length);

        if (!settingsByPlugin.has(pluginId)) {
          settingsByPlugin.set(pluginId, new Map());
        }
        settingsByPlugin.get(pluginId)!.set(key, this.deserializeValue(setting.value, setting.type));
      }
    });

    // Update cache
    settingsByPlugin.forEach((settings, pluginId) => {
      this.settingsCache.set(pluginId, settings);
    });
  }

  // Fast access for frequently used settings
  async getFastSetting(pluginId: string, key: string): Promise<any> {
    if (this.settingsCache.has(pluginId)) {
      return this.settingsCache.get(pluginId)!.get(key) ?? null;
    }

    // Direct database query for single setting
    const settingName = `Plugin_${pluginId}_${key}`;
    const setting = await window.api.mainDb.settings.getSetting(settingName);
    return setting ? this.deserializeValue(setting.value, setting.type) : null;
  }
}

Strict Security Isolation Guarantees

CRITICAL: Plugin settings are ALWAYS prefixed with the plugin ID. No exceptions.

// SECURITY RULE: All plugin settings MUST follow this pattern:
// Database Key Format: Plugin_{pluginId}_{settingKey}

// Examples of properly isolated settings:
Plugin_ledblinky-control_enableLed = "true" (type: boolean)
Plugin_ledblinky-control_ledBlinkyPath = "C:\LEDBlinky\LEDBlinky.exe" (type: string)
Plugin_ledblinky-control_scanInterval = "300" (type: number)
Plugin_ledblinky-control_gameConfigs = '{"pacman":{"led":"blue"},"galaga":{"led":"red"}}' (type: object)

Plugin_mame-helper_romPath = "C:\MAME\roms" (type: string)
Plugin_mame-helper_autoScan = "false" (type: boolean)

Plugin_rom-organizer_sortBy = "name" (type: string)
Plugin_rom-organizer_backupEnabled = "true" (type: boolean)

// IMPOSSIBLE SCENARIOS - Security system prevents these:
// ❌ Plugin "mame-helper" CANNOT access Plugin_ledblinky-control_enableLed
// ❌ Plugin "ledblinky-control" CANNOT access Plugin_mame-helper_romPath
// ❌ Plugin "rom-organizer" CANNOT access any other plugin's settings
// ❌ No plugin can access HyperAI's core settings (no "Plugin_" prefix)

Security Enforcement Mechanisms

class PluginSettingSecurityEnforcer {
  // 1. MANDATORY PREFIXING
  // Every setting save/retrieve automatically prefixes with plugin ID
  savePluginSetting(pluginId: "mame-helper", key: "romPath", value: "/path/to/roms")
  // → Stores as: "Plugin_mame-helper_romPath"

  // 2. ACCESS VALIDATION
  // Plugin can only access settings that start with its exact prefix
  validatePluginAccess("mame-helper", "Plugin_mame-helper_romPath") // ✅ ALLOWED
  validatePluginAccess("mame-helper", "Plugin_ledblinky-control_enableLed") // ❌ BLOCKED
  validatePluginAccess("mame-helper", "coreHyperAISetting") // ❌ BLOCKED

  // 3. KEY SANITIZATION
  // Only safe characters allowed in setting keys
  isValidSettingKey("romPath") // ✅ Valid
  isValidSettingKey("rom_path-v2") // ✅ Valid
  isValidSettingKey("rom.path") // ❌ Invalid (dots not allowed)
  isValidSettingKey("rom path") // ❌ Invalid (spaces not allowed)
  isValidSettingKey("../../../secret") // ❌ Invalid (path traversal blocked)

  // 4. AUDIT LOGGING
  // All setting access is logged for security monitoring
  logSettingAccess("mame-helper", "romPath", "read")
  logSecurityViolation("malicious-plugin", "../../etc/passwd", "unauthorized_access_attempt")
}

Plugin Developer Perspective (Secure & Simple)

# Plugin developers use simple keys - security happens automatically
class MyPlugin:
    def save_user_preference(self, theme_color):
        # Developer writes simple code
        self.save_setting('themeColor', theme_color)
        # HyperAI automatically stores as: Plugin_my-plugin-id_themeColor

    def load_configuration(self):
        # Developer requests by simple key
        rom_path = self.get_setting('romPath')
        # HyperAI automatically looks up: Plugin_my-plugin-id_romPath

        # Impossible scenarios (these will return None/null):
        other_plugin_data = self.get_setting('Plugin_other-plugin_secretKey') # ❌ Blocked
        system_setting = self.get_setting('hyperaiAdminPassword') # ❌ Blocked

Cross-Plugin Access Prevention

// Example: Plugin "ledblinky-control" tries to access MAME settings
const maliciousPlugin = "ledblinky-control";

// All these attempts are automatically blocked:
❌ getPluginSetting(maliciousPlugin, "Plugin_mame-helper_romPath")
   → Returns: null (access denied)
   → Logs: SECURITY VIOLATION

❌ getPluginSetting(maliciousPlugin, "../mame-helper/romPath")
   → Returns: null (invalid key format)
   → Logs: SECURITY VIOLATION

❌ getPluginSetting(maliciousPlugin, "mame-helper_romPath")
   → Returns: null (wrong prefix)
   → Logs: SECURITY VIOLATION

✅ getPluginSetting(maliciousPlugin, "enableLed")
   → Returns: value (accessing own setting Plugin_ledblinky-control_enableLed)
   → Logs: Normal access

The system guarantees that plugins can ONLY access settings prefixed with their exact plugin ID. This isolation is enforced at every level - database queries, cache access, and API calls.

Plugin Developer Experience

# Plugin developer sees simple API (Python example)
class MyPlugin:
    def initialize(self, data):
        self.settings = data.get('settings', {})

        # Access settings by simple names
        self.led_path = self.settings.get('ledBlinkyPath', '')
        self.enable_led = self.settings.get('enableLed', False)
        self.scan_interval = self.settings.get('scanInterval', 300)

        return "ok"

    def get_setting(self, key):
        # Request setting from HyperAI
        request = {
            'hyperaiMethod': 'getSetting',
            'params': {'key': key}
        }
        response = self.send_request(request)
        return response.get('data')

    def save_setting(self, key, value):
        # Save setting through HyperAI
        request = {
            'hyperaiMethod': 'setSetting',
            'params': {'key': key, 'value': value}
        }
        return self.send_request(request)

This database-centric approach provides optimal performance for HyperAI while maintaining complete plugin isolation and a simple development experience.

Development Workflow

1. Create Plugin Package Structure

JavaScript Plugin Structure (Recommended for simple plugins)

my-simple-plugin/
├── plugin.json          # Manifest (type: "javascript")
├── plugin.js           # Main JavaScript file
├── icon.png            # Plugin icon
├── README.md           # Documentation
└── LICENSE             # License file

Executable Plugin Structure (For complex system integration)

my-complex-plugin/
├── plugin.json          # Manifest (type: "executable")
├── plugin.exe          # Self-contained executable
├── icon.png            # Plugin icon
├── README.md           # Documentation
└── LICENSE             # License file

2. Plugin Development Process

JavaScript Plugin Development (Zero Build Required)

# Create JavaScript plugin - no build step needed!
mkdir my-simple-plugin && cd my-simple-plugin

# Create manifest
cat > plugin.json << 'EOF'
{
  "id": "my-simple-plugin",
  "name": "My Simple Plugin",
  "version": "1.0.0",
  "type": "javascript",
  "main": "plugin.js",
  "ui": { "displayName": "My Plugin", "category": "utility", "settings": [] }
}
EOF

# Write plugin logic
cat > plugin.js << 'EOF'
class MyPlugin {
  async initialize(data) {
    this.settings = data.settings || {};
    return "ok";
  }

  async execute(data) {
    // Plugin logic here
    return true;
  }

  async test(data) {
    return true;
  }
}
globalThis.plugin = new MyPlugin();
EOF

# Package for distribution (just zip the folder!)
cd .. && zip -r my-simple-plugin.zip my-simple-plugin/

WebSocket Plugin Development (Real-time Applications)

# hardware_monitor.py - Always-running hardware monitor
import websocket
import json
import psutil
import threading
import time

class HardwareMonitor:
    def __init__(self):
        self.ws = None
        self.running = False
        self.plugin_id = "hardware-monitor"

    def connect(self):
        self.ws = websocket.WebSocketApp(
            f"ws://localhost:52789/plugin/{self.plugin_id}",
            on_open=self.on_open,
            on_message=self.on_message,
            on_error=self.on_error,
            on_close=self.on_close
        )

    def on_open(self, ws):
        # Authenticate with HyperAI
        auth_message = {
            "type": "authenticate",
            "data": {
                "pluginId": self.plugin_id,
                "token": self.get_auth_token()
            }
        }
        ws.send(json.dumps(auth_message))
        self.running = True

        # Start monitoring thread
        threading.Thread(target=self.monitor_hardware, daemon=True).start()

    def on_message(self, ws, message):
        data = json.loads(message)

        if data["type"] == "event":
            event_type = data["event"]
            if event_type == "game:launched":
                self.on_game_launched(data["data"])
            elif event_type == "system:shutdown":
                self.cleanup_and_exit()

    def monitor_hardware(self):
        while self.running:
            # Collect hardware data
            cpu_percent = psutil.cpu_percent()
            memory = psutil.virtual_memory()

            # Send real-time updates to HyperAI
            if cpu_percent > 90:  # High CPU alert
                self.send_alert("high_cpu", {
                    "cpu_percent": cpu_percent,
                    "timestamp": time.time()
                })

            if memory.percent > 85:  # High memory alert
                self.send_alert("high_memory", {
                    "memory_percent": memory.percent,
                    "available_gb": memory.available / (1024**3)
                })

            time.sleep(1)  # Monitor every second

    def send_alert(self, alert_type, data):
        message = {
            "type": "alert",
            "alert": alert_type,
            "data": data,
            "timestamp": time.time()
        }
        self.ws.send(json.dumps(message))

    def on_game_launched(self, game_data):
        # Boost monitoring frequency when game starts
        self.monitoring_interval = 0.5  # Monitor every 500ms during games

        # Could also trigger performance optimizations
        self.optimize_for_gaming()

if __name__ == "__main__":
    monitor = HardwareMonitor()
    monitor.connect()
    monitor.ws.run_forever()  # Keep connection alive

Executable Plugin Development (For Complex Scenarios)

Python (using PyInstaller)

# Install dependencies
pip install pyinstaller

# Create executable
pyinstaller --onefile --name plugin main.py

# Package for distribution
mkdir my-complex-plugin
cp dist/plugin.exe my-complex-plugin/
cp plugin.json my-complex-plugin/
zip -r my-complex-plugin.zip my-complex-plugin/

C# (Self-Contained Deployment)

# Build self-contained executable
dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true

# Package
mkdir my-plugin
cp bin/Release/net6.0/win-x64/publish/MyPlugin.exe my-plugin/plugin.exe
cp plugin.json my-plugin/
zip -r my-plugin.zip my-plugin/

Go (Native Compilation)

# Build executable
go build -o plugin.exe main.go

# Package
mkdir my-plugin
cp plugin.exe my-plugin/
cp plugin.json my-plugin/
zip -r my-plugin.zip my-plugin/

3. Testing

Test your plugin by launching it through HyperHQ, which will start the Socket.IO server and provide the necessary authentication credentials.

Language Examples

Python Plugin Template

#!/usr/bin/env python3
import socketio
import asyncio
import os

class MyPlugin:
    def __init__(self):
        self.sio = socketio.AsyncClient()
        self.settings = {}
        self.session_token = None

        # Read environment variables
        self.plugin_id = os.environ.get('HYPERHQ_PLUGIN_ID')
        self.auth_challenge = os.environ.get('HYPERHQ_AUTH_CHALLENGE')
        self.socket_port = os.environ.get('HYPERHQ_SOCKET_PORT', '52789')

        self.setup_handlers()

    def setup_handlers(self):
        @self.sio.on('connect', namespace='/plugin')
        async def on_connect():
            await self.sio.emit('authenticate', {
                'pluginId': self.plugin_id,
                'challenge': self.auth_challenge
            }, namespace='/plugin')

        @self.sio.on('authenticated', namespace='/plugin')
        async def on_authenticated(response):
            if response.get('success'):
                self.session_token = response.get('sessionToken')
                await self.register()

        @self.sio.on('request', namespace='/plugin')
        async def on_request(data):
            response = await self.handle_request(data)
            await self.sio.emit('plugin:response', {
                'id': data['id'],
                'type': 'response',
                'data': response,
                'sessionToken': self.session_token
            }, namespace='/plugin')

    async def initialize(self, data):
        self.settings = data.get('settings', {})
        return "ok"

    async def execute(self, data):
        action = data.get('action')
        if action == 'game_start':
            return True
        elif action == 'game_stop':
            return True
        return False

    def test(self):
        return True

    async def connect(self):
        await self.sio.connect(f'http://localhost:{self.socket_port}',
                              namespaces=['/plugin'])

    async def register(self):
        await self.sio.emit('plugin:register', {
            'id': self.plugin_id,
            'version': '1.0.0',
            'capabilities': ['socketio']
        }, namespace='/plugin')

    async def handle_request(self, message):
        method = message.get('method')
        data = message.get('data', {})

        if method == 'initialize':
            return await self.initialize(data)
        elif method == 'execute':
            return await self.execute(data)
        elif method == 'test':
            return self.test()
        elif method == 'shutdown':
            return 'ok'
        else:
            return {'error': f'Unknown method: {method}'}

async def main():
    plugin = MyPlugin()
    await plugin.connect()
    await plugin.sio.wait()

if __name__ == '__main__':
    asyncio.run(main())

C# Plugin Template

using System;
using SocketIOClient;
using System.Threading.Tasks;

public class MyPlugin
{
    private SocketIOClient.SocketIO socket;
    private string sessionToken;
    private Dictionary<string, object> settings = new();

    public async Task Initialize()
    {
        var pluginId = Environment.GetEnvironmentVariable("HYPERHQ_PLUGIN_ID");
        var authChallenge = Environment.GetEnvironmentVariable("HYPERHQ_AUTH_CHALLENGE");
        var socketPort = Environment.GetEnvironmentVariable("HYPERHQ_SOCKET_PORT") ?? "52789";

        socket = new SocketIOClient.SocketIO($"http://localhost:{socketPort}/plugin");

        socket.On("authenticated", response => {
            var data = response.GetValue<Dictionary<string, object>>();
            if (data["success"].ToString() == "True") {
                sessionToken = data["sessionToken"].ToString();
                Register();
            }
        });

        socket.On("request", async response => {
            var data = response.GetValue<Dictionary<string, object>>();
            var result = await HandleRequest(data);
            await socket.EmitAsync("plugin:response", new {
                id = data["id"],
                type = "response",
                data = result,
                sessionToken = sessionToken
            });
        });

        await socket.ConnectAsync();

        await socket.EmitAsync("authenticate", new {
            pluginId = pluginId,
            challenge = authChallenge
        });
    }

    private async Task Register()
    {
        await socket.EmitAsync("plugin:register", new {
            id = Environment.GetEnvironmentVariable("HYPERHQ_PLUGIN_ID"),
            version = "1.0.0",
            capabilities = new[] { "socketio" }
        });
    }

    private async Task<object> HandleRequest(Dictionary<string, object> message)
    {
        var method = message["method"].ToString();
        var data = message.GetValueOrDefault("data", new Dictionary<string, object>()) as Dictionary<string, object>;

        return method switch
        {
            "initialize" => InitializePlugin(data),
            "execute" => await ExecutePlugin(data),
            "test" => TestPlugin(),
            "shutdown" => "ok",
            _ => new { error = $"Unknown method: {method}" }
        };
    }

    private string InitializePlugin(Dictionary<string, object> data)
    {
        if (data.ContainsKey("settings"))
            settings = (Dictionary<string, object>)data["settings"];
        return "ok";
    }

    private async Task<bool> ExecutePlugin(Dictionary<string, object> data)
    {
        var action = data.GetValueOrDefault("action", "").ToString();
        return action switch
        {
            "game_start" => true,
            "game_stop" => true,
            _ => false
        };
    }

    private bool TestPlugin() => true;
}

class Program
{
    static async Task Main()
    {
        var plugin = new MyPlugin();
        await plugin.Initialize();
        await Task.Delay(-1); // Keep running
    }
}

Security Considerations

Process Isolation

• Each plugin runs in its own process
• Plugin crashes don't affect HyperAI
• Memory isolation between plugins
• Controlled resource usage

Permission System

interface PluginPermissions {
  filesystem: string[];        // Only allowed paths
  processLaunching: boolean;   // Can spawn processes
  networkAccess: boolean;      // Internet access
}

Validation

• Manifest validation before installation
• Executable signature verification (future)
• Runtime permission enforcement
• Settings namespace validation

Installation Process

export class PluginInstaller {
  async installPlugin(packageUrl: string): Promise<void> {
    // 1. Download and extract package
    const tempDir = await this.downloadAndExtract(packageUrl);

    // 2. Validate manifest and executable
    const manifest = await this.validatePackage(tempDir);

    // 3. Security scan
    await this.securityScan(tempDir, manifest);

    // 4. Install to plugins directory
    const pluginDir = this.getPluginPath(manifest.id);
    await this.installFiles(tempDir, pluginDir);

    // 5. Register in database
    await this.registerPlugin(manifest);

    // 6. Test installation
    await this.testPlugin(manifest.id);
  }
}

Complex Plugin Example: MAME Integration

MAME integration represents one of the most complex plugin scenarios, involving multi-step wizards, bulk database operations, and long-running processes. Here's how our architecture handles this complexity:

Extended Plugin Manifest

interface ComplexPluginManifest extends PluginManifest {
  ui: {
    type: 'wizard';
    steps: PluginStep[];       // Multi-step wizard definition
    displayName: string;
    category: string;
    settings: PluginSettingDefinition[];
  };

  // Extended permissions for complex operations
  permissions: {
    database: {
      read: string[];          // ['roms', 'systems', 'emulators']
      write: string[];         // ['roms', 'systems', 'emulators']
      bulk: boolean;           // Allow bulk operations (thousands of ROMs)
      createEntities: string[]; // ['system', 'emulator']
    };
    filesystem: {
      read: boolean;
      write: boolean;
      scan: boolean;           // Can scan large directories
      paths: string[];         // Restricted paths for ROM scanning
    };
    network: {
      enabled: boolean;        // External API calls
      domains: string[];       // ['api.github.com', 'emumovies.com']
      download: boolean;       // Can download MAME binaries
    };
    longRunning: boolean;      // Operations that take hours
    systemIntegration: boolean; // Can create emulators/systems
  };

  // Resource requirements
  resources: {
    memory: 'high';           // For processing thousands of ROMs
    disk: 'high';             // For MAME downloads (GB)
    cpu: 'medium';            // For ROM matching algorithms
  };
}

MAME Plugin Communication Flow

// Step 1: Plugin requests MAME versions
await pluginManager.sendCommand('mame-plugin', {
  method: 'get_mame_versions',
  data: {}
});

// Step 2: User selects version, plugin downloads MAME
await pluginManager.sendCommand('mame-plugin', {
  method: 'download_mame',
  data: {
    version: '0.250',
    installPath: '/path/to/emulators/mame'
  }
});

// Step 3: Plugin creates emulator entry
const emulator = await pluginManager.sendCommand('mame-plugin', {
  method: 'create_emulator',
  data: {
    name: 'MAME 0.250',
    path: '/path/to/mame.exe',
    version: '0.250'
  }
});

// Step 4: Plugin creates system entry
const system = await pluginManager.sendCommand('mame-plugin', {
  method: 'create_system',
  data: {
    name: 'Arcade',
    romPath: '/path/to/roms',
    emulatorId: emulator.id
  }
});

// Step 5: Long-running ROM import with progress
await pluginManager.sendCommand('mame-plugin', {
  method: 'import_roms',
  data: {
    systemId: system.id,
    romPath: '/path/to/roms',
    filters: {
      categories: ['Fighting', 'Puzzle'],
      regions: ['USA', 'Japan'],
      clones: 'merge'
    }
  }
});

Multi-Step Wizard Support

interface PluginStep {
  id: string;
  title: string;
  component: 'version-selector' | 'path-selector' | 'filter-config' | 'progress';
  required: boolean;
  validation?: PluginValidation;
  fields?: PluginField[];
}

// MAME wizard steps
const mameWizardSteps: PluginStep[] = [
  {
    id: 'version_selection',
    title: 'Select MAME Version',
    component: 'version-selector',
    required: true,
    fields: [
      {
        name: 'version',
        type: 'dropdown',
        label: 'MAME Version',
        options: [] // Populated by plugin
      }
    ]
  },
  {
    id: 'rom_path',
    title: 'ROM Directory',
    component: 'path-selector',
    required: true,
    validation: {
      type: 'directory_exists',
      message: 'ROM directory must exist and be accessible'
    }
  },
  {
    id: 'filters',
    title: 'Import Filters',
    component: 'filter-config',
    required: false,
    fields: [
      {
        name: 'categories',
        type: 'multiselect',
        label: 'Game Categories',
        options: [] // Populated by plugin
      }
    ]
  },
  {
    id: 'progress',
    title: 'Installing MAME',
    component: 'progress',
    required: false
  }
];

Bulk Database Operations

// Plugin service handles bulk operations efficiently
export class PluginDatabaseService {
  async bulkInsertRoms(pluginId: string, roms: Rom[]): Promise<void> {
    // Validate plugin has permission for bulk operations
    if (!this.hasPermission(pluginId, 'database.bulk')) {
      throw new Error('Plugin does not have bulk database permissions');
    }

    // Process in chunks to avoid memory issues
    const chunkSize = 1000;
    for (let i = 0; i < roms.length; i += chunkSize) {
      const chunk = roms.slice(i, i + chunkSize);
      await this.insertRomChunk(chunk);

      // Send progress updates
      this.sendProgress(pluginId, {
        step: 'importing_roms',
        progress: Math.round((i / roms.length) * 100),
        message: `Imported ${i + chunk.length} of ${roms.length} ROMs`
      });
    }
  }
}

Long-Running Process Management

// Plugin manager handles long-running operations via Socket.IO
export class PluginProcessManager {
  async executeLongRunningCommand(
    pluginId: string,
    command: PluginCommand,
    onProgress?: (progress: PluginProgress) => void
  ): Promise<any> {

    const socket = this.getPluginSocket(pluginId);

    // Send command via Socket.IO
    const commandId = uuid();
    socket.emit('request', {
      id: commandId,
      method: command.method,
      data: command.data
    });

    // Listen for progress updates
    return new Promise((resolve, reject) => {
      const timeout = setTimeout(() => {
        reject(new Error('Plugin operation timed out'));
      }, 30 * 60 * 1000); // 30 minute timeout

      const progressHandler = (message: any) => {
        if (message.id === commandId || message.taskId === commandId) {
          onProgress?.(message);
        }
      };

      const responseHandler = (message: any) => {
        if (message.id === commandId) {
          clearTimeout(timeout);
          socket.off('plugin:progress', progressHandler);
          socket.off('plugin:response', responseHandler);
          socket.off('plugin:error', errorHandler);
          resolve(message.data);
        }
      };

      const errorHandler = (message: any) => {
        if (message.id === commandId) {
          clearTimeout(timeout);
          socket.off('plugin:progress', progressHandler);
          socket.off('plugin:response', responseHandler);
          socket.off('plugin:error', errorHandler);
          reject(new Error(message.error));
        }
      };

      socket.on('plugin:progress', progressHandler);
      socket.on('plugin:response', responseHandler);
      socket.on('plugin:error', errorHandler);
    });
  }
}

MAME Plugin Implementation (Python Example)

#!/usr/bin/env python3
import socketio
import asyncio
import os
import zipfile
import requests
from pathlib import Path

class MamePlugin:
    def __init__(self):
        self.sio = socketio.AsyncClient()
        self.settings = {}
        self.session_token = None
        self.temp_dir = Path("/tmp/mame-plugin")

        # Read environment variables
        self.plugin_id = os.environ.get('HYPERHQ_PLUGIN_ID')
        self.auth_challenge = os.environ.get('HYPERHQ_AUTH_CHALLENGE')
        self.socket_port = os.environ.get('HYPERHQ_SOCKET_PORT', '52789')

        self.setup_handlers()

    def setup_handlers(self):
        @self.sio.on('connect', namespace='/plugin')
        async def on_connect():
            await self.sio.emit('authenticate', {
                'pluginId': self.plugin_id,
                'challenge': self.auth_challenge
            }, namespace='/plugin')

        @self.sio.on('authenticated', namespace='/plugin')
        async def on_authenticated(response):
            if response.get('success'):
                self.session_token = response.get('sessionToken')
                await self.register()

        @self.sio.on('request', namespace='/plugin')
        async def on_request(data):
            response = await self.handle_request(data)
            await self.sio.emit('plugin:response', {
                'id': data['id'],
                'type': 'response',
                'data': response,
                'sessionToken': self.session_token
            }, namespace='/plugin')

    async def get_mame_versions(self, data):
        """Fetch available MAME versions from GitHub"""
        response = requests.get("https://api.github.com/repos/mamedev/mame/releases")
        releases = response.json()

        versions = []
        for release in releases[:10]:  # Last 10 versions
            versions.append({
                "name": release["name"],
                "tag": release["tag_name"],
                "download_url": self.get_windows_asset_url(release)
            })

        return {"versions": versions}

    async def download_mame(self, data):
        """Download and install MAME"""
        version = data["version"]
        install_path = Path(data["installPath"])

        # Download MAME
        await self.send_progress(data['id'], "downloading", 10, "Downloading MAME...")

        # Extract MAME
        await self.send_progress(data['id'], "extracting", 50, "Extracting MAME...")

        # Configure MAME
        await self.send_progress(data['id'], "configuring", 90, "Configuring MAME...")

        return {"success": True, "path": str(install_path / "mame.exe")}

    async def import_roms(self, data):
        """Import ROMs with filtering"""
        rom_path = Path(data["romPath"])
        filters = data.get("filters", {})

        # Scan ROM files
        await self.send_progress(data['id'], "scanning", 10, "Scanning ROM files...")
        rom_files = list(rom_path.glob("*.zip"))

        # Match against MAME database
        await self.send_progress(data['id'], "matching", 30, "Matching ROMs...")
        matched_roms = self.match_roms(rom_files, filters)

        # Send bulk insert request
        await self.send_progress(data['id'], "importing", 70, "Importing ROMs to database...")

        return {
            "imported": len(matched_roms),
            "total_found": len(rom_files)
        }

    async def send_progress(self, task_id, step, progress, message):
        """Send progress update to HyperHQ via Socket.IO"""
        await self.sio.emit('plugin:progress', {
            "taskId": task_id,
            "step": step,
            "progress": progress,
            "message": message
        }, namespace='/plugin')

    async def connect(self):
        await self.sio.connect(f'http://localhost:{self.socket_port}',
                              namespaces=['/plugin'])

    async def register(self):
        await self.sio.emit('plugin:register', {
            'id': self.plugin_id,
            'version': '1.0.0',
            'capabilities': ['socketio', 'mameIntegration']
        }, namespace='/plugin')

    async def handle_request(self, message):
        method = message.get('method')
        data = message.get('data', {})

        if method == "get_mame_versions":
            return await self.get_mame_versions(data)
        elif method == "download_mame":
            return await self.download_mame(data)
        elif method == "import_roms":
            return await self.import_roms(data)
        else:
                result = {"error": f"Unknown method: {method}"}

            response = {
                "id": message["id"],
                "type": "response",
                "data": result
            }
            print(json.dumps(response))

        except Exception as e:
            error_response = {
                "id": message.get("id", "unknown"),
                "type": "error",
                "error": str(e)
            }
            print(json.dumps(error_response))

if __name__ == '__main__':
    main()

This architecture allows MAME-level complexity while maintaining the simplicity and isolation of the executable plugin approach. The key innovations are:

1. Extended Permissions: Granular control over what complex plugins can do
2. Multi-Step Wizards: Support for complex UI flows
3. Progress Reporting: Real-time updates for long operations
4. Bulk Operations: Efficient handling of large datasets
5. Resource Management: Proper handling of memory/CPU intensive operations

Benefits

For Users

• ✅ Zero Setup: No need to install runtimes or dependencies
• ✅ Consistent Experience: All plugins look and behave the same
• ✅ Reliable: Plugins work regardless of system configuration
• ✅ Safe: Isolated execution prevents system corruption
• ✅ Easy Management: Simple install/uninstall process

For Developers

• ✅ Language Freedom: Use any language that compiles to executable
• ✅ Familiar Tools: Use existing build processes and tools
• ✅ Simple API: JSON-based communication is universal
• ✅ Easy Testing: Test plugins independently from HyperAI
• ✅ Clear Interface: Well-defined manifest and protocol

For HyperAI

• ✅ Simple Integration: Just spawn process and exchange JSON
• ✅ Crash Isolation: Plugin bugs don't crash main app
• ✅ Resource Control: Monitor and limit plugin resource usage
• ✅ Security: Process boundaries provide natural sandboxing
• ✅ Maintainability: Clear separation of concerns

Implementation Roadmap

Phase 1: JavaScript Plugin Foundation (Weeks 1-2)

• JavaScript plugin runner with V8 context isolation
• Plugin manifest validation (supporting both JavaScript and executable types)
• Basic plugin discovery and loading for JavaScript plugins
• Settings isolation service with namespace support
• Safe HyperAI API for JavaScript plugins (data, settings, UI access)

Phase 2: JavaScript Plugin System Complete (Weeks 3-4)

• Complete JavaScript plugin API implementation
• Data request handling and routing for JS plugins
• Dynamic UI generation from plugin manifests
• Component library for common plugin UI elements
• Error handling and recovery for JavaScript plugins
• Plugin hot-reload capability
• WebSocket server for real-time communication

Phase 3: Executable Plugin Support (Weeks 5-6)

• Executable plugin process management
• JSON message protocol for executable plugins
• Unified plugin manager (handles both JavaScript and executable types)
• Cross-process communication and lifecycle management
• Process isolation and resource monitoring

Phase 4: Security & Permissions (Weeks 7-8)

• JavaScript sandbox security enforcement (V8 context isolation)
• Executable plugin permission system
• Process sandboxing and resource limits
• Security validation and enforcement for both types
• Plugin signing and verification system

Phase 5: Developer Experience (Weeks 9-10)

• JavaScript plugin development templates and examples
• Executable plugin development templates (Python, C#, Go)
• Plugin type decision guide and migration tools
• Documentation and guides for both plugin types
• Testing framework and debugging tools
• Plugin development SDK

Phase 6: Advanced Features & Ecosystem (Weeks 11-12)

• Plugin performance monitoring and optimization
• Automatic updates and version management
• Plugin ecosystem management tools
• Advanced debugging features (especially for JavaScript plugins)
• Plugin distribution and installation system
• Real-world validation with complex plugins (MAME integration)

Recommended Development Strategy:

1. JavaScript First (80% of use cases, simpler implementation)
   • Faster development cycle, better security, cross-platform by default
2. Add Executable Support once JavaScript foundation is solid
   • For complex system integration that JavaScript can't handle
3. Provide Clear Guidelines to help developers choose the right approach
4. Migration Tools to upgrade JavaScript plugins to executables when needed

---

This document represents the complete architectural design for the HyperAI Plugin System, utilizing a hybrid approach with JavaScript plugins for simplicity and executable plugins for complex system integration.

Potential Issues and Challenges

While the executable-based plugin system provides significant benefits, there are several challenges and potential issues to consider:

1. Performance Overhead

Process Spawning Costs

// ARCHITECTURE CONSTRAINT: Plugins only start when requested
const pluginProcess = spawn('./plugins/mame-helper/plugin.exe'); // Only when user accesses plugin
// CPU cost: ~10-50ms per plugin startup (but only 1-2 plugins active)
// Memory cost: ~2-10MB per plugin process (but only when needed)
// With on-demand loading: No startup delay, minimal memory usage

Built-in Mitigation via Architecture:

• ✅ On-demand loading: Plugins only start when user requests them
• ✅ Minimal memory footprint: Only active plugin(s) consume resources
• ✅ No startup penalty: HyperAI starts immediately, plugins load as needed
• ✅ Resource efficiency: Never have 20+ plugins running simultaneously

JSON Communication Overhead

// Large data transfers are expensive
const romData = {
  roms: Array(10000), // 10,000 ROMs = ~20MB JSON
  systems: systems,
  metadata: metadata
};
// JSON.stringify(romData) = ~50-100ms
// Process IPC transfer = ~100-200ms additional

Mitigation Strategies:

• Implement data pagination by default
• Use binary protocols for large transfers
• Cache frequently requested data
• Stream large datasets in chunks

2. Platform Compatibility Issues

Multi-Platform Executables

# Plugin developers must build for each platform
my-plugin/
├── win32/
│   └── plugin.exe          # Windows x64
├── darwin/
│   └── plugin              # macOS Intel
├── darwin-arm64/
│   └── plugin              # macOS Apple Silicon
└── linux/
    └── plugin              # Linux x64

Issues:

• 4x larger download packages
• Complex build pipelines for developers
• Platform-specific bugs and testing
• Dependency hell for compiled languages

Mitigation Strategies:

• Provide build automation tools
• Docker-based build environments
• Platform-specific plugin repositories
• Fallback to JavaScript for simple plugins

3. Security Vulnerabilities

Executable Trust Model

// Users downloading executable files is inherently risky
interface SecurityRisks {
  maliciousExecutables: boolean;    // Plugin could be malware
  codeSigningRequired: boolean;     // Need certificate validation
  sandboxingLimits: boolean;        // Process isolation isn't perfect
  permissionEscalation: boolean;    // Plugin could bypass restrictions
}

Mitigation Strategies:

• Mandatory code signing for plugin distribution
• Runtime permission enforcement
• File system sandboxing (chroot/containers)
• Network traffic monitoring
• Plugin reputation/review system

Process Isolation Limitations

// Plugins still have significant system access
const pluginLimitations = {
  fileSystemAccess: "Limited but not completely sandboxed",
  networkAccess: "Can make requests if permitted",
  processSpawning: "Can launch other executables",
  memoryAccess: "Isolated but can consume excessive resources"
};

4. Development Complexity

Cross-Process Debugging

# Debugging plugin issues is more complex
class PluginDebuggingChallenges:
    def debug_plugin(self):
        # Can't easily step through HyperAI + Plugin code together
        # Need separate debugging sessions
        # Error stack traces span process boundaries
        # Async communication makes debugging harder
        pass

Mitigation Strategies:

• Rich debugging tools and logging
• Plugin development SDK with test harnesses
• Local debugging mode (in-process for development)
• Comprehensive error reporting

Build Complexity

# Plugin developers need complex build setups
build_requirements:
  - Cross-platform compilation
  - Dependency bundling
  - Executable optimization
  - Package creation automation
  - Testing across platforms

5. Resource Management Issues

Memory Leaks and Resource Exhaustion

class ResourceManagementConcerns {
  // ARCHITECTURE CONSTRAINT: Only 1-2 plugins active at a time
  memoryLeaks(): void {
    // ✅ REDUCED IMPACT: Only one plugin can leak memory at a time
    // ✅ EASIER DETECTION: Clear which plugin is causing issues
    // ✅ SIMPLE RECOVERY: Kill single problematic plugin, not entire system
  }

  cpuThrottling(): void {
    // ✅ REDUCED IMPACT: Only active plugin can cause CPU issues
    // ✅ EASIER MONITORING: Monitor single active plugin process
    // ✅ USER CONTROL: User can simply close plugin settings to stop it
  }

  processAccumulation(): void {
    // ✅ MINIMAL ISSUE: At most 1-2 zombie processes vs dozens
    // ✅ EASIER CLEANUP: Clear lifecycle when user closes plugin UI
    // ✅ AUTOMATIC TIMEOUT: Can timeout idle plugins after period
  }
}

Simplified Mitigation (due to architecture):

• ✅ Isolated impact: Resource issues affect only single plugin
• ✅ User control: Closing plugin settings stops problematic process
• ✅ Easy recovery: Restart just the problematic plugin, not system
• ✅ Simple monitoring: Track single active plugin vs dozens

6. User Experience Challenges

Plugin Crashes and Error Handling

interface PluginFailureScenarios {
  pluginCrash: "Plugin executable crashes unexpectedly";
  communicationTimeout: "Plugin stops responding to requests";
  corruptedData: "Plugin sends malformed JSON responses";
  startupFailure: "Plugin fails to initialize properly";
}

// Impact on user experience:
// - UI components may freeze or show errors
// - Game launches could fail
// - Settings might not save properly
// - Overall system instability perception

Mitigation Strategies:

• Graceful degradation when plugins fail
• Clear error messages for users
• Plugin auto-recovery mechanisms
• Fallback functionality when plugins are unavailable

Startup Performance

// ARCHITECTURE CONSTRAINT: Plugins don't start until requested
const startupBehavior = {
  hyperaiStartup: "Fast - no plugins loaded at startup",
  pluginActivation: "On-demand when user accesses plugin settings/features",
  memoryUsage: "Minimal - only active plugins consume resources",
  userExperience: "Immediate HyperAI availability, plugins load as needed"
};

// Old concern (if all plugins started at once):
// sequentialLoading: "Plugins start one after another" ❌ NOT AN ISSUE
// resourceContention: "All plugins compete for CPU/disk" ❌ NOT AN ISSUE
// userFrustration: "Long delay before usable" ❌ NOT AN ISSUE

7. Distribution and Update Challenges

Package Management Complexity

interface DistributionChallenges {
  versionCompatibility: "Plugin version X only works with HyperAI version Y";
  updateMechanisms: "How to update plugins without breaking user setups";
  dependencyConflicts: "Two plugins need different versions of same library";
  rollbackSupport: "What happens if plugin update breaks user's setup";
}

Mitigation Strategies:

• Semantic versioning with compatibility matrices
• Atomic updates with rollback capability
• Plugin compatibility testing in CI/CD
• User notification system for updates

8. Communication Protocol Limitations

JSON-Only Communication

// Current design limitations
interface CommunicationLimitations {
  binaryData: "Can't efficiently transfer images, videos, executables";
  streaming: "No support for real-time data streams";
  latency: "JSON serialization adds overhead to every request";
  sizeLimit: "Large objects cause performance issues";
}

Potential Solutions:

• Hybrid protocol: JSON for commands, binary for data
• Shared memory for large data transfers
• Plugin-specific communication channels
• WebSocket-like streaming for real-time data

9. Long-Term Maintenance Burden

Plugin Ecosystem Management

class EcosystemChallenges {
  abandonedPlugins(): void {
    // Developer stops maintaining plugin
    // Security vulnerabilities accumulate
    // Compatibility breaks with new HyperAI versions
    // Users lose functionality they depend on
  }

  qualityControl(): void {
    // No enforcement of code quality
    // Poorly written plugins impact system performance
    // Inconsistent user experience across plugins
    // Support burden increases
  }
}

10. Technical Debt Accumulation

API Evolution Challenges

interface APIEvolutionIssues {
  backwardCompatibility: "New HyperAI features may break old plugins";
  apiVersioning: "Need to support multiple API versions simultaneously";
  deprecationManagement: "How to retire old APIs without breaking plugins";
  documentationMaintenance: "Keeping plugin docs updated with changes";
}

Risk Assessment Summary

Updated for On-Demand Plugin Architecture (settings for one plugin at a time, startup only when requested):

Risk Category	Severity	Likelihood	Mitigation Complexity	Architecture Impact
Performance Issues	Medium Low	High Low	Medium Low	✅ On-demand loading eliminates startup delays
Security Vulnerabilities	High	Medium	High	⚠️ Still high - executable trust model unchanged
Development Complexity	Medium	High	Low	⚠️ Cross-platform compilation still complex
Resource Management	Medium Low	Medium Low	Medium Low	✅ Only 1-2 plugins active, easier monitoring
User Experience	Medium Low	High Low	Medium Low	✅ Fast startup, isolated plugin issues
Distribution Challenges	Low	Medium	Medium	➖ No change
Communication Limits	Low	Low	High	➖ No change
Maintenance Burden	High	High	High	➖ Still complex ecosystem management

Key Improvements:

• Performance: No longer a major concern due to on-demand loading
• Resource Management: Greatly simplified with 1-2 active plugins max
• User Experience: Much better with instant HyperAI startup

Recommended Mitigation Approach

1. Phase 1: Start with simple, well-sandboxed plugins
2. Phase 2: Implement robust monitoring and resource management
3. Phase 3: Add advanced security and signing requirements
4. Phase 4: Develop plugin ecosystem management tools

The key is to start simple and evolve the system based on real-world usage patterns and identified issues.