ECM Protocol: Data Source Connector Specification

Connector Architecture

ECM Protocol data source connectors bridge source systems to context stores. This specification defines the connector interface and implementation requirements.

Connector Interface

Required Methods

interface ECMConnector {
  // Lifecycle
  initialize(config: ConnectorConfig): Promise<void>;
  shutdown(): Promise<void>;
  
  // Discovery
  listEntities(): Promise<EntityType[]>;
  describeEntity(type: string): Promise<EntitySchema>;
  
  // Extraction
  extractSnapshot(type: string, options: ExtractOptions): AsyncIterator<Context>;
  streamChanges(types: string[], cursor: string): AsyncIterator<ChangeEvent>;
  
  // Metadata
  getCapabilities(): ConnectorCapabilities;
  healthCheck(): Promise<HealthStatus>;
}

Configuration Specification

Connection Configuration

{
  "connector_type": "postgresql",
  "connection": {
    "host": "db.example.com",
    "port": 5432,
    "database": "production",
    "ssl_mode": "require"
  },
  "authentication": {
    "type": "credentials",
    "username": "ecm_reader",
    "password_secret": "vault://db/postgres/password"
  },
  "options": {
    "batch_size": 1000,
    "change_tracking": "cdc"
  }
}

Entity Mapping

Schema Mapping

Map source schemas to ECM context types:

{
  "mappings": [
    {
      "source": {"table": "customers"},
      "target": {"context_type": "customer-context"},
      "fields": [
        {"source": "customer_id", "target": "context_id"},
        {"source": "email", "target": "data.email"},
        {"source": "preferences", "target": "data.preferences", "transform": "json_parse"}
      ]
    }
  ]
}

Change Data Capture

CDC Protocol

Connectors should support CDC for incremental extraction:

{
  "operation": "connector.stream_changes",
  "cursor": "pg_lsn:0/16B3748",
  "batch_size": 100,
  "include_operations": ["insert", "update", "delete"]
}

Cursor Management

Connectors track position for resumption. Durable cursor storage. Checkpoint on batch completion. Resume from last checkpoint after restart.

Error Handling

Retry Specification

Standard retry behavior: exponential backoff (initial 1s, max 5mins, factor 2), idempotent operations, dead-letter handling for persistent failures.

Error Categories

TRANSIENT: Retry with backoff
CONFIGURATION: Alert, require fix
DATA: Record error, continue
FATAL: Stop, alert

Conclusion

The Connector Specification enables standardized data source integration. Implement the required interface, support configuration-driven mapping, and provide CDC capabilities for efficient incremental extraction.

> ECM Protocol: Data Source Connector Specification