Streaming Pipeline Protocol for Real-Time Context

Pipeline Architecture

Real-time context requires streaming pipelines that process events as they occur. This specification defines the pipeline protocol for ECM-compliant streaming systems.

Pipeline Definition

Pipeline Specification

{
  "pipeline_id": "customer-context-pipeline",
  "stages": [
    {
      "stage_id": "ingest",
      "type": "source",
      "config": {"connector": "salesforce", "entities": ["Account", "Contact"]}
    },
    {
      "stage_id": "transform",
      "type": "processor",
      "config": {"processor": "field_mapper", "mapping": {...}}
    },
    {
      "stage_id": "enrich",
      "type": "processor", 
      "config": {"processor": "lookup_enricher", "lookup_source": "reference_data"}
    },
    {
      "stage_id": "sink",
      "type": "sink",
      "config": {"store": "context_store", "context_type": "customer-context"}
    }
  ]
}

Message Protocol

Pipeline Message Format

{
  "message_id": "uuid",
  "timestamp": "ISO-8601",
  "source": {"stage": "ingest", "connector": "salesforce"},
  "key": "customer-123",
  "value": {...},
  "headers": {
    "ecm_pipeline": "customer-context-pipeline",
    "ecm_stage": "transform",
    "ecm_correlation_id": "req-abc"
  }
}

Processing Guarantees

Delivery Semantics

ECM pipelines support configurable semantics: at-most-once (fastest, may lose), at-least-once (default, may duplicate), exactly-once (strongest, requires transactions).

Ordering Guarantees

Ordering by partition key. Per-entity ordering preserved. Cross-entity ordering best-effort. Configurable ordering requirements.

Backpressure Protocol

Flow Control

Pipeline stages implement backpressure: bounded buffers between stages, blocking when downstream slow, configurable buffer sizes, and overflow handling policies.

Rate Limiting

{
  "rate_limit": {
    "requests_per_second": 1000,
    "burst": 100,
    "strategy": "token_bucket"
  }
}

Windowing Operations

Window Types

Support standard windowing: tumbling windows (fixed, non-overlapping), sliding windows (fixed, overlapping), session windows (gap-based), and global windows (unbounded).

Window Specification

{
  "window": {
    "type": "tumbling",
    "size": "1 minute",
    "allowed_lateness": "30 seconds",
    "trigger": "watermark"
  }
}

Conclusion

The Streaming Pipeline Protocol enables real-time context processing with configurable guarantees. Implement pipeline stages, respect backpressure, and support windowing for aggregate context computation.

> Streaming Pipeline Protocol for Real-Time Context