Vector Embedding Protocol for Context Retrieval

Embedding Protocol Overview

Semantic context retrieval requires vector embeddings computed and stored alongside context. This specification defines the embedding protocol for ECM-compliant systems.

Embedding Generation

Embedding Request

{
  "operation": "embedding.generate",
  "content": "Customer prefers email communication, has purchased premium tier",
  "model": "text-embedding-3-large",
  "dimensions": 1536,
  "options": {
    "normalize": true,
    "task_type": "retrieval_document"
  }
}

Embedding Response

{
  "embedding": [0.023, -0.145, ...],
  "model": "text-embedding-3-large",
  "dimensions": 1536,
  "token_count": 12,
  "metadata": {
    "computation_ms": 45
  }
}

Storage Integration

Context with Embedding

Embeddings stored alongside context:

{
  "context_id": "ctx-123",
  "context_type": "user-preferences",
  "data": {...},
  "embeddings": [
    {
      "vector": [...],
      "model": "text-embedding-3-large",
      "text_hash": "sha256:abc...",
      "generated_at": "2024-01-15T10:00:00Z"
    }
  ]
}

Embedding Lifecycle

Recompute embeddings when content changes. Track source text hash for change detection. Support multiple embeddings per context.

Vector Query Protocol

Similarity Search

{
  "operation": "vector.search",
  "query_embedding": [...],
  "options": {
    "metric": "cosine",
    "top_k": 10,
    "min_score": 0.7,
    "filters": {
      "context_type": "user-context"
    }
  }
}

Hybrid Search

Combine vector and keyword search. RRF (Reciprocal Rank Fusion) for score combination. Configurable weights for vector vs. keyword.

Index Specifications

Index Types

ECM supports standard index types:

HNSW: Default for balanced performance
IVF: For filtered search at scale
Flat: For exact search on small datasets

Index Parameters

Configurable index parameters. M and efConstruction for HNSW. nlist and nprobe for IVF. Trade-off recall vs. latency.

Protocol Extensions

Vector-specific extensions:

x-ecm-embedding-model: Embedding model identifier
x-ecm-similarity-metric: Distance/similarity function
x-ecm-index-type: Vector index specification

Conclusion

The Vector Embedding Protocol enables semantic context retrieval through standardized embedding generation, storage, and querying. Implementations should support multiple embedding models and index configurations.

> Vector Embedding Protocol for Context Retrieval