Cache: fix key to be request-based, not Example-based

[darinkishore]

Problem

dspy-rs caches LM responses keyed on Vec<(String, Value)> derived from the Example struct. This is semantically wrong:

• Same logical request through different Example shapes = cache miss
• The cache should key on the actual formatted request (messages + params), not the pre-formatting input

DSPy Python keys on SHA256(json(request_params)) with ignored args stripped (api_key, api_base, etc.), which correctly caches at the LM request boundary.

What's needed

• Cache key = hash of (model, messages, temperature, max_tokens, tools, ...) — the actual CompletionRequest
• Strip non-deterministic fields (api_key, timestamps) from key
• Consider rollout_id semantics for cache-busting during optimization (DSPy uses this when temperature > 0)
• CacheEntry should store raw response + metadata, not Prediction

Related

• data storage #55 (data storage)
• History decoupling (history should be separate from cache)

[darinkishore]

Codebase audit — current cache keying

utils/cache.rs:

type CacheKey = Vec<(String, Value)>;

// Cache::get takes a RawExample (HashMap<String, Value>)
async fn get(&self, key: RawExample) -> Result<Option<Prediction>>;

The key is derived from the Example's data HashMap — field name/value pairs from the pre-formatting input. This means:

• Same logical LM request formatted through different Example shapes = cache miss
• Temperature, model, max_tokens, tools are NOT part of the key
• Changing the adapter's prompt template invalidates nothing (stale hits)

What the key should be:

// Hash of the actual request sent to the LM
struct CacheKey {
    model: String,
    messages: Vec<Message>,  // the formatted chat
    temperature: f32,
    max_tokens: u32,
    tools: Vec<ToolDefinition>,
    // NOT included: api_key, api_base, timestamps
}

Then SHA256(serde_json::to_string(&cache_key)) as the foyer key.

rollout_id semantics: DSPy uses a rollout_id field that gets rotated during optimization to bust the cache when temperature > 0 (you want different samples, not cached ones). This matters for COPRO/GEPA/MIPRO — without it, optimizers may get stale cached responses when they need fresh samples.

Migration path:

1. Change CacheKey type to String (the SHA256 hex)
2. Compute key from CompletionRequest fields in LM::call() before sending
3. Store CacheEntry { messages, params, raw_response, parsed: BamlValue, usage, timestamp }
4. Add rollout_id: Option<String> to key for optimizer cache-busting