Caching

1) General Definition

• Caching = storing the result of an expensive computation (or frequently accessed data) in a fast-access storage so future requests can be served quickly without recomputing.
• A core systems optimization, especially important in ML pipelines where computations are heavy and repeated.

2) Where Caching is used in ML

a) Data caching

• Store preprocessed datasets (e.g., tokenized text, augmented images) so they don’t need to be recomputed every training run.
• Example: HuggingFace Datasets library caches tokenized data on disk.

b) Feature caching

• Store frequently used engineered features, embeddings, or intermediate representations.
• Example: A recommendation system might cache user/item embeddings for fast retrieval.

c) Model caching

• Store preloaded models in memory for inference instead of reloading from disk each time.
• Example: ONNX Runtime or TensorFlow Serving keeps the model cached in RAM.

d) Inference result caching

• Cache predictions for repeated queries (e.g., same image uploaded twice).
• Example: Search engines and recommender systems often cache results for popular items.

e) Web/service-level caching

• Use Redis, Memcached, or CDN edge caching to reduce latency for inference APIs.

3) Benefits

• Latency reduction → faster responses for repeated requests.
• Compute savings → avoid redundant GPU/TPU computations.
• Cost efficiency → lower cloud/compute bills.
• Scalability → handle higher throughput without adding servers.

4) Trade-offs

• Staleness: cached predictions/features may become outdated if model/data drifts.
• Memory usage: caches consume RAM/disk.
• Invalidation: must decide when to refresh (cache invalidation is famously hard in CS ).

5) Example: Inference Caching with Redis

import redis, hashlib, json

cache = redis.Redis()

def predict_with_cache(model, input_data):
    key = hashlib.sha256(json.dumps(input_data).encode()).hexdigest()
    if cache.exists(key):
        return json.loads(cache.get(key))
    else:
        result = model.predict(input_data)
        cache.set(key, json.dumps(result), ex=3600)  # expire in 1 hour
        return result

import redis, hashlib, json

cache = redis.Redis()

def predict_with_cache(model, input_data):
    key = hashlib.sha256(json.dumps(input_data).encode()).hexdigest()
    if cache.exists(key):
        return json.loads(cache.get(key))
    else:
        result = model.predict(input_data)
        cache.set(key, json.dumps(result), ex=3600)  # expire in 1 hour
        return result

If the same input_data comes again, prediction is served from cache instantly.

6) Applications in Production ML

• LLMs: cache embeddings or responses for repeated prompts.
• Recommender systems: cache top-N results for popular users/items.
• Computer vision APIs: cache results for duplicate uploads.
• Training pipelines: cache intermediate feature extraction to save preprocessing time.

Summary

• Caching = storing expensive results for reuse.
• Used at data, feature, model, and inference levels in ML.
• Benefits: faster, cheaper, more scalable.
• Challenges: freshness, invalidation, memory management.