Tachyon

The serving layer in front of the model: batching, a semantic cache and a circuit breaker, live.

Not connected

Request path, live

Every request is embedded and checked against the semantic cache. Only misses reach the breaker, the batch queue and the model. Line weight shows where the last five seconds of traffic went.

Cache hit: 0.0/s, answered in – msFallback when the model cannot answer0.0/s degraded from cache, 0.0/s rejected fastIncoming0.0/srequests per secondEmbed– ms–, p50Semantic cache–similarity –– of – keptCircuit breaker0 of 0 failures0 trips so farClosedBatch queue– waitingp50 wait – msbatching offModel– msbatches of –Response– msend to end, p50p99 – ms
  1. Incoming
    requests per second
    0.0/s
  2. Embed
    – embedder, p50
    – ms
  3. Semantic cache
    hit share, similarity –
    –
  4. Circuit breaker
    0 of 0 failures
    Closed
  5. Batch queue
    waiting, p50 wait – ms
    –
  6. Model
    per batch of –
    – ms
  7. Response
    end to end, p50
    – ms

Latency, p95

  • End to end
  • Model call
  • Cache hit
Waiting for data. Send some requests from the playground below.
Each point covers the last five seconds. A gap between end to end and model call is time spent queueing, not computing.

Requests per second, by who answered

  • Model
  • Cache
  • Degraded
  • Rejected
Waiting for data. Send some requests from the playground below.

Batch sizes, last minute

0 batches
No batches yet.
Dispatched because the batch filled: 0. Because the window closed: 0.

Try it

Questions go through the same endpoint as any client. Ask one, then ask it again in other words.

Try a preset, then its rewording. The second answer should come from the cache in about a millisecond.

Background traffic

Break it

Make the model fail or slow down and watch the breaker open, the cache take over, and the probes bring it back.

Request log

One structured line per request, the same lines the server writes to stdout.

TimeServed byCacheBatchQueue msModel msTotal msInput
Nothing logged yet for this filter.