Tachyon
The serving layer in front of the model: batching, a semantic cache and a circuit breaker, live.
Not connected
Request path, live
Every request is embedded and checked against the semantic cache. Only misses reach the breaker, the batch queue and the model. Line weight shows where the last five seconds of traffic went.
- Incoming
requests per second0.0/s - Embed
– embedder, p50– ms - Semantic cache
hit share, similarity –– - Circuit breaker
0 of 0 failuresClosed - Batch queue
waiting, p50 wait – ms– - Model
per batch of –– ms - Response
end to end, p50– ms
Latency, p95
- End to end
- Model call
- Cache hit
Waiting for data. Send some requests from the playground below.
Requests per second, by who answered
- Model
- Cache
- Degraded
- Rejected
Waiting for data. Send some requests from the playground below.
Batch sizes, last minute
0 batchesNo batches yet.
Try it
Questions go through the same endpoint as any client. Ask one, then ask it again in other words.
Try a preset, then its rewording. The second answer should come from the cache in about a millisecond.
Background traffic
Break it
Make the model fail or slow down and watch the breaker open, the cache take over, and the probes bring it back.
Request log
One structured line per request, the same lines the server writes to stdout.
| Time | Served by | Cache | Batch | Queue ms | Model ms | Total ms | Input |
|---|---|---|---|---|---|---|---|
| Nothing logged yet for this filter. | |||||||