Product updates, postmortems, and deep dives into how we run telemetry at scale.
Product
Introducing streaming alert rules v2
Alert rules used to evaluate on 30-second ticks. With rules v2, evaluation happens in-stream: thresholds, absence detection and anomaly bands now fire within 2 seconds of the triggering data point landing — a 15× improvement in mean time to notify.
Rules v2 also adds composite conditions (cpu.temp > 85 AND battery < 20) and per-rule notification budgets, so a flapping fleet can't drown your on-call channel. Existing rules migrate automatically; nothing to change on your side. Full syntax is in the API docs.
MKMarta Keller — Product Lead, Alerting
Engineering
How we cut p95 query latency below 250ms
Six months ago our p95 GTQL latency sat at 890ms and climbing. The fix wasn't one big rewrite — it was three boring ones: columnar block caching keyed on query-shape, pushing GROUP BY time() aggregation down to the storage nodes, and a small LRU for repeated dashboard refreshes.
The result: p95 of 212ms across the last 30 days, with the 99th percentile down 4×. The post walks through the flamegraphs that got us there and the two optimizations that didn't work.
DKDevon Krishnan — Staff Engineer, Query
Engineering
Ingesting 2 billion data points a day over MQTT
About 60% of our traffic arrives over MQTT from customer brokers and device gateways. This post covers how we shard topics across the broker cluster, why we moved per-device certificate validation out of the hot path, and the back-pressure protocol that keeps a slow storage tier from ever disconnecting a device.
Spoiler: the hardest bug was a 40-byte keepalive packet that arrived exactly every 29,999 milliseconds.