Observability costs creep up on you. Someone adds custom metrics, you instrument a new service, enable distributed tracing "just for debugging" — then you get the invoice. At Zendrop, our Datadog bill had grown to the point where it was worth optimizing. Not because observability isn't valuable — but because we were paying for visibility we weren't using.
We started by checking which dashboards people actually open. We had dozens; the team used about six. Then logs: we were shipping everything including debug-level logs from services that hadn't had a bug in months. Switched to error/warning by default, info only for active development, debug on-demand with auto-expiry. Then custom metrics: found services emitting high-cardinality tags (user IDs, order IDs) generating thousands of unique series from a single metric. Finally, trace sampling: 100% of errors and slow requests, statistical sample of everything else.
Result: roughly 50% cost reduction. Same ability to detect and debug incidents. Observability needs the same hygiene as code — periodic review, removal of dead things, clear ownership. Cut the noise, keep the signal, the bill follows.