01Why Observability Is Its Own Discipline
The previous concept deep-dives are about building systems: how to choose a database, how to shard, how to load balance. Observability is about operating them. It's the discipline of being able to ask new questions about what your system is doing in production, especially when something is wrong and you don't yet know why.
Most candidates conflate this with monitoring. Monitoring tells you whether a known state is healthy: is CPU below 80%? Is the error rate under 1%? Observability is broader: when a user reports that the checkout flow is slow on Tuesdays at 3 PM but only for European customers using Safari, can you actually answer why? Monitoring covers known failures. Observability covers everything else.
The depth in observability interviews lives in three areas. First, knowing the three pillars and their costs: metrics, traces, and logs aren't equally cheap or equally useful, and the right balance matters. Second, SLOs and error budgets: the operational primitive that turns abstract reliability goals into concrete deployment decisions. Third, alerting strategy: the difference between paging on symptoms (user-visible problems) and paging on causes (resource utilization), and why most teams alert on the wrong things.
This page covers all three. Tools are mentioned in passing, but the focus is on the underlying decisions, because the tools change every few years and the decisions don't.
The Senior Move
The senior signal in observability interviews isn't naming Datadog or Prometheus. It's recognizing that monitoring and observability are different tools for different problems, that the three pillars have different cost-and-value characteristics, and that SLOs change how teams operate beyond just being a number on a dashboard. Naming these distinctions explicitly is what separates senior candidates from "we'd add monitoring" candidates.
02Monitoring vs Observability: The Actual Distinction
These words get used interchangeably, but they describe different practices. Naming the distinction precisely is a depth signal.
Monitoring
Monitoring asks pre-defined questions: is the error rate below threshold? Is latency within target? Is the disk full? You set up the questions in advance, the system reports yes or no, and alerts fire when an answer crosses a threshold. Monitoring is great for known failure modes; it's how you keep the lights on.
The limitation: monitoring only answers questions you knew to ask. If users start reporting a problem you didn't anticipate, monitoring tells you nothing useful. The dashboards are green. The alerts haven't fired. And yet something is wrong.
Observability
Observability lets you ask new questions about your system after the fact. The data you collected (metrics, traces, logs) is rich and high-cardinality enough that you can slice it by dimensions you didn't anticipate. "Show me p99 latency for European users on Safari, on requests to /checkout, between 2 PM and 4 PM, grouped by feature flag." Monitoring couldn't answer this. Observability can.
The difference comes down to what data you collect and how. Monitoring records aggregated metrics ("error rate per minute"). Observability records detailed events ("this specific request from this specific user took this long, hit these services, returned this error"). When you need to ask a new question, you can re-aggregate the detailed data; you can't disaggregate the summary.
The honest answer for most teams
Most production teams do monitoring and call it observability. They have dashboards (showing pre-defined metrics) and alerts (firing on pre-defined thresholds), but they can't actually slice their data by arbitrary dimensions when something novel breaks. This is fine for known failure modes. It's not enough for unknown ones.
The interview move: when asked about observability, distinguish it from monitoring explicitly. "Monitoring catches known failure modes through dashboards and threshold alerts. Observability is about being able to debug novel problems by querying high-cardinality event data after the fact. Most production setups need both, but they're not the same thing."
03The Three Pillars: Metrics, Traces, Logs
The standard framing of observability is "the three pillars": metrics, traces, and logs. Most prep material describes them as if they're equivalent and you collect all three. They're not equivalent, and the right balance is opinionated.
One Incident, Three Perspectives
The same checkout-slowness incident, viewed through metrics (error rate spike), traces (slow database call inside cart-svc), and logs (connection pool exhausted). Each pillar reveals a different layer of the truth.
Metrics
Cheapest · aggregate
Pre-aggregated numerical measurements over time: request count, error rate, latency percentiles, queue depth. Stored efficiently because they're aggregated at write time. The thing your dashboards and alerts run on.
What they answer"How is the system doing right now?" Great for known failure modes and trend detection. Bad for debugging specific user complaints.Traces
Mid cost · per-request
Per-request records of how a request flowed through the system: which services it hit, how long each step took, where errors occurred. A trace is one request's journey across distributed services.
What they answer"Where in the system did this slow request spend its time?" The pillar most teams under-invest in despite being the most useful for distributed systems.Logs
Most expensive · verbose
Timestamped event records emitted by application code: "user X did Y," "request Z failed because of W." High volume, high storage cost, often noisy. Most useful when paired with structured fields you can search.
What they answer"What specifically happened during this request?" The deepest detail, but also the most expensive to store and search at scale.The right balance (and why most teams get it wrong)
Most teams over-invest in logs and under-invest in traces. Logs feel safe because they capture everything; traces feel optional because applications work without them. This is backwards. In a distributed system, traces are how you understand how a request flowed, which is the question that gets asked most often during incidents.
The healthier balance:
- Metrics first. Cheap to store, drives dashboards and alerts. Aim for ~100 high-quality metrics per service rather than thousands of low-value ones.
- Traces for distributed paths. Sample at 1-10% of requests for high-traffic services; trace 100% of error or slow requests. Tools like OpenTelemetry have made this much cheaper to set up than it used to be.
- Logs sparingly. Structured logs (JSON, with consistent fields) are useful. Unstructured logs ("user did the thing") are mostly noise. Cut volume aggressively; keep what you need.
The interview move on the three pillars: name them, but also name the cost-and-value tradeoff. "Metrics for trends and alerts, traces for distributed debugging, logs for the specific event detail. Most teams over-collect logs and under-trace; the healthier balance flips that."
04SLIs, SLOs, and Error Budgets
This is the operational primitive that distinguishes mature production teams from teams that just have dashboards. Naming it explicitly in interviews is a staff-level signal because it changes how you think about reliability from "make it as reliable as possible" to "stay within a measurable target."
The three terms
| Term | What it is | Example |
|---|---|---|
| SLI | Service Level Indicator. A measurement of some aspect of service quality. | "Fraction of requests that return success in under 200ms" |
| SLO | Service Level Objective. A target value for the SLI over a time window. | "99.9% over a rolling 30-day window" |
| Error Budget | The allowable failure rate implied by the SLO. The complement of the target. | "0.1% of requests can fail per 30 days · ~43 minutes of downtime" |
Why error budgets matter operationally
The error budget changes the conversation from "should we deploy?" to "do we have budget for the risk?" When the error budget is intact, the team can deploy aggressively, take risks, ship features. When the error budget is being burned too fast, the team slows down: stricter testing, rollback faster, prioritize reliability work over features.
This is the operational discipline. Without an error budget, "is the system reliable enough?" is a subjective argument between teams. With one, it's a number. The product team and the platform team are both bound by the same target. Disagreements get resolved by data, not opinion.
Burn rate and multi-window alerting
The naive alert on an SLO would be "page me when we miss the SLO." That's too late: the budget is already gone. The mature pattern is burn rate alerting: page when the budget is burning fast enough that, if it continues, the budget will be exhausted within some window.
Two-window alerting is common: a fast-burn alert (consuming the 30-day budget at 10x rate over 1 hour) for sudden incidents, and a slow-burn alert (consuming the budget at 2x rate over 6 hours) for gradual degradations. The first catches outages; the second catches the slow drift toward a degraded state that nobody notices until it's too late.
What SLIs to actually pick
Pick SLIs that reflect user experience, not internal proxies. The four golden signals (from the Google SRE book) are a good starting point:
- Latency. How long requests take. Usually as a percentile (p99 latency under 300ms).
- Traffic. Demand on the service. Requests per second, often as context rather than as an alerting metric.
- Errors. Rate of failed requests. Critical: define "failure" by user-visible outcomes, not by HTTP status codes alone.
- Saturation. How full the system is. CPU, memory, disk, queue depth. Useful for capacity planning more than for alerting.
For most user-facing services, latency and errors are the two most important SLIs. Saturation matters when you're running close to capacity. Traffic is usually a denominator (errors per request, latency per request) rather than something you alert on directly.
The Interview Move
"How would you measure reliability?" Strong response: "We'd define an SLI as the fraction of requests returning success in under 200ms, set an SLO of 99.9% over 30 days, and the error budget gives us roughly 43 minutes of allowed badness per month. Burn rate alerting fires when we're consuming the budget too fast: a fast-burn alert at 10x burn over 1 hour for outages, a slow-burn at 2x over 6 hours for gradual degradation." That sentence covers the framework, the alerting strategy, and signals real operational thinking.
05Alerting: Symptoms vs Causes
Most production teams alert on the wrong things. The wrong thing: paging on resource utilization, internal service health, low-level failure modes. The right thing: paging on user-visible symptoms.
The symptom-vs-cause distinction
A symptom is what users experience: errors returned, slow responses, features that don't work. A cause is what produced the symptom: high CPU, low memory, connection pool exhaustion, dependency outage. The two are correlated but not identical. A cause without a symptom is fine (the system is handling it). A symptom without an obvious cause is the worst kind of incident, but still better to know about.
Alerting on symptoms means paging on user-visible problems: "error rate is above 1%" or "p99 latency is above 500ms" or "the SLO error budget is burning at 10x the sustainable rate." When this fires, something is genuinely wrong that users will notice.
Alerting on causes means paging on internal indicators: "CPU is above 80%" or "memory is at 90%" or "the database has 50 active connections." When this fires, something might be wrong, or the system might be handling it fine, or you might be near a threshold that doesn't actually matter. These pages create alert fatigue without commensurate value.
Why teams alert on causes anyway
Two reasons, both bad. First, cause-based alerts are easier to set up: you can buy them off the shelf from any monitoring tool. Symptom-based alerts require you to know what your users actually experience, which requires instrumentation and thought. Second, cause-based alerts let you "predict" problems before users see them, which sounds good but in practice means paging on noise.
The right approach is symptom-based alerting paired with rich diagnostic data (traces and logs) so you can investigate causes after a symptom fires. The page tells you to look; the diagnostic data tells you why.
Alert fatigue is operationally fatal
Every false-positive alert that wakes someone up trains the team to ignore alerts. Every alert that fires without clear action erodes trust in the alerting system. Production teams with too many alerts eventually hit a state where genuine incidents get acknowledged and ignored because the on-call assumed it was another false positive.
The fix is to be ruthless about what gets paged on. The bar should be "if this fires, someone needs to take action right now, and the action is clear." Anything that doesn't meet that bar belongs on a dashboard or in a daily report, not on a pager. Most production teams have 10x more alerts than they should. Cutting them to a small set of high-quality, symptom-based alerts is one of the highest-leverage operational improvements available.
Alert on what users feel, not what your servers feel. The page should mean "users are experiencing a real problem right now." Anything else belongs on a dashboard.
06Dashboards vs Exploration
Two different uses of observability data, often confused. Dashboards are for at-a-glance status checks of pre-defined views. Exploration is for ad-hoc investigation when something novel happens.
Dashboards
A dashboard shows pre-defined metrics with pre-defined views. It's optimized for being read quickly: "is the system OK right now?" The good ones have a small number of charts (5-15), each communicating something specific, organized hierarchically (high-level service health at the top, deeper diagnostics below). The bad ones have 40 charts and tell you nothing.
Dashboards are great for known states: are the SLOs healthy? Is traffic in the expected range? Are the dependencies OK? They're useless for novel problems because the dashboard wasn't built to answer the question you have right now.
Exploration
Exploration is the act of querying your observability data interactively to answer questions you didn't plan for. "Why are European Safari users seeing slow checkouts?" requires slicing the trace data by region, by user agent, by endpoint, by latency percentile, all in real time as you investigate.
Tools that support exploration well: Honeycomb, Lightstep (now ServiceNow), various OpenTelemetry-based stacks. Tools that are dashboard-first (Datadog dashboards, classic Grafana setups) often struggle here because their data model is optimized for pre-aggregation rather than ad-hoc query.
The interview move: name both uses. "Dashboards are for known states; exploration is for unknown ones. Production setups need both. The pattern that distinguishes mature teams is being able to slice their data by arbitrary dimensions in real time, not just look at pre-built charts."
07The Production-Readiness Probe
"Walk me through your monitoring strategy" or "how would you make sure this system is production-ready?" is one of the most common observability questions in senior interviews. It's where the page should land most directly.
The strong response, structured
A complete answer covers four dimensions:
- SLOs and error budgets. What does "good" look like quantitatively? What's the latency target, the error rate target, the time window? What happens when we burn through the budget?
- The three pillars, balanced. What metrics, traces, and logs do we collect? What's the sampling strategy for traces? What's the retention for each? Are the metrics low-cardinality enough to aggregate cheaply but useful enough to debug with?
- Alerting strategy. What pages, when, and to whom? Are we alerting on symptoms (user-visible) or causes (internal)? What's the on-call rotation? What's the escalation path?
- Runbooks and game days. When the alert fires, what does the on-call do? Are there written runbooks? Have we practiced incident response with game days or chaos exercises? Is the documentation actually updated?
Hitting all four dimensions in a structured response is the staff signal. Most candidates hit one or two and stop. Going through all four shows that production-readiness is a system, not a list of tools.
What "production-ready" actually means
Production-ready isn't a feature checklist. It's the ability to detect, diagnose, and recover from problems faster than they impact users. Specifically:
- Detect. When something breaks, do we know within minutes (not hours)? Are we paged automatically, or do we find out from user reports?
- Diagnose. When the page fires, can the on-call figure out what's wrong in a reasonable time (10-30 minutes for most incidents)? Or does diagnosis require waking up the team?
- Recover. Once we know what's wrong, can we fix it without waiting for engineers to write new code? Are there documented runbooks, deployable configurations, feature flags, or rollback procedures?
The interview move: don't list tools. Frame production-readiness as detect/diagnose/recover and show how observability supports each. "Detection comes from symptom-based alerting on SLOs. Diagnosis comes from traces and high-cardinality logs. Recovery comes from runbooks, feature flags, and the ability to roll back. Observability is the foundation, but it's not the whole story."
08Failure Modes
Failure 01
Cardinality explosion
Engineers add high-cardinality dimensions to metrics: per-user, per-request-id, per-arbitrary-string. The metrics backend's storage explodes because each unique combination of dimensions is a separate time series. Costs balloon. Queries slow down. Eventually the metrics system itself becomes the problem.
The fix is to push high-cardinality data to traces or logs instead of metrics. Metrics should have low cardinality (single-digit unique values per dimension); traces and logs are designed for the per-request detail. Discipline at the instrumentation layer prevents the explosion; bucketing or aggregating high-cardinality dimensions at write time is a fallback.
Failure 02
Logs as the only debugging tool
The team has no traces and minimal metrics. When something breaks, they grep through logs trying to reconstruct what happened. The logs are unstructured, so grep is the best they have. Incidents take hours to diagnose because reconstructing a request's path across services from log lines is genuinely hard.
The fix is to instrument traces. OpenTelemetry has made this much cheaper than it used to be: instrumenting incoming requests and outgoing calls with a few hundred lines of code transforms the debugging experience. The cost of adding traces is much smaller than the cost of every incident taking hours to diagnose.
Failure 03
Alert fatigue from cause-based alerting
The team has alerts on every internal metric: CPU, memory, queue depth, connection count. Every alert has fired hundreds of times for non-issues. The on-call has trained themselves to ignore the pager because most pages are noise. Then a real incident fires the same kind of alert and gets ignored.
The fix is to delete most alerts and replace them with a small set of high-quality symptom-based alerts. Anything that doesn't require immediate action goes on a dashboard, not the pager. The team's trust in the alerting system is more valuable than the marginal "predictive" value of the cause-based alerts.
Failure 04
SLOs without consequences
The team defines SLOs because the SRE book says to. The dashboard shows SLO compliance. But when the error budget is burned, nothing changes: the team keeps deploying, the priorities don't shift, the SLO is decorative. After a few quarters, nobody trusts the SLOs because they're not connected to operational decisions.
The fix is to make the error budget actually matter. When budget is healthy, deploy aggressively. When budget is burned, slow down: stricter review, fewer deploys, prioritize reliability work over features. The connection between budget and behavior is what makes the SLO useful; without that connection, it's a vanity metric.
09How Observability Interacts With Other Concepts
- Observability × Load balancing. The LB sees every request and is the natural source of high-level traffic metrics. Load balancing covers what the LB can and can't tell you about backend health, including the partial-failure modes that health checks miss.
- Observability × Replication. Replication lag is one of the metrics you must monitor in production. Lag spikes are an early warning of capacity issues, network problems, or hot keys. Replication and consistency covers the underlying dynamics.
- Observability × Rate limiting. Rate limit rejections are a metric to alert on. A spike in 429s tells you something specific: either real abuse, a buggy client, or limits set too tight. Rate limiting without observability is operating blind. Rate limiting covers the integration.
- Observability × Message queues. Queue depth is a key metric for backpressure. Consumer lag is a warning signal. The DLQ depth is a health indicator. Message queues covers the operational concerns.
- Observability × Database selection. Different databases expose different metrics. Postgres exposes lock waits and query plans; DynamoDB exposes throttling events. Database choice has observability implications. Database selection covers the broader pattern.
For more cross-concept interactions, see the concepts library hub.
10Practice Scenarios
Three scenarios. Read the setup. Decide your approach before opening the reveal.
Scenario 01
Your service's p99 latency has been climbing for two weeks. No alerts have fired. Users are starting to complain. How do you investigate, and what does this tell you about your observability setup?
Service is a payment API serving roughly 5K requests per second. Average latency is steady at 80ms. p99 has grown from 250ms to 900ms over two weeks. Existing monitoring includes basic metrics dashboards and CPU/memory alerts on the underlying instances.
How to think about this
Two distinct problems here: the immediate diagnostic, and the systemic gap.
The immediate diagnostic. p99 climbing while average is steady means the slow tail is getting worse but the median user is fine. Likely causes: a specific endpoint or path is degrading, a specific dependency is getting slower, or some segment of traffic is being routed inefficiently. The investigation starts with traces: pull traces for slow requests, look for what they have in common. Group by endpoint, by region, by feature flag, by downstream service called. Whatever dimension separates slow from fast is the lead.
The systemic gap. The fact that no alerts fired during a 4x p99 climb is the actual problem. The team is alerting on cause (CPU, memory) instead of symptom (user-visible latency). The fix is to add SLO-based alerting: "p99 latency above 500ms" or "burn rate on the latency SLO above 2x" would have fired weeks ago. The cause-based alerts didn't fire because resources were fine; the symptoms got worse anyway.
Strong answer: "Trace-based investigation grouped by endpoint and dependency to find the tail's source. The bigger fix: add SLO-based alerts on user-visible latency. The current setup alerts on resource health, which missed this entirely. p99 doubling without firing a page is an alerting strategy failure, not just a service performance problem."
Scenario 02
A new microservice is going to production next week. The team has Datadog set up. They ask, "is this enough monitoring?" What do you say?
The service is a new feature for personalized recommendations, calling several internal services and an LLM API. Datadog auto-instrumentation is enabled, capturing default metrics. No SLOs defined yet. No alerts configured beyond Datadog's defaults.
How to think about this
"Datadog is set up" is the start, not the finish. Auto-instrumentation gives you metrics; it doesn't give you observability strategy. Four things missing:
1. SLOs. What does "good" look like quantitatively? Latency target, error rate target, availability over what window? Without SLOs, you can't decide when something is wrong vs slow vs broken. Pick 2-3 SLIs based on user experience (latency, error rate, possibly recommendation quality) and a target for each.
2. Symptom-based alerts. Default alerts in monitoring tools are usually cause-based (high CPU, low memory). Replace these with SLO-based burn rate alerts so the on-call gets paged when users are affected, not when servers are warm.
3. Trace coverage. A microservice calling several other services and an LLM is a distributed system. Auto-instrumentation usually gives you spans for the standard libraries; verify it's actually capturing the LLM call latency, the inter-service calls, and the database hits. Sampling strategy: 100% of errors and slow requests, 1-5% of normal traffic.
4. Runbooks. When the alert fires, what does the on-call do? Without a written runbook, every incident is a fresh investigation. Even a basic "check these dashboards, look at these logs, contact this team if it's still broken" is much better than nothing.
Strong answer: "Datadog gives you instrumentation. Observability strategy includes SLOs, symptom-based alerts, trace coverage of the distributed paths including the LLM call, and at least a basic runbook. Auto-instrumentation isn't observability; it's the data you observe with."
Scenario 03
A team's monthly observability bill has tripled in six months. They want to cut costs. Where do you look?
Datadog (or similar) bill is now $30K/month. The team can't articulate which metrics, traces, or logs are worth what. The CFO wants a 50% cut. The team is worried that cutting observability will cause incidents.
How to think about this
The cost growth is almost certainly driven by one or two patterns. Investigate before cutting:
1. High-cardinality metrics. Metrics costs scale with unique time series. A single metric with a high-cardinality dimension (per-user, per-request-id, per-product) explodes into millions of series. Audit the metrics for cardinality; remove or aggregate the worst offenders. This often cuts the metrics bill in half.
2. Verbose logs. Logs costs scale with volume. Audit log volume by service. The biggest offenders are usually one or two services emitting verbose debug-level logs in production. Move to structured logs with appropriate levels; cut info-level logs that aren't actually consulted.
3. Trace sampling. If traces are 100% sampled at high traffic, the cost is huge. Drop to 1-5% sampling for normal traffic, 100% for errors and slow requests. The diagnostic value barely changes; the cost drops dramatically.
4. Retention. Most teams retain everything for 30-90 days "just in case." Most incidents are diagnosed within 24-48 hours. Tier the storage: high-resolution data for a week, downsampled for longer windows.
Strong answer: "Don't cut blindly; audit. Cardinality is usually the metrics killer; volume is usually the logs killer; sampling is usually the traces killer. Cut high-cardinality dimensions, drop info-level logs, sample traces at 1-5% of normal traffic with 100% on errors. This gets you to 50% cuts without losing diagnostic capability for actual incidents."
11Observability FAQ
Datadog or Prometheus?
Different tools for different teams. Datadog is a hosted service that handles ingestion, storage, dashboards, and alerts in one product; you pay for the convenience. Prometheus is open-source and self-hosted, with rich querying via PromQL but more operational burden. Most cloud-native teams default to Prometheus or its hosted variants (Grafana Cloud, AWS Managed Prometheus). Most product teams default to Datadog or Honeycomb because they don't want to operate the observability stack. The choice is operational fit, not capability.
What's OpenTelemetry and do I need it?
OpenTelemetry (OTEL) is the modern standard for instrumenting application code. It's vendor-neutral: you instrument once with OTEL APIs, then send the data to whichever backend you choose. It has effectively replaced earlier vendor-specific SDKs. New systems should use OTEL by default. Existing systems with vendor-specific instrumentation can migrate gradually. The interview move is naming OTEL as the default for new instrumentation.
How do I sample traces without losing important ones?
Two approaches. Head-based sampling: the decision is made at the start of the trace, before you know whether it's interesting. Cheap but loses information. Tail-based sampling: the decision is made after the trace completes, so you can keep all errors and slow requests while sampling normal traffic. More accurate but requires buffering all traces briefly. Modern collectors (OTEL collector, Honeycomb's Refinery) implement tail-based sampling. Most teams should use it.
What's the difference between an SLA and an SLO?
SLA is contractual: a promise to customers about service quality, with penalties if breached. SLO is internal: a target the team operates against, usually stricter than the SLA. The pattern: SLA is what you promise externally; SLO is what you aim for internally; the gap between them is your safety margin. Most product teams have SLOs without SLAs (no formal customer contract); some enterprise products have both.
How do I instrument an LLM call?
Treat it like any external API call but capture additional dimensions: model name, prompt token count, completion token count, total latency, time-to-first-token (for streaming), error/timeout. The token counts are critical because they drive cost; you want to know which paths are token-expensive. Most LLM SDKs and OTEL plugins now expose this directly. The dedicated AI infrastructure deep-dive covers more.
Should I log everything just in case?
No. Logging everything is expensive in storage, slow to search, and noise-heavy. The pattern that works: emit structured logs at INFO level for important state changes (request started, request completed, error occurred), DEBUG level only when explicitly enabled, and rely on traces for the per-request detail rather than logging it. The mental model: logs are for events you want to know about; traces are for understanding request flow; metrics are for trends. Don't conflate them.
How do I get visibility into third-party services I depend on?
Wrap every external call with timing and error tracking. Most tracing instrumentation does this automatically: an outgoing HTTP call becomes a span with timing and status. Then look at error rates and latency for each external dependency as part of your dashboards. The third-party itself may publish a status page; integrate that into your incident response if relevant. The lesson: your observability is only as good as your weakest dependency's, but at least you can detect problems quickly.
What's chaos engineering and is it worth it?
Chaos engineering is the practice of deliberately introducing failures in production to test recovery. The famous example is Netflix's Chaos Monkey, which randomly kills instances during business hours. The goal is to find weaknesses in your system before users do. It's worth it for mature teams with strong observability and incident response; premature for teams that haven't yet hit basic SLO and alerting maturity. Don't introduce chaos until you can detect it cleanly.
