Honeycomb

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 App Interface, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-20 14:04 UTC

   We are currently investigating the degradation of various services in our US region.

2. investigating 2026-10-20 15:18 UTC

   We've identified and isolated the impacted service and are seeing recovery in query execution. SLO processing remains degraded.

3. resolved 2026-10-20 16:26 UTC

   SLO processing has fully recovered. All services are now functional.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-10-20 13:30 UTC

   From 13:57 to 14:09 UTC, event ingest in Honeycomb's US production environment was degraded by an concurrent incident . During this window, approximately 53% of incoming events received 5xx errors.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 App Interface, ui.eu1.honeycomb.io - EU1 App Interface

Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-08-21 15:21 UTC

   Canvas is not loading for customers in the US and EU regions and we have identified what we believe to be the cause and are rolling out a fix.

2. monitoring 2026-08-21 15:25 UTC

   We have deployed the fix and are seeing recovery of normal Canvas functionality.

3. resolved 2026-08-21 15:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-08-10 13:01 UTC

   From 11:36UTC to 12:43UTC, queries in US1 that read older data returned errors or timed out. Queries over recent data were un-affected, and no telemetry is being lost. The issue has been identified, and a fix is in place. We are currently monitoring the situation to confirm...

2. resolved 2026-08-10 20:35 UTC

   The fix is in place and we've confirmed resolution. Queries and SLOs are processing normally.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-07-31 14:30 UTC

   Between 10:30 and 11:00 AM EDT, we experienced elevated querying failures and slowness. The impact has subsided, and we are currently investigating and monitoring the situation.

2. postmortem 2026-08-21 13:18 UTC

   Starting at 9:50 AM Central Time on July 31st we received alerts around querying health and latency. The cause was eventually tracked down to a clash of two different factors. First, we have been introducing Lambda Managed Instances \(LMI\) into our query architecture, which...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Querying, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-07-27 13:13 UTC

   We are seeing triggers intermittently failing in the EU region. We are actively investigating the cause.

2. investigating 2026-07-27 13:52 UTC

   We are continuing to see issues with triggers and are also noticing issues with querying. We are investigating for the cause of both.

3. monitoring 2026-07-27 16:01 UTC

   We have identified the source of the issue, applied a temporary fix, and are working on a permanent fix. We continue to monitor the situation.

4. monitoring 2026-07-27 19:46 UTC

   We are seeing a regression in performance with partial degradation of both triggers and querying. We are working to remediate.

5. monitoring 2026-07-27 22:51 UTC

   The partial degradation has been resolved for all customers except for those specifically contacted. We are working on mitigation measures so that we can fully restore trigger functionality for everyone.

6. monitoring 2026-07-28 18:21 UTC

   We are continuing to monitor for any further issues.

7. resolved 2026-07-28 18:30 UTC

   The degradation in querying and triggers has been resolved and we're back to full functionality.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: www.honeycomb.io

Failure mode: CDN or edge layer failed or served bad content. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-07-02 18:10 UTC

   We are currently investigating this issue.

2. monitoring 2026-07-02 18:13 UTC

   Clearing the CDN cache appears to have solved the issue.

3. resolved 2026-07-03 00:31 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-06-29 22:07 UTC

   Starting at 14:24 PT, there was an issue with the Activity Log causing events to stop being processed. You may notice a gap starting at that time. We are slowly backfilling these events and do not expect any data loss to occur. We expect the full data log to be restored around...

2. resolved 2026-06-30 00:47 UTC

   Backfill of the Activity Log events has completed. All events during the affected period should be present.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.eu1.honeycomb.io - EU1 Event Ingest, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-25 18:10 UTC

   We are investigating an issue with delayed ingest in the EU region. SLOs and Trigger evaluations may also be delayed.

2. monitoring 2026-06-25 18:31 UTC

   We are monitoring the situation. Events may have been delayed between 17:53 and 18:04 UTC.

3. resolved 2026-06-25 20:41 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.honeycomb.io - US1 Event Ingest, ui.honeycomb.io - US1 Trigger & SLO Alerting, api.eu1.honeycomb.io - EU1 Event Ingest, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-06-24 18:14 UTC

   We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

2. monitoring 2026-06-24 18:22 UTC

   We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

3. monitoring 2026-06-24 18:31 UTC

   Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

4. resolved 2026-06-24 19:04 UTC

   All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in...

5. postmortem 2026-07-02 20:10 UTC

   On June 24, we experienced approximately 30 minutes of severe data ingestion degradation in Honeycomb’s US and EU instances. During this degradation, Honeycomb rejected a significant percentage of inbound telemetry, and customers would have seen up to a half-hour gap in ingested...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-06-19 16:39 UTC

   We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.

2. monitoring 2026-06-19 17:18 UTC

   We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.

3. resolved 2026-06-19 17:21 UTC

   MCP Write Tools have been restored. MCP users may need to reconnect to see all available tools.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Enhance, ui.eu1.honeycomb.io - EU1 Enhance

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-06-18 19:10 UTC

   Beginning June 17th, enhance-related requests were failing to complete. We've since identified and deployed a fix, and enhance is now fully operational. Thank you for your patience.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Querying

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-17 14:01 UTC

   We are continuing to investigate intermittent slowness and query failures affecting our Production EU Region. We will provide an update as soon as we have more information.

2. monitoring 2026-06-17 15:29 UTC

   Querying in our Production EU Region is fully operational. We will continue to monitor for any abnormal behavior.

3. resolved 2026-06-17 21:14 UTC

   We have identified the root cause of the intermittent slowness and query failures affecting our Production EU Region. The issue was triggered by a large volume of dataset deletions that caused elevated database load, resulting in query instability. No data was lost during this...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Querying

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-17 12:01 UTC

   We are currently investigating the cause of slowness and query failures in our Production EU Region.

2. resolved 2026-06-17 12:32 UTC

   The issue is now resolved and querying is back to fully functional.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-06-11 15:30 UTC

   Requests to Honeycomb's Query Data and Management APIs in the production-eu1 environment encountered increased error rates between approximately 15:35 and 22:24 UTC. Event ingestion was not affected. The issue has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.eu1.honeycomb.io - EU1 Event Ingest, ui.eu1.honeycomb.io - EU1 Querying, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-10 20:58 UTC

   We are investigating an issue with delayed ingest in the EU region. Received events are still being stored, but there may be a delay in event retrieval in queries. SLOs and Trigger evaluations may also be delayed.

2. investigating 2026-06-10 21:28 UTC

   Queries should no longer be delayed. SLOs and Trigger evaluations remain impacted.

3. identified 2026-06-10 21:29 UTC

   The issue has been identified and a fix is being implemented.

4. resolved 2026-06-10 22:02 UTC

   All services are now healthy.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 App Interface, ui.eu1.honeycomb.io - EU1 App Interface

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-06-08 16:37 UTC

   We are currently investigating this issue.

2. resolved 2026-06-08 17:41 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 App Interface, ui.eu1.honeycomb.io - EU1 App Interface

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-06-03 20:50 UTC

   Canvas agent cannot load honeycomb internal tools.

2. resolved 2026-06-03 20:51 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.eu1.honeycomb.io - EU1 Querying

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-21 23:22 UTC

   We have identified and are working to resolve an issue that is causing query results to return inconsistent results.

2. investigating 2026-05-22 00:49 UTC

   Impact from this issue has concluded. For a period of time (different between US and EU instances, noted below), queries spanning data older than the most recent 2 hours with certain GROUP / WHERE clauses returned inconsistent results. Data was never lost, and reruns of the same...

3. resolved 2026-05-22 16:42 UTC

   This is an administrative update, marking the incident as Resolved. There has been no further impact since 23:20UTC on May 21 2026.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-05-08 08:00 UTC

   Activity log delay is rising, and is more than an hour behind due to a MySQL replica failover. Estimated recovery by 04:00 PDT.

2. monitoring 2026-05-08 10:13 UTC

   Backfilling of the past 2h30m of data is now in progress and should complete shortly.

3. monitoring 2026-05-08 10:21 UTC

   We believe that the activity log should now be caught up.

4. monitoring 2026-05-08 10:22 UTC

   We are continuing to monitor for any further issues.

5. monitoring 2026-05-08 10:40 UTC

   All activity log streams are now caught up, except for the query runs table, which should have all data since the start of the incident backfilled by 1600 PDT.

6. monitoring 2026-05-08 20:46 UTC

   Activity Log recovered at 07:00 UTC-07:00 today (about 6.5h ago). All streams are caught up.

7. resolved 2026-05-11 16:37 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-05-01 13:00 UTC

   The database failover that caused query issues earlier in the EU (https://status.honeycomb.io/incidents/n855d8kzp32y) has also had knock-on effects on our Activity Log beta feature. Due to recovery issues on the data replication flow used for this mechanism, activity log events...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-05-01 12:30 UTC

   At 12:50, our main database underwent an automated failover. This failover led to issues with some of our query engine connection pools, and caused queries to fail for roughly 10 minutes before self resolving.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Trigger & SLO Alerting, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-04-30 15:03 UTC

   We discovered that links in Slack triggers are not working and have identified the cause and are in process of implementing a fix

2. monitoring 2026-04-30 15:07 UTC

   The fix has been implemented and links on Slack triggers are functional again.

3. resolved 2026-04-30 16:37 UTC

   We are confident the issue is fully resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-20 16:56 UTC

   Activity Log events are interrupted, starting from around 15:51UTC. Customers may experience some data lost since the beginning of the incident.

2. resolved 2026-04-20 17:02 UTC

   Activity Log is back to operational, and most events should be recovered.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Activity Log, ui.eu1.honeycomb.io - EU1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-15 19:04 UTC

   Activity Log is experiencing a delay in processing events. We are working on remediating the issue.

2. resolved 2026-04-15 19:25 UTC

   Activity Log is ingesting events again. We had no loss in data.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-14 21:36 UTC

   Activity Log events may be delayed, starting from around 19:55UTC. We're actively working on remediation.

2. monitoring 2026-04-14 22:08 UTC

   A fix has been implemented and we are monitoring recovery.

3. resolved 2026-04-14 22:37 UTC

   All activity log data is now live and up to date.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.honeycomb.io - US1 Event Ingest

Failure mode: A configuration change caused the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-04-10 13:41 UTC

   We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.

2. identified 2026-04-10 13:46 UTC

   We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.

3. identified 2026-04-10 13:55 UTC

   We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.

4. monitoring 2026-04-10 14:19 UTC

   We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.

5. monitoring 2026-04-10 14:32 UTC

   The manual fix has been applied to all environments.

6. monitoring 2026-04-10 15:21 UTC

   We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.

7. resolved 2026-04-10 17:24 UTC

   We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-04-09 19:00 UTC

   We have detected and remediated an issue around our SLO processing pipeline. From 17:00 to 17:15 UTC, events were being delayed, and alerts may likewise have been a few minutes late. Processing has caught up and is currently within normal operational ranges.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Trigger & SLO Alerting, ui.honeycomb.io - US1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-09 11:52 UTC

   We have identified the cause and are working to resolve the issue.

2. identified 2026-04-09 20:24 UTC

   We have identified an issue with triggers and SLOs and are actively investigating. Customers may experience delays when creating or updating triggers and SLOs. A fix for the activity log delay has been implemented and is in the process of recovering. Customers may continue to...

3. identified 2026-04-09 21:03 UTC

   We have identified an issue with triggers and SLOs and have implemented a fix. Customers should no longer see degraded performance for triggers or SLOs. A fix for the activity log delay has been implemented and is in the process of recovering. Customers may continue to see...

4. identified 2026-04-09 23:43 UTC

   SLOs experienced a period of additional processing delays. A fix has been deployed and recovery is ongoing.

5. identified 2026-04-10 00:05 UTC

   SLO processing has fully recovered.

6. resolved 2026-04-10 19:10 UTC

   All activity log data is now live and up to date.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-04-09 10:12 UTC

   We experienced a partial querying outage from approximately 8:58 - 9:18 UTC. This outage may have resulted in some Trigger and SLO alert failures for the duration of the outage. The cause of the outage has been identified and mitigated.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 App Interface

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-03-18 18:07 UTC

   We are currently investigating this issue.

2. identified 2026-03-18 18:10 UTC

   We have identified the issue and are working on a fix.

3. resolved 2026-03-18 18:18 UTC

   The issue has been identified and we have shipped a fix.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-03-12 11:55 UTC

   We are investigating elevated error rates affecting queries in our US1 region. Our team is actively working on a resolution. We will provide updates as we learn more.

2. identified 2026-03-12 13:17 UTC

   We have rolled back a code change to our query engine, but our triggers service has not recovered. We are investigating.

3. resolved 2026-03-12 13:39 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.honeycomb.io - US1 Event Ingest

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-03-10 13:50 UTC

   We are currently seeing elevated errors in connecting to some of our databases, which results in low amounts of Ingest traffic being rejected. We are looking into correcting the situation.

2. monitoring 2026-03-10 13:56 UTC

   We have stabilized the cluster and are monitoring the state while investigating to understand what triggered the degradation. Overall, 0.04% traffic was rejected with an error due to this degradation during the 20 minute window we were impaired.

3. resolved 2026-03-10 14:14 UTC

   This incident is now resolved. We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22—13:44 UTC).

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Querying

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-02-18 23:07 UTC

   Between 22:10 and 22:40 UTC on Feb18, 2026 Honeycomb experienced a partial query outage in our EU environment. The incident has been resolved, and we are continuing to investigate the contributing factors.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.eu1.honeycomb.io - EU1 Querying

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-02-10 20:18 UTC

   We are aware of elevated errors in trigger evaluations and queries, and are working to resolve it

2. resolved 2026-02-10 20:30 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-01-22 23:27 UTC

   We are currently investigating what appears to be a transient issue related to querying and triggers

2. resolved 2026-01-22 23:42 UTC

   This appears to have been a minor transient network issue; we do not expect this to recur

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-01-15 15:46 UTC

   We observed a period of increased latency accompanied by a higher-than-normal error rate in some event ingest requests in the US. The service has since stabilized and is operating normally. We are continuing to monitor the system and are evaluating improvements to reduce the...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Activity Log, ui.eu1.honeycomb.io - EU1 Querying, ui.eu1.honeycomb.io - EU1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-01-06 22:14 UTC

   We are seeing a spike in querying errors and the Activity Log has delayed ingest of new activity. We have identified the issue and are working to resolve

2. identified 2026-01-06 22:37 UTC

   We have resolved the issue which impacted querying but are still working to get Activity Log up to date

3. identified 2026-01-06 23:05 UTC

   We are currently expecting to lose the last three hours of data for the beta activity log feature as part of remediation of this issue.

4. identified 2026-01-07 01:17 UTC

   We have restored basic activity log functionality, though some delays remain, and we will analyze further tomorrow during business hours.

5. identified 2026-01-07 19:53 UTC

   We have stabilized activity log. We have confirmed activity log entries lost from 4:54pm-7:04pm ET in the US and 9:43pm-11:55pm GMT in the EU. Event ingest was unimpacted. All systems are currently operating normally

6. resolved 2026-01-07 19:54 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: api.eu1.honeycomb.io - EU1 Event Ingest, ui.eu1.honeycomb.io - EU1 Querying, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting, ui.eu1.honeycomb.io - EU1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-12-05 20:49 UTC

   We are currently investigating this issue.

2. investigating 2025-12-05 21:19 UTC

   A subset of customer environments may see higher than usual error rates when sending events to api.eu1.honeycomb.io, and notifications for SLOs and Triggers may be delayed for that subset. We are continuing to investigate the issue.

3. investigating 2025-12-05 22:33 UTC

   We have identified that 0.23% of datasets are fully affected, and a larger percentage are seeing intermittent ingestion and query failures (500s at the API level). We are also investigating a replication error in our ingestion pipeline.

4. investigating 2025-12-05 23:46 UTC

   We are continuing the investigation after business hours to stabilize ingest and determine what work will be needed to fully recover. Known impact will be updated here as the situation changes. We will post at least one more update this evening.

5. investigating 2025-12-06 00:04 UTC

   Ingest, querying, SLOs, and Trigger alerting is back to normal. Activity Log is still impacted, we have identified the cause and are working to resolve. We are still evaluating the scope of the outage for ingest and expect to have a full answer for that posted during US business...

6. investigating 2025-12-06 07:30 UTC

   Automated systems are still working to catch up and Activity Log will remain offline until that that completes. We will post another update Saturday whether or not the Activity Log outage has been remediated.

7. investigating 2025-12-06 15:19 UTC

   We have temporarily disabled event ingestion for the EU Region, in service of restoring full functionality to our EU Kafka fleet. External API access will also be disabled. Additionally, during the ongoing outage, our Service Level Objectives feature has been impacted and down...

8. investigating 2025-12-06 16:15 UTC

   External API access is restored, event ingestion is still impacted

9. investigating 2025-12-06 19:20 UTC

   We are continuing to investigate issues with the Kafka cluster.

10. investigating 2025-12-06 21:23 UTC

    We are continuing to investigate issues with the Kafka cluster. We will continue providing updates every 2 hours unless there are significant changes.

11. identified 2025-12-06 21:51 UTC

    We have resumed ingest to api.eu1.honeycomb.io as we were able to restore partial service to the Kafka cluster.

12. identified 2025-12-06 22:08 UTC

    Ingest has been fully re-enabled, and SLOs and Triggers are now up-to-date.

13. identified 2025-12-07 00:10 UTC

    We have gotten the Kafka cluster to a stable-enough state, however it will still require remediation before the Activity Log is fully functional and it is back to a level of service sustainable until business hours. We will provide another update in 2 hours or as the situation...

14. identified 2025-12-07 02:18 UTC

    The Kafka cluster is still operational, but is not back to full capacity and resiliency. Work will continue tomorrow to ensure sufficient capacity, as well as during business hours to resume full Activity Log functionality. We will provide more updates tomorrow and as-needed if...

15. identified 2025-12-07 20:23 UTC

    Our Kafka cluster is still operational and we are continuing to work towards restoring full reliability. We will resume work during business hours to restore full Activity Log functionality. We will continue to update as work progresses.

16. identified 2025-12-08 21:52 UTC

    The Activity Log is operational but is experiencing ingestion delays, which means recent activity may not be reflected immediately. We are working to restore full functionality. Currently, a small subset of data for certain customers may not be queryable due to a partition...

17. identified 2025-12-09 01:02 UTC

    We continue to work on restoring full functionality to Activity Log and repair the partition causing certain customers to have subsets of unqueryable data. We will next provide an update during US business hours Tuesday.

18. identified 2025-12-09 02:01 UTC

    We fixed the issue that was causing subsets of temporarily missing data in queries against certain partitions. We continue to work on restoring full functionality to Activity Log. We'll provide another update during US business hours Tuesday.

19. identified 2025-12-09 02:05 UTC

    We are continuing to work on a fix for this issue.

20. identified 2025-12-09 22:40 UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we have a clear path to recovery, and remediation work is underway. Our next steps involve coordinated infrastructure changes that will allow us to resume Activity Log...

21. identified 2025-12-10 17:34 UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is actively working through the remaining steps required to...

22. identified 2025-12-11 01:08 UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is pausing for the evening and will resume working through the...

23. identified 2025-12-11 18:02 UTC

    We continue to work toward fully restoring the Activity Log. We’ve made meaningful progress on the underlying issues and continue to work through our remediation plan. Our engineering team is actively working through the next steps required to restore Activity Log processing.

24. identified 2025-12-12 00:09 UTC

    We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is actively working through the next steps required...

25. identified 2025-12-12 23:20 UTC

    We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is working through the next steps required to...

26. identified 2025-12-16 01:52 UTC

    As of today, our old EU cluster is still functional, yet in a bad state. We do not consider this outage over. However, after running a full evacuation in a pre-production cluster, we are now ready to run a production version of it. This will only be in the impacted EU cluster....

27. monitoring 2025-12-16 20:17 UTC

    We successfully created and migrated over to a new kafka cluster in the EU. We have resumed consuming events for our SLO and Service Maps features. No further data loss is expected for service maps. SLOs are caught up and are fully synchronized. Activity logs are still delayed...

28. resolved 2025-12-17 15:47 UTC

    As of 23:23UTC on December 16, the Activity Log has fully caught up and has remained up to date. All systems are now operational.

29. postmortem 2025-12-18 19:09 UTC

    This incident started on December 5th, and is one of the longest in Honeycomb history, having been actively worked on and closed only on December 17th. Due to its impact and duration, we wanted to offer a partial and preliminary report to explain, at a high level, what happened....

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Activity Log

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2025-11-25 22:51 UTC

   We have identified the cause and are working to resolve the issue.

2. identified 2025-11-26 05:02 UTC

   We are continuing to work on a fix for this issue.

3. monitoring 2025-11-26 05:07 UTC

   Due to issues with replication streams, the activity log will have a gap in its data. We estimate that a little over 6 hours of content will be missing, from 20:27 UTC until 2:43 UTC. We do not expect missing data to be backfilled. No other Honeycomb data or feature has been...

4. monitoring 2025-11-26 17:24 UTC

   We are continuing to work on catching up late data from yesterday night's replication issues and expect to complete this process in approximately 6 hours.

5. resolved 2025-11-27 00:14 UTC

   As of 23:00 UTC, all systems are back to synchronized.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: A zone, region or data center became unavailable. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2025-10-20 14:16 UTC

   Due to constrained EC2 instance capacity in the AWS us-east-1 region, we are choosing to allocate the capacity we have to incoming events and telemetry. As such, customers may see over 5 minute delays in our processing of - SLOs - Service Maps We do not expect a degradation of...

2. monitoring 2025-10-20 15:14 UTC

   SLO processing has recovered. Service Maps continues to be degraded but historical data can be queried.

3. monitoring 2025-10-20 15:30 UTC

   The networking issues in us-east-1 is affecting our query engine - customers may see errors or delays when running queries as a result.

4. monitoring 2025-10-20 17:39 UTC

   We’re currently observing that querying is seeing signs of recovery. We are continuing to monitor the situation

5. monitoring 2025-10-20 20:06 UTC

   AWS is starting to show signs of recovery. Querying remains partially impacted and may take longer to return results. We are continuing to monitor the situation.

6. monitoring 2025-10-20 23:08 UTC

   Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. We are continuing to monitor the situation.

7. monitoring 2025-10-21 00:11 UTC

   Honeycomb core functionality is operational. Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. This is our final update for the night unless the...

8. resolved 2025-10-21 15:41 UTC

   This incident has been resolved. At no point did we lose any customer data that hit our load balancers. Querying has been stable for hours, and all features that were degraded are functional.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying, ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. identified 2025-10-20 07:24 UTC

   We are currently experiencing degraded performance for queries that access older data stored in cold storage. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes queries for SLOs and...

2. identified 2025-10-20 08:14 UTC

   We are currently experiencing degraded performance for queries. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes queries for SLOs, Triggers, and Traces. Our team is closely monitoring...

3. identified 2025-10-20 08:46 UTC

   We are currently experiencing degraded performance. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes trace queries. SLO and Trigger alerts may be missed or delayed. Queries may return...

4. monitoring 2025-10-20 09:49 UTC

   We are beginning to see recovery across our systems. We will continue to monitor the situation in case of any regressions.

5. resolved 2025-10-20 10:04 UTC

   All affected systems have fully recovered.

6. postmortem 2025-10-22 22:52 UTC

   At 07:05 UTC on October 22, 2025, Honeycomb Platform On-Call received a page indicating that our [End to End tests](https://www.honeycomb.io/blog/how-honeycomb-uses-honeycomb-part-3-end-to-end-failures) that run every 5 minutes had failed. The responding engineer logged in to...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2025-10-08 14:26 UTC

   We are currently experiencing issues with our configuration with one of our AI providers. Features such as Query Assistant and our Calculated Field assistant are not functional, and some other parts of Honeycomb Intelligence may be partially degraded at this moment.

2. monitoring 2025-10-08 15:10 UTC

   We have made changes to our configuration and errors rates seem to have gone back to normal. We are keeping an eye on things to know the features remain stable.

3. resolved 2025-10-08 15:24 UTC

   The system has remained stable, we do believe everything to now be in order.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-09-25 00:07 UTC

   An outage with an upstream service provider is currently affecting Honeycomb's infrastructure. Functionality of the ServiceMaps feature is degraded at this time.

2. resolved 2025-09-25 01:22 UTC

   Service Maps functionality has been restored.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2025-08-28 04:30 UTC

   A PagerDuty outage prevented some Trigger and SLO notifications from reaching your workspaces. The issue has since been resolved, and notification systems are now operating normally. Impact Window: 21:54 PT – 23:27 PT

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-08-27 15:19 UTC

   We're see a partial degradation of data liveness, which may impact triggers and SLOs in a small percentage of users

2. resolved 2025-08-27 15:36 UTC

   We have determined the cause and implemented a fix. Customers should be seeing normal operations again.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2025-08-13 09:30 UTC

   For a period of approximately 1h30, from 9:34 UTC to 11:08 UTC, our ingestion of events in the EU region was delayed. All events were eventually processed, but for that duration, triggers, SLOs, and regular queries could have seen missing data for a brief period of time. As we...

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.eu1.honeycomb.io - EU1 Querying, ui.eu1.honeycomb.io - EU1 Trigger & SLO Alerting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2025-08-12 15:17 UTC

   Data ingestion is currently being delayed in the EU region. All traffic is being accepted, but it may take some time to show up in queries, and may have an impact on trigger-based alerting as well.

2. monitoring 2025-08-12 15:30 UTC

   We have load-balanced ingestion traffic to bring back performance under control, and are currently monitoring to make sure it stays that way.

3. resolved 2025-08-12 15:37 UTC

   Our load-balancing of ingestion traffic is doing its job and everything is back to normal.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Querying

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2025-07-21 18:01 UTC

   We are seeing a querying slowdown in the US. We have identified the cause and are working to mitigate

2. monitoring 2025-07-21 19:52 UTC

   A fix has been implemented and customers should see query speed returning to normal

3. resolved 2025-07-21 20:53 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.

Affected: ui.honeycomb.io - US1 Trigger & SLO Alerting

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2025-07-21 15:25 UTC

   An error in our production AWS configuration caused a brief interruption that affected Service Maps and caused an delay in delivering SLO notifications, and errors delivering Trigger notifications between 9:54AM UTC-5 and 10:05AM UTC-5. If any longer Triggers (1 day) would have...

2. resolved 2025-07-21 15:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Honeycomb's status page.