Buildkite

1. investigating 2026-10-20 22:13 UTC

   The private keys for eight certificates issued by eMudhra, an external certificate authority, have been compromised. Software that still trusts these certificates may accept fraudulent HTTPS certificates. We’re updating the services that power Buildkite and preparing new CLI and...

2. monitoring 2026-10-20 23:25 UTC

   We're continuing to monitor the situation, and are making releases to many of our components that run in customer infrastructure. At time of writing, we've released versions that distrust the affected eMudhra certificates for: - Buildkite agent:...

3. monitoring 2026-10-21 05:07 UTC

   We’ve finished updating the services that power Buildkite so they no longer trust the affected eMudhra certificates. No action is needed from customers for Buildkite hosted services. For components that run in customer infrastructure, we still recommend upgrading to the releases...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-10-20 19:57 UTC

   We are investigating delays to build and job notifications for a subset of customers.

2. monitoring 2026-10-20 20:15 UTC

   We've identified the issue, and added some additional capacity. Notification latency has returned to normal levels. We are monitoring for stability.

3. resolved 2026-10-20 21:05 UTC

   Adding additional capacity has resolved the notifications latency.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

Started during GitHub: Degraded availability for GitHub Actions, GitHub Pages and Webhooks.

1. resolved 2026-10-19 15:45 UTC

   We received reports from a few customers that Hosted Agent jobs were delayed in starting. We investigated and found this related to a GitHub incident with webhooks that caused throttling in Hosted Agent job creation once webhooks were once again being ingested. Any affected...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

1. investigating 2026-10-06 18:19 UTC

   We've received reports of degraded checkout performance with GitHub repositories when using Hosted Agents and are investigating.

2. identified 2026-10-06 19:07 UTC

   We continue to observe degraded checkout performance. Our engineering teams are working on establishing an alternative network transit path to alleviate networking performance issues.

3. resolved 2026-10-06 23:59 UTC

   The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Webhooks

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

1. identified 2026-10-01 06:10 UTC

   Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related builds are not created. For people who use GitHub.com may affect ability to receive status checks. We have identified the...

2. monitoring 2026-10-01 06:33 UTC

   The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update.

3. resolved 2026-10-01 06:57 UTC

   The error rate has returned to normal levels.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-30 01:03 UTC

   We are investigating delays to build and job notifications for a subset of customers.

2. identified 2026-09-30 01:19 UTC

   We've resolved the underlying error, and outbound notifications queues are now in the process of catching up for impacted customers.

3. monitoring 2026-09-30 01:43 UTC

   Outbound notification queues have now fully caught up for all impacted customers.

4. resolved 2026-09-30 02:32 UTC

   Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-09-28 01:30 UTC

   Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own....

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-09-22 20:07 UTC

   We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.

2. identified 2026-09-22 20:26 UTC

   Our engineers have addressed the issue and we're monitoring recovery.

3. resolved 2026-09-22 20:50 UTC

   Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC.

4. postmortem 2026-09-28 04:12 UTC

   ## Service Impact On September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out. During this period, 2.0% of requests to our webhook ingestion endpoints...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2026-09-21 23:47 UTC

   We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.

2. resolved 2026-09-22 00:32 UTC

   We did not experience any customer impact during this AWS us-east-1 outage.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-21 22:35 UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring 2026-09-21 23:12 UTC

   We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement.

3. resolved 2026-09-21 23:46 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: REST API, Web, Agent API, Job Queue

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-09-09 19:07 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. investigating 2026-09-09 19:41 UTC

   We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.

3. investigating 2026-09-09 20:28 UTC

   We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.

4. monitoring 2026-09-09 20:53 UTC

   We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.

5. resolved 2026-09-10 01:54 UTC

   We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API...

6. postmortem 2026-09-11 06:03 UTC

   ## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents, MacOS, Linux (ARM64), Linux (AMD64)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-04 01:42 UTC

   We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.

2. monitoring 2026-09-04 02:23 UTC

   We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

3. resolved 2026-09-04 02:39 UTC

   Jobs running on Hosted Agents are now being dispatched promptly.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-03 06:00 UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. investigating 2026-09-03 06:09 UTC

   Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating and will update again soon.

3. monitoring 2026-09-03 06:44 UTC

   We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs.

4. resolved 2026-09-03 07:06 UTC

   Jobs running on Hosted Agents are now being dispatched promptly.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents, MacOS, Linux (ARM64), Linux (AMD64)

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-02 14:45 UTC

   We're investigating an issue with job dispatch to our Hosted Agents.

2. investigating 2026-09-02 15:16 UTC

   Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

3. identified 2026-09-02 15:29 UTC

   We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

4. monitoring 2026-09-02 15:43 UTC

   We are continuing to process through the backlog of delayed jobs and monitoring.

5. resolved 2026-09-02 15:51 UTC

   Job dispatch for Hosted Agents is no longer experiencing delays.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Email Notifications

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-09-01 15:32 UTC

   We are experiencing delivery issues with email notifications due to an issue with an upstream provider.

2. identified 2026-09-01 16:19 UTC

   Our on-call engineers are working on failing over to an alternative upstream email provider to restore outbound email notifications. No other Buildkite services have been impacted by this disruption.

3. monitoring 2026-09-01 16:38 UTC

   We are seeing successful email deliveries and are monitoring processing of our outbound email queue.

4. resolved 2026-09-01 16:52 UTC

   Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-08-27 10:56 UTC

   Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all data should be processed over the next hour.

2. monitoring 2026-08-27 11:23 UTC

   Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing.

3. resolved 2026-08-27 11:47 UTC

   Processing of uploaded test execution data has caught up and is back to normal.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-08-26 01:42 UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. investigating 2026-08-26 02:02 UTC

   We identified issues affecting certain customers when the buildkite-agent attempts to register with Buildkite. When agents are not able to register, builds may start to back up. We've identified a potential root case and are attempting changes to fix it.

3. investigating 2026-08-26 02:36 UTC

   We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery. We'll keep monitoring until all systems are fully recovered and communicate once we are confident everything is stable again.

4. monitoring 2026-08-26 02:43 UTC

   Recovery is underway; we'll keep monitoring until recovery is complete.

5. resolved 2026-08-26 03:13 UTC

   Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: REST API, Web, GitHub Commit Status Notifications, Email Notifications, Agent API, Remote MCP Server, Slack Notifications, Job Queue, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-08-25 22:55 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. investigating 2026-08-25 23:05 UTC

   We are currently investigating a sitewide outage.

3. investigating 2026-08-25 23:23 UTC

   We have identified a fix and applied mitigations, and we are seeing signs of recovery.

4. monitoring 2026-08-25 23:38 UTC

   Systems have recovered and we are seeing new builds created. For a period of about 20min new builds could not be created due to a failure of our internal DNS services. No build requests were lost during this time and we are continuing to process the backlog

5. resolved 2026-08-26 00:18 UTC

   We're considering this incident resolved.

6. postmortem 2026-08-27 23:53 UTC

   ## Service Impact Customers experienced a site-wide disruption affecting the Buildkite web interface, REST API, Agent API, job queue, and notifications. The disruption began at 22:47 UTC and continued until 23:16 UTC. During this period, all customers were unable to create new...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-08-25 06:41 UTC

   At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are...

2. monitoring 2026-08-25 07:42 UTC

   Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.

3. resolved 2026-08-25 08:16 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-17 00:46 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. investigating 2026-08-17 01:21 UTC

   We’re investigating elevated latency in background worker processing between approximately 12:39am and 1:02am UTC. Processing has since caught up and was near real time as of 1:13am UTC. We’re continuing to monitor the service and are checking remaining retries. Some builds or...

3. monitoring 2026-08-17 02:00 UTC

   We have confirmed that all background workers have caught up as of 1:13am UTC. Hosted Agents dispatch has returned to normal. We'll continue to to monitor the service and are checking remaining retries.

4. resolved 2026-08-17 02:43 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Web, Web

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-08-13 15:41 UTC

   We've spotted that something has gone wrong and there's elevated latency. We're currently investigating the issue, and will provide an update soon.

2. identified 2026-08-13 16:14 UTC

   We've identified the cause of the issue and are actively mitigating.

3. monitoring 2026-08-13 16:29 UTC

   We have identified and fixed the issue. We are monitoring and seeing signs of improvement.

4. resolved 2026-08-13 16:40 UTC

   We think the impact from the issue is over.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-08-12 02:44 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. identified 2026-08-12 02:59 UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently addressing the cause and will provide status updates as they become available.

3. identified 2026-08-12 03:22 UTC

   For a subset of customers, <1% of their Agent API traffic is experiencing issues. We are rolling back the change that caused this issue while we investigate it further.

4. monitoring 2026-08-12 03:27 UTC

   We have reverted the change which introduced the elevated error rate. We believe the issue to be resolved but will continue to monitor.

5. resolved 2026-08-12 04:08 UTC

   For the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully. At 23:01 11 August 2026 UTC we started routing <1% of agent traffic for a small subset of our customer to this EKS infrastructure. Within that subset of...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Web, Web

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-08-10 23:00 UTC

   We've spotted that something has gone wrong and there's elevated latency. We're currently investigating the issue, and will provide an update soon.

2. investigating 2026-08-10 23:14 UTC

   Our Web UI is slow at the moment, we are investigating the issue.

3. identified 2026-08-10 23:25 UTC

   We've identified the cause of the issue and are actively mitigating.

4. monitoring 2026-08-10 23:55 UTC

   We have identified and fixed the issue. We are monitoring and seeing signs of improvement. We've temporarily disabled the integration between Pipelines and Test Engine. This affects the Tests tab on build pages. We’ll provide another update as recovery progresses. Test results...

5. monitoring 2026-08-11 00:50 UTC

   The integration between Pipelines and Test Engine remains temporarily disabled, and we're actively working to bring this back online. This affects the Tests tab on build pages. We’ll provide another update as recovery progresses. Test results are available directly through Test...

6. monitoring 2026-08-11 01:04 UTC

   The integration between Pipelines and Test Engine has been re-enabled, restoring the Tests tab on build pages. We’re monitoring the recovery to confirm the integration remains stable. Test results continue to be available directly through Test Engine.

7. resolved 2026-08-11 02:04 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-08-05 22:00 UTC

   Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Job Queue

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-08-03 05:11 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. identified 2026-08-03 05:17 UTC

   We're experiencing latency with processing job dispatch, pipeline uploads and incoming webhooks. We have identified the cause and are rolling forward with a fix now.

3. identified 2026-08-03 05:22 UTC

   We've identified that the impact is more widespread that we initially understood - scheduled builds as well as job dispatch. The fix that is rolling out now is expected to address all impact.

4. monitoring 2026-08-03 05:29 UTC

   The fix has been fully deployed and we are seeing signs of recovery. All events impacted by this issue will be automatically retried. We are continuing to monitor the fix for stability.

5. resolved 2026-08-03 05:36 UTC

   We've confirmed all impact from this incident has been resolved.

6. postmortem 2026-08-10 00:03 UTC

   ## Service Impact Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was concentrated in Pipelines. During this window, scheduled builds did...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-07-28 20:38 UTC

   We are investigating delays to build and job notifications for a subset of customers.

2. resolved 2026-07-28 20:57 UTC

   Notification latency for the affected shard is now under 5s.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: REST API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-06-18 08:05 UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. identified 2026-06-18 08:59 UTC

   We've identified the issue, we're continuing working on rolling out a fix. Impact is restricted to the builds create API for a subset of tenants.

3. monitoring 2026-06-18 09:17 UTC

   A fix has been deployed, services are recovering.

4. resolved 2026-06-18 09:43 UTC

   We have seen a full recovery of services.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: REST API, Remote MCP Server

Failure mode: Traffic surge or denial-of-service attack. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-11 23:18 UTC

   We're observing increased latency for all our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring 2026-06-11 23:49 UTC

   We've isolated the issue to elevated load on our REST API service and are working to mitigate. The agent and stacks API isn’t affected.

3. monitoring 2026-06-11 23:51 UTC

   We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected.

4. resolved 2026-06-12 00:16 UTC

   The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-06-11 00:32 UTC

   We're observing increased latency and error rates for a subset of our customers on the Agent API. We're currently investigating and will provide status updates as they become available.

2. resolved 2026-06-11 00:51 UTC

   Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-05-30 00:30 UTC

   We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

1. investigating 2026-05-28 20:20 UTC

   We are investigating delays to build and job notifications for a subset of customers.

2. identified 2026-05-28 20:47 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

3. resolved 2026-05-28 21:18 UTC

   This incident has been resolved.

4. postmortem 2026-06-09 22:20 UTC

   ## Service Impact Customers experienced delayed Buildkite notification delivery. The customer impact varied depending on how those notifications are used. For some customers, delayed notifications also delayed downstream CI, merge, or deployment workflows. ## Incident Summary On...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. identified 2026-05-26 09:56 UTC

   We're observing increased latency and error rates for a subset of our customers. We're currently remediating and will provide status updates as they become available.

2. monitoring 2026-05-26 10:08 UTC

   We've identified the problem and have completed the remediation steps, we are now monitoring as service resumes.

3. monitoring 2026-05-26 10:37 UTC

   We see processing time for all affected services has returned to normal as of 20 minutes ago.

4. resolved 2026-05-26 10:38 UTC

   We think the impact from the issue is over.

5. postmortem 2026-05-28 04:20 UTC

   ## Service Impact A subset of customers experienced elevated latency in notification delivery. ## Incident Summary While migrating a subset of our background processing services to Amazon EKS, we encountered an issue with delivery of internal metrics. The discovered issue did...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-05-20 16:40 UTC

   We are investigating delays to notifications across all customers

2. identified 2026-05-20 17:06 UTC

   We have identified the issue and applied mitigations and are monitoring recovery We have determined that only a subset of customers are affected by the notification latency.

3. monitoring 2026-05-20 17:26 UTC

   We are seeing recovery across affected customers and continue to monitor

4. resolved 2026-05-20 17:39 UTC

   The incident is resolved

5. postmortem 2026-05-25 07:02 UTC

   ## Service Impact A subset of our customers experienced elevated latency in our notification delivery, build dispatch and metrics services. ## Incident Summary We are in the process of migrating our underlying compute platform from AWS Fargate to AWS EKS for our production...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-05-15 06:51 UTC

   Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring...

2. resolved 2026-05-15 07:35 UTC

   Processing of the backlog is complete.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: REST API, Web, Remote MCP Server

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-13 15:14 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. resolved 2026-05-13 15:34 UTC

   Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-05-12 12:59 UTC

   We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog

2. resolved 2026-05-12 16:09 UTC

   The fix was successful and the backlog has now been cleared.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-08 21:10 UTC

   We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog. At the current processing rate we expect the backlog to be cleared by...

2. resolved 2026-05-08 23:42 UTC

   The delayed backlog is now cleared and Test Engine ingestion is operating normally.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Ingestion, Slack Notifications, Webhook Notifications

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-05-08 12:20 UTC

   We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience continued processing delays while we clear the ingestion backlog. At the current processing rate we expect the...

2. resolved 2026-05-08 13:06 UTC

   The Test Engine backlog is now cleared and operating normally.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-05-08 01:12 UTC

   We're aware that AWS is reporting availability zone failures in us-east-1. We are monitoring the situation but so far there is no customer impact.

2. investigating 2026-05-08 01:50 UTC

   A small subset of our customers are experiencing delayed notifications. We are actively provisioning additional capacity for these customers. Availability zone automatic failovers are occurring in response to the outage, and this is causing some brief error blips for some...

3. investigating 2026-05-08 02:35 UTC

   We are continuing to actively monitor the impacts of this availability zone outage for Buildkite customers. Some transient errors are visible due to availability zone failover events.

4. investigating 2026-05-08 04:08 UTC

   We have provisioned additional capacity in unaffected availability zones so that they are able to support the additional load. Automatic failovers continue to occur where necessary. Some latency and transient errors will be visible to customers.

5. investigating 2026-05-08 05:10 UTC

   We are actively moving resources out of us-east-1c. Similar brief latency and error blips will be visible to customers while these manual failovers occur.

6. investigating 2026-05-08 05:45 UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips will unfortunately continue while these manual failovers occur.

7. investigating 2026-05-08 07:04 UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

8. investigating 2026-05-08 07:22 UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur. (Apologies if you receive duplicated notifications for this update.)

9. monitoring 2026-05-08 08:07 UTC

   Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for further action should the need arise. We are also watching AWS services closely as they recover.

10. resolved 2026-05-09 04:34 UTC

    The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident.

11. postmortem 2026-05-13 05:07 UTC

    ## Service impact On 8th May 2026 UTC, between 00:00 and 07:30 UTC, some customers would have seen intermittent errors and latency spikes across many areas of the platform. ## Incident Summary The AWS availability zone incident in `use1-az4` triggered our automatic availability...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Job Queue, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-07 22:45 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. identified 2026-05-07 23:13 UTC

   We've identified the issue and are working on applying mitigations. At this time we can confirm inbound and outbound webhooks, and notifications are delayed.

3. monitoring 2026-05-07 23:50 UTC

   We have now monitoring the incident. We are seeing most customers have recovered, and some showing signs of recovery.

4. resolved 2026-05-08 00:13 UTC

   All customer workloads have now recovered.

5. postmortem 2026-05-21 05:53 UTC

   ## Service Impact On 2026-05-07 between 22:22 UTC and 00:14 UTC the next day \(~1 hour 52 minutes\), Buildkite customers experienced delays across several Pipelines features: * Inbound webhook processing was delayed. * Outbound webhook delivery and build notifications were...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-07 08:48 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue with new builds not starting.

2. investigating 2026-05-07 09:09 UTC

   We've identified issue with job acquiring endpoint. We're rolling back now. We'll provide next update in ~20 minutes.

3. identified 2026-05-07 09:24 UTC

   We're currently seeing recovery at 50% rate. We'll provide next update soon.

4. resolved 2026-05-07 09:37 UTC

   We've reverted a change that caused stale environment variables provided to acquire job used in Hosted Agents, agent-stack-k8s and other agent implementations using acquire job.

5. postmortem 2026-05-15 01:15 UTC

   ## Service Impact On May 7th between 08:14 and 09:25 UTC customers using [hosted agents](https://buildkite.com/docs/agent/buildkite-hosted), the [k8s stack](https://buildkite.com/docs/agent/self-hosted/agent-stack-k8s) and the Buildkite agent [acquire job...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-06 03:57 UTC

   A process writing test results to our Test Engine data store stalled, we've restarted the process and are seeing it catching up. We expect to be fully caught up on the backlog within the next couple of hours.

2. monitoring 2026-05-06 04:21 UTC

   We've identified the issue and the system is currently processing the backlog of test executions

3. resolved 2026-05-06 05:26 UTC

   Processing of test execution ingestion data has successfully caught up.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-04 17:02 UTC

   We are investigating delays with build and job notifications for a subset of customers.

2. identified 2026-05-04 18:06 UTC

   We've identified the source of the notification delays affecting a subset of our customers. Our engineers are applying remediations to reduce these delays.

3. monitoring 2026-05-04 18:54 UTC

   Applied remediations have resolved the previous notification delays affecting a subset of our customers. We're continuing to monitor the affected services for stability.

4. resolved 2026-05-04 19:36 UTC

   Notifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-04 06:02 UTC

   We're observing increased latency and error rates in the Agent API for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. resolved 2026-05-04 06:30 UTC

   An increase in requests has lead to the API service being temporarily saturated. We have updated rate limits to ensure this doesn't re-occur and will add further resources if necessary

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-05-01 02:41 UTC

   We've identified a service change that is causing a service disruption. We are reverting this change.

2. monitoring 2026-05-01 02:52 UTC

   We've corrected the issue that caused this disruption and normal service has been restored. We are monitoring the situation now.

3. resolved 2026-05-01 03:30 UTC

   This incident has been resolved.

4. postmortem 2026-05-08 06:08 UTC

   ### Service Impact Between 02:29 and 02:48 UTC on May 1st 2026, the Buildkite dashboard, Agent API, REST API and GraphQL were severely degraded for the majority of customers. During this period, customers were unable to reliably load pages in the dashboard, and agents...

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Agent API

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-04-29 17:43 UTC

   We're observing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring 2026-04-29 18:18 UTC

   We have identified and fixed the issue with the underlying database for a subset of customers. We are now monitoring the issue.

3. resolved 2026-04-29 18:48 UTC

   We have confirmed that latency and error rates have returned to normal for impacted customers.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Hosted Agents

Failure mode: Traffic surge or denial-of-service attack. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-04-28 18:00 UTC

   We're observing increased error rates and dispatch latency for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. identified 2026-04-28 18:26 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2026-04-28 18:45 UTC

   We have mitigated the issue causing increased Hosted Agents dispatch latency and intermittent timeout errors for a subset of customers. We identified abnormal workload activity that was placing elevated load on a supporting service, and have now blocked that activity and applied...

4. resolved 2026-04-28 19:16 UTC

   Previously elevated loads with Hosted Agents dispatch have fully recovered.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-04-22 21:19 UTC

   We are currently investigating reports of authentication failures with the remote MCP server.

2. investigating 2026-04-22 22:07 UTC

   We are continuing to investigate errors when authenticating to the remote MCP server.

3. monitoring 2026-04-22 22:44 UTC

   We have rolled back a change on the remote MCP server that was contributing to authentication failures.

4. resolved 2026-04-22 22:59 UTC

   The issue is resolved.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-04-22 02:32 UTC

   We noticed a lag in data processing, but our systems are operational and currently working through the backlog. We expect to be fully caught up within the next couple of hours.

2. resolved 2026-04-22 05:07 UTC

   The backlog has been cleared and all systems are fully operational. Thank you for your patience.

Excerpts are shortened. Read the full incident on Buildkite's status page.

Affected: GitHub Commit Status Notifications, Email Notifications, Agent API, Slack Notifications, Webhook Notifications

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-04-08 22:26 UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. monitoring 2026-04-08 22:40 UTC

   We have identified and fixed the issue. We are monitoring and seeing signs of improvement.

3. resolved 2026-04-08 23:12 UTC

   We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.

4. postmortem 2026-04-21 06:04 UTC

   ## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and...

Excerpts are shortened. Read the full incident on Buildkite's status page.