Braintrust

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-30 11:36 UTC

   We are investigating elevated errors affecting the Braintrust control plane APIs. Some users may experience failures when accessing projects, prompts, settings, or creating new resources.

2. investigating 2026-09-30 11:50 UTC

   We are investigating elevated error rates and increased latency on the US centrally-hosted data plane. Some requests, including function invocations, BTQL queries and loading logs or experiments in the UI, may fail or time out.

3. monitoring 2026-09-30 12:35 UTC

   We have applied a fix and are monitoring the results.

4. resolved 2026-09-30 12:54 UTC

   The issue has been resolved.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-09 21:30 UTC

   We're investigating slow or failed data loading affecting some customers on our US-hosted service. Affected pages may load slowly, hang, or show errors.

2. monitoring 2026-09-09 21:34 UTC

   We've applied a mitigation for the data-loading issue affecting our US-hosted service and are monitoring performance. We'll share another update once we've confirmed stability.

3. resolved 2026-09-09 22:10 UTC

   The issue causing slow or failed data loading in our US-hosted service has been resolved. We're sorry for the disruption.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-04 15:19 UTC

   We are investigating elevated errors affecting the Braintrust control plane APIs. Some users may experience failures when accessing projects, prompts, settings, or creating new resources.

2. monitoring 2026-09-04 15:34 UTC

   We have mitigated the issue causing elevated errors in the Braintrust control plane APIs, we are continuing to monitor closely.

3. resolved 2026-09-04 15:52 UTC

   The issue causing elevated errors in the Braintrust control plane APIs has been resolved.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-28 18:41 UTC

   The gateway encountered a load spike leading to 5xx errors.

2. resolved 2026-08-28 18:54 UTC

   The load spike has subsided and 5xxs are mitigated

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-28 15:47 UTC

   A fresh gateway deploy was attempted and led to 5xx errors, and then was immediately rolled back.

2. resolved 2026-08-28 15:55 UTC

   The deploy was rolled back and gateway 5xxs were mitigated.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-27 01:50 UTC

   A deployment caused the gateway to begin returning 5xxs for a portion of traffic.

2. resolved 2026-08-27 13:15 UTC

   The faulty deploy was rolled back and 5xxs were mitigated.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2026-08-19 18:01 UTC

   The upstream APT package mirror on AWS us-east-1 is degraded. **There is currently no downtime** and we are looking into alternative package mirrors in the meantime. Please note that for **both the Braintrust-hosted data plane and self-hosted data planes:** EC2 instance...

2. resolved 2026-08-19 22:35 UTC

   The upstream APT package mirror in AWS us-east-1 has recovered and we're no longer seeing impact when starting new brainstore instances.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-08-03 19:52 UTC

   We identified an issue causing 5XXs introduced by a recent Gateway deployment and are currently rolling it back.

2. monitoring 2026-08-03 20:06 UTC

   The issue has been resolved. We are monitoring the service to confirm normal operation.

3. resolved 2026-08-03 20:19 UTC

   The issue has been resolved and service has remained stable following the rollback. We will continue to monitor internally, but this incident is now considered resolved.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-07-16 08:32 UTC

   The hosted EU data plane is currently experiencing a very high rate of 504 errors.

2. resolved 2026-07-16 09:01 UTC

   We have reverted to an older configuration to avoid an AWS outage with Cloudfront.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-06-12 18:16 UTC

   Our US data plane is encountering issues.

2. resolved 2026-06-12 18:26 UTC

   A configuration change was identified as the root cause and reverted.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-06-04 22:51 UTC

   We're experiencing issues loading logs and some other pages in Braintrust. Our team is actively investigating and working to resolve.

2. resolved 2026-06-04 23:07 UTC

   We have pushed a fix and the loading issues have been remediated.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-15 20:00 UTC

   On May 15, some automations stopped running which impacted scorer executions.

2. resolved 2026-05-16 18:50 UTC

   We determined the issue was caused by a role used by the automation service losing required permissions following a configuration change. The issue was reported on May 15 at approximately 1:00 PM PT and resolved on May 16 at 11:50 AM PT. To help prevent similar incidents, we...

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-26 21:33 UTC

   Our upstream auth provider is having issues. Fresh logins to Braintrust will fail with an error.

2. resolved 2026-05-26 22:11 UTC

   Our auth provider has recovered.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-05 23:35 UTC

   We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.

2. identified 2026-05-06 00:08 UTC

   We have identified the cause and are working with an upstream provider on a fix.

3. monitoring 2026-05-06 00:10 UTC

   We are actively monitoring the fix.

4. resolved 2026-05-06 00:23 UTC

   We are no longer seeing issues.

5. resolved 2026-05-06 00:41 UTC

   The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-04-26 07:17 UTC

   Our upstream auth provider is having an outage. Users unable to login to Braintrust.

2. resolved 2026-04-26 07:23 UTC

   Provider has recovered. Error levels back to zero.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-20 03:33 UTC

   We're currently experiencing an issue with database connectivity. We're currently attempting to mitigate by connecting through an alternative method.

2. monitoring 2026-04-20 03:46 UTC

   We've updated our database connection method and are observing recovery. We're monitoring to look for remaining impact.

3. resolved 2026-04-20 03:49 UTC

   We're no longer seeing impact from the connectivity issues and we are still investigating the root cause.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Traffic surge or denial-of-service attack. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-04-13 21:38 UTC

   We experienced a brief period of degraded performance on the Braintrust hosted control plane due to elevated database load. The issue was caused by an internal traffic spike to a specific query pattern.

2. resolved 2026-04-13 21:51 UTC

   Performance quickly returned to normal when the problematic query was identified and remediated, and we are implementing safeguards to prevent similar future issues.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-03-31 17:52 UTC

   We're seeing a high rate of 5xx errors

2. resolved 2026-03-31 18:05 UTC

   Error rates have dropped and functionality is restored.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-03-20 23:07 UTC

   The control plane is experiencing very high load.

2. monitoring 2026-03-20 23:14 UTC

   Error rates have recovered by database usage is still high.

3. monitoring 2026-03-21 00:09 UTC

   Database size is being increased. This will cause a short outage.

4. monitoring 2026-03-21 00:15 UTC

   DB upgrade complete. CPU usage appears to be at normal levels. Monitoring for now.

5. resolved 2026-03-21 00:23 UTC

   Load issues have subsided

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-03-20 09:53 UTC

   Degraded performance of our authorization provider is causing issues with accessing the control plane.

2. monitoring 2026-03-20 10:21 UTC

   The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.

3. resolved 2026-03-20 10:32 UTC

   The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-03-10 19:03 UTC

   An upstream provider had a brief outage that caused high errors in our control plane

2. resolved 2026-03-10 19:11 UTC

   Provider recovered

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-03-10 16:18 UTC

   Degraded performance of our authorization provider is causing issues with accessing the control plane. The service provider is investigating the root issue.

2. monitoring 2026-03-10 16:35 UTC

   The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.

3. resolved 2026-03-10 16:38 UTC

   The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-03-03 15:50 UTC

   S3 automations have stopped running on the expected schedule they were set to.

2. resolved 2026-03-03 16:24 UTC

   Affected a small subset of users.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-02-19 16:34 UTC

   Degraded performance of our authorization vendor is causing issues with user authorization accessing the control plane. The vendor has identified the issue and is working on a fix.

2. investigating 2026-02-19 17:20 UTC

   We've observed improvement in authorization errors but our authorization provider is still reporting a degraded state. We're continuing to monitor.

3. monitoring 2026-02-19 17:53 UTC

   Impact for Braintrust users appears to have been resolved for the past 30 minutes. We'll continue to monitor until our authorization provider reports full resolution.

4. resolved 2026-02-19 18:35 UTC

   We have not observed any recurrence of issues and our authorization provider has reported full recovery and stable performance across their entire system.

Excerpts are shortened. Read the full incident on Braintrust's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-02-18 21:32 UTC

   At the current moment in time we have reports of hanging in the UI particularly in dataset or trace views. We are currently narrowing down the code change causing the issue. Please stand by we are aware and currently working to resolve.

2. resolved 2026-02-18 21:35 UTC

   The issue has been resolved and you should be seeing no slowness in the Braintrust Control Plane UI. Please let us know if you have any issues.

Excerpts are shortened. Read the full incident on Braintrust's status page.