turbopuffer
AI. Source: turbopuffer status page.
- Incidents, last 90 days
- 4
- Median to mitigation
- 22m
- Median to resolution
- 1h 2m
- p95 to resolution
- 7h 27m
- Naming an upstream provider
- 25%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A provider or third-party service this vendor depends on failed | 1 |
| Orchestration, control plane, service discovery or load balancing failed | 1 |
| The text does not say enough to tell | 1 |
| A zone, region or data center became unavailable | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
Elevated rate of 5xx in gcp-us-central1
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
Started during Google Cloud: Google Cloud Storage (GCS) customers experienced elevated latencies and error rates when performing upload or write operations..
-
investigating
Some namespaces are experiencing an elevated rate of 5xx errors. We have identified the issue and are working to restore full service.
-
identified
Some write operations are intermittently failing due to an ongoing incident with [Google Cloud Storage](https://status.cloud.google.com/incidents/mSFVt6PiP557bjRyYsut).
-
monitoring
Error rates have recovered as Google rolls out a fix to Google Cloud Storage. We will continue monitoring until the upstream incident is resolved.
-
resolved
The issue upstream in Google Cloud Storage is resolved and us-central1 operations have returned to normal.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Dashboard outage
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The turbopuffer dashboard is currently experiencing an outage. turbopuffer regions are fully available. Writes and queries are not affected.
-
resolved
From 7:21-7:41 UTC, the turbopuffer dashboard was unavailable due to an issue with our backing datastore. This issue has been resolved and dashboard access has been restored.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated rate of 5xx in gcp-us-central1
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
monitoring
Some namespaces are experiencing an elevated rate of 5xx errors. We have identified the issue and are working to restore full service.
-
resolved
The issue causing elevated 5xx errors for some namespaces has been resolved. We repaired the affected namespaces, and error rates have returned to normal. We will continue monitoring the service closely.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Outage in gcp-us-southeast1
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Full outage identified in gcp-southeast1
-
monitoring
We’ve identified the problem and rolled out a fix. We’re closely monitoring the situation.
-
resolved
Service has been fully restored.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Errors when accessing dashboard
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We're aware of issues affecting some customers who are logging into the dashboard. We're investigating and will provide an update shortly. Note that the database is not affected - this issue is limited to the dashboard.
-
identified
Dashboard logins (Google, SSO, and email) are intermittently failing due to an ongoing incident at [our authentication provider](https://status.workos.com/incidents/b52fkqjzcl20). We're monitoring and will update here.
-
monitoring
Logins are succeeding again as of 10:08 UTC. Our authentication provider is still working through their incident, so some logins may be slower than usual. We'll continue monitoring until it's fully resolved.
-
resolved
Our authentication provider has resolved the incident on their end, and dashboard logins have been fully operational since 10:08 UTC. Existing sessions and the database were unaffected throughout.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated 500 rate on delete_by_filter writes, filtered queries
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
monitoring
Investigating elevated 500 error rate on filtered queries, delete_by_filter operations.
-
resolved
Starting at ~17:55 UTC, we experienced an elevated 500 rate affecting delete_by_filter, and filtered queries as a result of a recent change deployed to filtering. This change was reverted, and errors fully subsided by ~18:30 UTC. This incident affected approximately 38.2% of...
Excerpts are shortened. Read the full incident on turbopuffer's status page.
partial outage in aws-us-west-2
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
Our team has identified an issue with a deploy in aws-us-west-2, causing queries and writes to 500 for affected namespaces. We are deploying a fix.
-
resolved
Our deploy has rolled out and affected namespaces have recovered.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Errors when accessing dashboard
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We've confirmed reports of dashboard errors and are actively investigating. Note that the database is not affected - this issue is limited to the dashboard.
-
identified
This issue is currently affecting the dashboard and API key management. **turbopuffer regions and namespaces are all operating normally**, and existing API keys continue to authenticate as expected. The root cause is a dependency service that we use for metadata. We'll share an...
-
monitoring
We're seeing recovery with traffic returning to normal. We're monitoring closely and will share another update shortly.
-
monitoring
Traffic has returned to normal. We're monitoring closely!
-
resolved
This incident is resolved. Dashboard access is working as usual. As noted earlier, this incident only affected the dashboard and had no impact on turbopuffer regions or namespaces!
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Errors when accessing dashboard
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We've confirmed reports of dashboard errors and our team is investigating.
-
resolved
We were impacted by [an issue with one of our providers](https://www.vercel-status.com/incidents/5r9bp5y8rql2), which we were able to resolve. We will continue to monitor for regressions.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated 408 and 429s
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The gcp-us-east4 cluster is experiencing elevated rates of 408 and 429s affecting both reads and writes affecting approximately 4% of traffic. Our team is investigating.
-
monitoring
Our team has taken remedial action and is observing recovery, we will continue to monitor.
-
resolved
We've observed a complete recovery of traffic.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Increased Request Latency and 429s in GCP us-east4
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
investigating
We are looking into an issue causing increased latency and 429s in us-east4.
-
monitoring
Latencies have returned to normal and 429s have fully recovered. The issue was related to transient failures in the underlying infrastructure. Our engineers have applied mitigations, and we are continuing to monitor.
-
resolved
The region is stable and has fully recovered. We will continue to monitor.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated 500s in region gcp-us-west1
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating an increase 500's in the region!
-
monitoring
A fix has been rolled out and 5xx errors in the region have reduced. We are continuing to monitor the situation!
-
resolved
The region has stabilized and returned to normal operation. We are continuing to monitor!
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Outage in aws-us-west-2
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating reports of an outage in aws-us-west-2.
-
monitoring
We've identified the problem and rolled out a fix. We're closely monitoring the situation.
-
resolved
Service has been fully restored. We'll be sharing a full RCA with affected customers as soon as possible.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated 500s in region gcp-us-east4 due to transient unavailability of turbopuffer query nodes
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Customers may notice a small number of their requests receiving a 500 response due to transient unavailability of turbopuffer query nodes. We have identified a probable cause and are moving to confirming and addressing it.
-
monitoring
We've identified the problem and taken action to stop customer impact.
-
resolved
Service has returned to normal. The cause has been identified and is being addressed.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
aws-us-west-2: Increased 429's on a subset of nodes
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We identified an issue causing high CPU utilization on a subset of query nodes in us-west-2. Some customers experienced 429 errors and increased query latency during this time. We have identified the root cause and are deploying a fix. We are actively monitoring the situation...
-
resolved
The issue has been resolved. We deployed a fix and confirmed that affected nodes have returned to normal operation.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Degraded performance in aws-us-west-2
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We're investigating reports of degraded performance in aws-us-west-2.
-
identified
We've identified a pathological performance issue when running on ARM hardware. We've shifted the region to run entirely on Intel hardware.
-
monitoring
The region has fully transitioned back to Intel hardware and API response latency has returned to normal. We're monitoring to ensure that latencies remain stable.
-
resolved
Latencies have remained stable.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
indexing backlog in aws-us-west-2
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
misconfiguration broke indexer auto-scaling, causing a backlog of jobs
-
resolved
incident was resolved
Excerpts are shortened. Read the full incident on turbopuffer's status page.
TLS certificate expiration in aws-us-east-1
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
We are investigating reports of connections to aws-us-east-1.turbopuffer.com failing TLS validation.
-
identified
We've identified that a stuck ACME challenge prevented cert-manager from automatically renewing the TLS certificate for aws-us-east-1.turbopuffer.com.
-
monitoring
The affected certificates were manually renewed. Connections to aws-us-east-1.turbopuffer.com should once again pass TLS validation.
-
resolved
Service has been fully restored.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Degraded performance in aws-us-west-2
Failure mode: Traffic surge or denial-of-service attack. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Some query nodes are currently experiencing excessive load, leading to 429s for certain namespaces. We have identified the problem and have a fix in preparation.
-
resolved
The problematic queries have been isolated and traffic has been restored to normal.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Degraded service (500 responses)
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
Some of our query nodes restarted repeatedly due to OOMs caused by a code path that was producing very large allocations. We have deployed a fix, and the service has recovered. We are monitoring.
-
resolved
Service is stable.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
degraded performance in aws-us-east-1
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
some namespaces in aws-us-east-1 may experience degraded performance and request timeouts. our team is working to rectify the issue.
-
monitoring
we've removed problematic query nodes from rotation and are monitoring for further issues.
-
resolved
we have not seen any re-occurences of the issue.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Elevated 500s in gcp-us-west1
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We're investigating an increase in 500 responses in the region.
-
resolved
We have taken steps to remediate the 500s, which have now subsided. We're investigating the root cause.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Partial dashboard outage
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
Some users are reporting issues accessing the turbopuffer dashboard due to a partial outage with our authentication provider: turbopuffer APIs are unaffected.
-
monitoring
Dashboard error rates have subsided. We continue to monitor the situation while we wait for confirmation from our authentication provider that the incident has resolved.
-
resolved
The upstream incident is resolved.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Degraded performance in gcp-us-east4
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
we're currently observing degraded performance in gcp-us-east4 leading to elevated rates of 408s and 429s. Our team is currently investigating.
-
monitoring
We have observed an increase in timeouts and a slow down in indexing starting at 9:38 UTC. The service has returned to normal at 10:00 UTC. We are monitoring and investigating the root cause.
-
resolved
Service is stable. We are continuing to investigate the root cause with high priority.
Excerpts are shortened. Read the full incident on turbopuffer's status page.
Outage in aws us-east-1
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
[AWS us-east-1 is experiencing a major outage and many services are affected. ](https://health.aws.amazon.com/health/status) turbopuffer clients in aws-us-east-1 will see elevated error rate and latencies during this outage. We're looking at mitigating the impact.
-
monitoring
We have deployed mitigations and are monitoring the situation.
-
monitoring
AWS is experiencing a major disruption in region us-east-1 impacting multiple services ( ). This caused turbopuffer to become unavailable in that region starting at 7:50 UTC. We've rolled out mitigations which resulted in a full recovery of turbopuffer at 8:20 UTC (despite the...
-
monitoring
The turbopuffer service remains healthy in the aws-us-east-1 region. We continue to closely monitor the situation while AWS continues to report service disruption in the region.
-
resolved
AWS has reported that the upstream incident is resolved.
Excerpts are shortened. Read the full incident on turbopuffer's status page.