CockroachDB Cloud
Infrastructure. Source: CockroachDB Cloud status page.
- Incidents, last 90 days
- 5
- Median duration
- 7h
- p95 duration
- 22h 12m
- Naming an upstream provider
- 80%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A zone, region or data center became unavailable | 2 |
| Network, DNS, routing or connectivity failure | 1 |
| A provider or third-party service this vendor depends on failed | 1 |
| The text does not say enough to tell | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
gcp-eu-west2-b-availability-zone-issues
Affected: CockroachDB Basic/Standard, CockroachDB Advanced
Failure mode: A zone, region or data center became unavailable. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
Started during Google Cloud: Multiple services are experiencing issues in the europe-west2 region..
-
investigating
GCP europe-west2-b and availability zone is currently experiencing issues. We will continue to monitor this situation and update status once we have more information.
-
identified
GCP has mitigated the power event that caused this disruption in the europe-west2 region and is working on mitigating impacted services. We will update once we have confirmation of mitigation.
-
monitoring
GCP has mitigated the power event that caused this disruption in the europe-west2 region and most GCP services have recovered. We will continue to track and ensure impacted CRL services return to normal operation before resolving.
-
resolved
GCP has mitigated the power event that caused this disruption in the europe-west2 region. All GCP and CRL services has returned to normal operation. No further action is required.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Stock out issues in GCP us-central1
Affected: CockroachDB Advanced, Continuum Mission Critical
Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
investigating
We are currently monitoring stock out (lack of capacity) issues for instances in the GCP us-central1 region. If possible, customers are advised to use a different region when creating new clusters. It is expected that cluster creations and edits will eventually succeed as...
-
monitoring
We are continuing to monitor these stockouts and will post updates as new information comes in.
-
resolved
We are continuing to monitor this issue; however Google can not provide an ETA as to when it will be resolved. We will continue to work with them to obtain the needed capacity. Customers are still advised to choose another region when spinning up new clusters to circumvent the...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Connection attempts failing for multiple Basic tenants in GCP South America (southamerica-east1).
Affected: CockroachDB Basic/Standard
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
investigating
CockroachDB Cloud (Basic) clusters hosted in GCP South America (São Paulo / southamerica-east1) are currently experiencing network connectivity issues. We are investigating and will provide an update shortly
-
resolved
This incident has been resolved. CockroachDB Cloud (Basic) clusters hosted in GCP South America (São Paulo / southamerica-east1) experienced network connectivity issues that have now been fully restored. Connectivity has returned to normal and all affected clusters are operating...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cloud Latency, Delays, Connection Issues
Affected: CockroachDB Basic/Standard
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
Started during Google Cloud: Multiple products in us-central1-b are experiencing network service degradation..
-
investigating
Google Cloud is experiencing networking outages which is affecting Cockroach Cloud customers. Cockroach Cloud customers may experience heightened latency, delays, or connection issues. Our teams are aware and fully engaged on this incident. Known current Impacts: us-central-1,...
-
investigating
Google Cloud is experiencing networking outages which is affecting Cockroach Cloud customers. Cockroach Cloud customers may experience heightened latency, delays, or connection issues. Our teams are aware and fully engaged on this incident. Known current Impacts: us-central-1,...
-
investigating
We are still monitoring the situation, seeing some improvement over the last 15 minutes, but not fully resolved. Known current Impacts: us-central1, asia-east1, europe-west4-b
-
monitoring
Google is now reporting that us-central1 and asia-east1 are both resolved, however europe-west4-b is still actively impacted. Our internal teams are confirming no further errors in the last 15 minutes for us-central1 and asia-east1 regions. Known current Impacts: europe-west4-b
-
monitoring
We are learning that Google is reporting us-central1-b is also impacted and experiencing degradation. Following network-level mitigation efforts, the underlying network infrastructure has fully recovered. Google Product engineering teams across Cloud are currently validating the...
-
resolved
Google impacts are now reported as fully resolved. Internal teams have confirmed this report. All services are restored. Should you need further help, please reach out and file a ticket with Support.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster connectivity and execution issues
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating an issue affecting a subset of Free Tier clusters in eu-west-1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
-
investigating
We are investigating an issue affecting a subset of Basic clusters in eu-west1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
-
investigating
We are investigating an issue affecting a subset of Basic and Standard clusters in eu-west1, where some clusters may be unable to connect or execute queries. Teams have been notified and are actively working on mitigation.
-
monitoring
Teams have mitigated the immediate active impact, but work continues to ensure the problem does not recur and to fully understand the root cause. For now we will continue to monitor the situation. If you continue to experience impacts, please reach out to Support.
-
resolved
Following an extended period of monitoring, the team has confirmed that the cluster remains healthy and stable.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster Creation Failure
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently aware of and investigating a cluster creation failure. Teams are engaged and actively working to mitigate as quickly as possible.
-
resolved
The cluster creation issue has been restored. Issue is resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Unable to accept invites
Affected: Cloud Console
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We have identified a minor issue affecting Cloud customers who are unable to accept invites, preventing them from inviting new users to their organizations. Internal teams are aware and a fix is in flight. ETA: 30min - 1hour. We will provide the next update at the top of the...
-
resolved
Fix deployment is complete, tested and verified.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Degraded performance in one AWS us-east-1 availability zone
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
monitoring
AWS is experiencing elevated error rates and degraded performance in a single availability zone in the us-east-1 region. CockroachDB clusters in us-east-1 remain available, and we are monitoring the situation and taking corrective action as appropriate to ensure SQL availability...
-
monitoring
AWS's latest status update reports that they are continuing to work on the situation, and progress is slow. AWS has advised a next update time at 05:00 UTC. We continue to monitor the situation and take appropriate action to ensure uptime and availability for Cockroach DB...
-
monitoring
AWS reports continued progress towards resolving the issue in the US-EAST-1 Region and is working towards full recovery. Affected services may continue to experience elevated error rates and latencies until recovery is complete. We continue to monitor the situation and take...
-
resolved
All CockroachDB clusters in the affected AWS availability zone have been restored to a fully operational state. We will continue to monitor the upstream incident from AWS, but do not expect any impact to CockroachDB clusters.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cloud Console Social Login Auth Unavailable
Affected: Cloud Console
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
Social logins for customers using personal/non-SSO Microsoft accounts is currently unavailable. Our teams are aware and currently investigating.
-
resolved
Social login for Microsoft accounts is now available. Please reach out to support if you continue to experience issues.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster unavailability and degraded cluster operations in AWS me-central...
Affected: CockroachDB Advanced
Failure mode: A zone, region or data center became unavailable. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS, Azure, Google Cloud.
Started during AWS: Multiple services (UAE): Region Availability.
-
monitoring
A power outage at an AWS facility in the me-central-1 (Mid-East) region may cause failures or slower than usual progress on cluster operations. AWS is aware of the issue and is working to restore power. We are monitoring the situation, and will provide updates as we receive them...
-
monitoring
Current conflict in the region has led to objects impacting the AWS data center creating a fire earlier today. Power was shut down to prevent further damage. AWS is working on multiple mitigation paths to restore services as quickly as possible. We will provide the next update...
-
monitoring
AWS is reporting positive signs of recovery at this time, but are still working on multiple mitigation paths to fully restore services. We continue to keep a close eye on the situation and will provide another update by 6PM PST, or sooner if we have additional information to...
-
monitoring
AWS is reporting significant signs of recovery for specific APIs and requests, but several still remain to be mitigated and will likely take several hours more to complete, if not into tomorrow. Our teams have begun testing with this update but have reported we are still...
-
monitoring
AWS continues to work on mitigation efforts, including stabilizing impacted APIs and implementing interim workarounds. Our teams are actively reviewing and validating these mitigations; however, this effort will continue into tomorrow. We will provide our next update at 7:00 AM...
-
monitoring
me-central-1 region is impacted more with two AZs out and the clusters in that region are unavailable. In addition to this, me-south-1 region is also experiencing an AZ outage, which causes degraded performance of clusters in me-south-1. We are closely tracking AWS's status....
-
monitoring
AWS me-central-1 region is impacted with two AZs out. Clusters in that region are currently unavailable. AWS me-south-1 region is also experiencing a single AZ outage, which causes degraded performance of clusters in that region. While no impact has been observed on GCP or Azure...
-
resolved
Beginning March 1, 2026, a power outage caused by physical damage to AWS infrastructure affected multiple Availability Zones in the AWS me-central-1 (UAE) region, resulting in the unavailability of CockroachDB Cloud clusters in that region. A single Availability Zone in...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Connectivity issues to Standard and Basic clusters on AWS
Affected: CockroachDB Basic/Standard
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
We are investigating connectivity issues to Standard and Basic clusters on AWS.
-
monitoring
We have resolved an internal misconfiguration, and see that Standard and Basic clusters are functioning normally again. We will continue to monitor to ensure that the error does not recur.
-
resolved
Standard and Basic clusters continue to function normally.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Advanced cluster creation on AWS taking longer than expected
Affected: CockroachDB Advanced
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We have identified an issue that is causing Advanced cluster creation on AWS to take longer than usual. We are preparing a fix, and will update this status page with more information when we have an ETA for resolution.
-
identified
We are preparing a fix for deployment, which should restore AWS cluster creation times to normal levels. We will update the status page when the fix is deployed, or by 14:30 US/Eastern with more information.
-
identified
We are continuing to work on the fix for this issue. Next update will be at 15:00 US/Eastern.
-
resolved
We have deployed and verified a fix. Advanced clusters should now create within normal time frames.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Standard and Basic cluster creation on AWS delayed
Affected: CockroachDB Basic/Standard
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: AWS, Google Cloud.
-
investigating
We are investigating errors that are causing delays in creation of Basic and Standard clusters on AWS. Basic and Standard cluster creation on GCP is operating normally at this time.
-
investigating
We are investigating errors with Basic and Standard clusters on AWS. Cluster creation, certain cluster edit operations (such as adding a region), and certain CockroachDB Cloud Console functionality (such as listing databases or users) are also impacted. Basic and Standard...
-
identified
We have identified the cause of cluster creation failures impacting Basic and Standard AWS clusters. As a precautionary measure, while we apply a fix, we have disabled all cluster creation in Cockroach Cloud. We will restore cluster creation when it is safe to do so. There is no...
-
identified
We are continuing to work on a fix for this issue. Next update will be by 4:00 PM US/Eastern time.
-
identified
We are continuing to work on a fix for this issue. Next update will be by 4:30 PM US/Eastern time.
-
monitoring
We have resolved the underlying issue that was preventing cluster creation and other cluster operations. We have re-enabled cluster creation on all clouds. We will monitor the situation to ensure that everything continues to operate normally.
-
resolved
All cluster operations are working normally.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Advanced Cluster Creation Delayed
Affected: CockroachDB Advanced
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating errors that cause creation for Advanced clusters to take longer than usual. Standard and Basic cluster creation is operating normally.
-
monitoring
We have applied a mitigation and observe that cluster creations are proceeding again. We will monitor all impacted cluster creations to ensure they succeed.
-
resolved
Cluster creation is operating normally, and all delayed clusters have finished creation.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
AWS us-east-1 outage impacting cluster operations and backups
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We’re currently impacted by an ongoing AWS outage in the us-east-1 region. Customers may experience failures or delays when creating new clusters, adding nodes, or running backups in this region. Existing clusters remain operational, but backup operations may be delayed or fail...
-
monitoring
AWS has resolved the underlying outage in the us-east-1 region, and we’re seeing recovery across CockroachDB Cloud operations. Cluster creation, scaling, and backups are now succeeding again. We’re continuing to monitor for full stability before marking this incident as resolved.
-
resolved
The earlier AWS outage in the us-east-1 region has been fully resolved. All CockroachDB Cloud operations, including cluster creation, scaling, and backups, are functioning normally again. We’ve verified recovery across affected systems and customer clusters, and normal...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Google Kubernetes incident impacting cluster operations in GCP.
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
identified
We are currently experiencing an incident impacting certain cluster operations on Google Cloud Platform due to an known ongoing Google Kubernetes-related issue. Customers may encounter errors when creating or editing clusters. Our engineering teams are actively monitoring the...
-
resolved
Google has reported the issue has been resolved and we've verified cluster operations are functional.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
SQL availability impacted across multiple Serverless clusters.
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are observing an incident impacting SQL availability across multiple Serverless clusters. We are currently investigating the issue.
-
monitoring
Correction from the original status update. This issue appears to have been isolated to cluster crl-prod-6rw. We have identified the cause, mitigated and confirmed that cluster availability has returned. We are currently completing additional validations.
-
resolved
We have confirmed resolution of the incident impacting SQL availability for the affected cluster. SQL availability is restored and the associated cluster is now healthy.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Dependent Service incident intermittently impacting CockroachCloud Operations
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
We are aware of an ongoing Incident with dependent service "Let's Encrypt": https://letsencrypt.status.io/, which may impact cluster creation and add region operations for CockroachDB Cloud Advanced clusters. We will update this status page with more information as it becomes...
-
resolved
LetsEncrypt has indicated the issue has been mitigated. We have validated cluster creation behaviour is now returning to normal.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cloud Console and API Planned Maintenance 16:00 - 17:00 UTC
Affected: Cloud Console
Failure mode: A code release introduced the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We will be performing a planned maintenance operation on Cockroach DB Cloud Console and the Cockroach DB Cloud API today from 16:00 - 17:00 UTC. During this time, you may experience errors attempting cluster create or edit operations. Cockroach DB Cloud Advanced, Standard, and...
-
identified
Maintenance is beginning now. The next update will be before 17:00 UTC.
-
resolved
The maintenance has been completed, and the cloud console and API are working normally.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster operations impaired
Affected: CockroachDB Basic/Standard, CockroachDB Advanced
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are investigating an issue that may prevent or delay cluster operations from succeeding.
-
resolved
We have corrected a misconfiguration which was preventing cluster operations from succeeding. We are monitoring the success of cluster operations already in progress and new operations.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster operations taking longer than expected
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are investigating cluster operations, including creating or editing clusters, that are taking longer than usual.
-
identified
We have identified the issue preventing cluster operations from proceeding, and we see that operations are moving forward. We will next update this incident by 16:00 EDT.
-
monitoring
We continue to see cluster operations succeeding, and will actively monitor to ensure the issue does not reoccur.
-
resolved
All cluster operations are succeeding normally at this time.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Issue impacting Basic and Standard tier cluster creation in AWS.
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We have identified an issue impacting Basic and Standard tier cluster creation in AWS. We have conducted an initial investigation and actively working to resolve the issue.
-
resolved
We have identified and rolled out a fix for the issue impacting cluster creation in AWS. This incident is now resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Upstream cloud provider API is unavailable
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
identified
We are aware of errors in the Google Cloud API which may impact cluster creation and edit operations for CockroachDB Cloud Advanced clusters in GCP. We will update this status page with more information as it becomes available.
-
identified
In addition to cluster operations, backups to Google Cloud Storage, including managed backups from CockroachDB Cloud clusters on GCP, may be impacted.
-
monitoring
Google has updated their public status page to indicate the underlying issue is resolved, and individual Google Cloud services are in the process of restoring full service. We are continuing to monitor the issue and impact to CockroachDB Cloud.
-
resolved
Google has reported that mitigations have been applied and services are now recovering.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Metrics Pages Unavailable in Cockroach Cloud Console for Basic and Standard Clusters
Affected: Cloud Console
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently investigating an issue that is preventing some metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters. Metrics data collection remains unaffected; however, we are seeing unexpected errors when retrieving metrics for...
-
identified
We’ve identified the source of the issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters. The underlying service has recovered, and metrics visualizations are now loading as expected. We are continuing to monitor...
-
monitoring
Metrics visualizations are now loading as expected. We are continuing to monitor system behavior to ensure stability and will mark this incident as resolved once we confirm full recovery.
-
resolved
The issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters has been fully resolved. Metrics are now loading as expected, and we’ve received confirmation from our service provider that the underlying issue has been...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
A brief network disruption affecting clusters in the AWS eu-west-1, eu-west-2, and eu-central-1 regions.
Affected: CockroachDB Basic/Standard
Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
resolved
Incident Start AWS experienced a brief network blip in the eu-west-1, eu-west-2, and eu-central-1 regions. Clusters in these regions were unavailable between June 1, 19:29–19:35 UTC. Incident End The network issue affecting clusters in the impacted AWS regions has been fully...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Elevated error rate for some Standard and Basic clusters
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We have received alerts indicating elevated error rates on isolated Standard and Basic clusters. The Engineering team is currently investigating the issue.
-
monitoring
The root cause of the issue was identified and resolved. Engineering is monitoring the situation.
-
resolved
The Engineering team has isolated and resolved the elevated error rate for all affected clusters.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Elevated errors on serverless platform
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
resolved
On Tuesday Nov 12, at approximately 1am UTC, we received alerts of elevated errors due to an underlying host hardware failure. The workloads on this host were migrated to a healthy host and the errors resolved once the migration was complete.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
AWS Advanced cluster creation delayed
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We are aware of an AWS incident that is preventing successful completion of Advanced cluster creation and region addition. Once the cloud provider has resolved their incident, we will ensure clusters create successfully.
-
identified
We are continuing to monitor the upstream incident.
-
resolved
The upstream incident has been resolved, and we have confirmed that cluster creation is working normally.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Small number of serverless clusters unavailable
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are aware of an issue affecting a small number of serverless clusters that is causing unavailability. Our SRE team is investigating.
-
resolved
We have deployed a mitigation and all clusters are available again.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Some AWS clusters not showing available backups on the "Backup and Restore" page
Affected: CockroachDB Advanced
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We have identified an issue where some AWS clusters are unable to list available backups on the "Backups and Restore" page. Backups are still being performed and available if a restore of a database or table is required.
-
resolved
We have deployed a fix which has resolved the issue.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Increased error rate in Serverless restore operations
Affected: CockroachDB Basic/Standard
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are investigating increased errors in restore operations to serverless clusters.
-
resolved
We have corrected a misconfiguration that was causing restore errors, and have applied the fix to all serverless clusters.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
CockroachDB Unavailable in Azure Central US
Affected: CockroachDB Advanced
Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: Azure.
Started during Azure: Storage – Impact to multiple services in Central US.
-
identified
CockroachDB clusters with nodes in Microsoft’s Central US region may be unavailable. We are monitoring the situation and waiting for an update from Microsoft. For more information see Microsoft’s status page at https://azure.status.microsoft/en-us/status/
-
monitoring
Latest update from Microsoft status page is all green, with some residual effects. CockroachDB clusters with nodes in Microsoft’s Central US region are all up and running with some background jobs like backup failing. We are continuously monitoring the status of the clusters.
-
resolved
We have now verified that all Azure clusters are operational and have completed backup jobs.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Dedicated Cluster Operations Errors
Affected: Cloud Console, CockroachDB Basic/Standard, CockroachDB Advanced
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We are investigating an issue that is preventing CockroachDB Cloud Dedicated cluster creation and edit operations from running successfully.
-
identified
We are continuing to work on a fix for this issue.
-
identified
We are continuing to work on a fix for this issue.
-
identified
We have further identified that this issue may impact pages within the Cloud Console that connect to Serverless and Dedicated clusters. SQL connections to both Serverless and Dedicated clusters are working normally. We are continuing to investigate the cause of this outage and...
-
identified
We are continuing to work on a fix for this issue.
-
monitoring
We have resolved the underlying cause of the outages, and are monitoring to ensure the system remains operational.
-
resolved
Cluster operations and Cloud Console pages are restored to normal working order.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Azure cluster creation and modification impaired
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Azure.
-
identified
Due to a change made by our cloud provider, cluster creation and cluster modification operations on Azure (only) are currently impaired while we prepare a fix.
-
monitoring
Azure cluster creation is now operational. Some existing clusters will be updated during their pre-scheduled maintenance windows to fix cluster modification on those clusters.
-
monitoring
Due to capacity issues in the Azure East US region, some cluster operations such as cluster creation and modification may fail if East US is the target region.
-
resolved
Azure is no longer experiencing capacity issues in eastus.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster setting update caused a brief latency increase and availability impact on a single Serverless cluster.
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
resolved
A cluster setting update resulted in higher than expected CPU activity, which caused increased latency and caused unavailability for a single Serverless cluster. The setting was rolled back to prevent further impact. The issue has been resolved. There should be no further...
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Intermittent failures when creating Azure CC dedicated clusters
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Azure.
-
monitoring
Azure is currently experiencing capacity issues for VMs in US regions. Users may experience intermittent failures creating clusters. Cockroach Labs is monitoring for errors. Azure notes that users "may receive error notifications when performing service management operations on...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
GCP Networking Outage
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
monitoring
We are aware of an ongoing Google Cloud Platform networking issue. This is impacting a select number of GCP Cockroach Cloud clusters. Cockroach Labs engineers are actively monitoring the situation. More information can be found here:...
-
resolved
The Google Cloud Platform networking issue has been resolved. If you have any questions or concerns please submit a ticket to our support portal.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
CockroachDB Cloud Dedicated cluster creation in AWS us-east-1 degraded
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
monitoring
We are currently monitoring an incident with AWS firehose creation in `us-east-1` that is causing cluster creation in `us-eastt-1` to take longer than normal. There is no impact to existing clusters. Our engineers are monitoring the situation.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Degraded connectivity between clusters and external services.
Affected: CockroachDB Basic/Standard, CockroachDB Advanced
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
A subset of clusters are unable to reach external services, affecting features like CMEK, log export, and metrics export on impacted clusters. The clusters themselves remain available. We're actively working on a mitigation and will post updates until the issue is resolved.
-
identified
We are continuing to work on a fix for this issue.
-
identified
Fix has been identified and deployment to impacted clusters in progress. All CockroachDB Serverless environments are mitigated now.
-
resolved
Fix deployed on all affected clusters. CockroachDB Dedicated is mitigated as well. All services back to normal and incident closed.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Clusters on Azure may be experiencing instability
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: Azure.
-
investigating
Some customers on Azure may be experiencing instability with their clusters, but all clusters remain up and functional at this time.
-
resolved
We have replaced the failing nodes and all clusters are stable now.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Failures on cluster operations
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
identified
We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.
-
identified
We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.
-
monitoring
Vertical scaling of clusters on AWS may take longer than expected.
-
resolved
All cluster operations should now succeed. We are monitoring for any additional failures.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Severless clusters in GCP europe-west1 inaccessible
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
resolved
Serverless clusters in GCP europe-west1 were inaccessible.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster operations failing due to quay.io registry outage
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
monitoring
The Quay.io container registry is experiencing issues at the moment. CockroachDB Cloud is monitoring updates - https://status.quay.io/
-
resolved
Cluster operations appear to once again be healthy. We will continue to monitor for subsequent events/impacts due to the ongoing Quay.io issue.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Single sign-on to console is failing
Affected: Cloud Console
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
monitoring
Due to an upstream provider outage, all authentication methods for the Cockroach Cloud console are failing, other than authenticating via username and password. Existing sessions will continue to work until they time out.
-
resolved
The upstream provider incident has been resolved, and SSO authentication is working again.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Cluster operations failing due to quay.io registry outage
Affected: CockroachDB Advanced
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
The Quay.io container registry is experiencing issues at the moment. Cockroach Cloud is monitoring updates - https://status.quay.io/incidents/3dq7w1tvs16f?u=bcn3g37v5rgb
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
Quay.io has indicated that the incident is resolved on their end.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
AWS is experiencing networking issues in the us-east-1 region
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
monitoring
We've been advised of a networking issue in the AWS us-east-1 region that may affect cluster creation. We're currently waiting on an update from AWS for an estimated recovery time. https://health.aws.amazon.com/health/status
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Connectivity Issues
Affected: CockroachDB Basic/Standard
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
investigating
We are currently looking into connectivity issues in our Serverless environment. The impact is currently isolated to our `Europe-West-1` Serverless cluster in GCP
-
investigating
We are continuing to investigate this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Some serverless clusters may be unable to restore backups
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating the issue.
-
identified
We've identified the root cause and are currently working on a mitigation.
-
monitoring
Incident has been mitigated and we are currently monitoring.
-
resolved
Incident has been resolved.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Serverless intermittent connection failures
Affected: CockroachDB Basic/Standard
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
Serverless clusters are experiencing intermittent connection failures in the AWS us-east-1 region.
-
resolved
Our engineers have deployed a fix and the issue is now resolved
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.
Serverless Cluster Resource Limit Throttling
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased. Currently only AWS eu-west-1 and ap-south-1 host clusters are impacted
-
investigating
Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased.
-
identified
The scope of the impact has been limited to serverless clusters running AWS eu-west-1 and AWS ap-south-1. Clusters that have hit their resource limits will be throttled despite having their limits increased. One of two issues identified has been mitigated.
-
resolved
This issue has now been resolved. Users with clusters that were previously throttled will now see their changes to resource limits reflected.
Excerpts are shortened. Read the full incident on CockroachDB Cloud's status page.