Hyperstack
AI. Source: Hyperstack status page.
- Incidents, last 90 days
- 3
- Median to mitigation
- 5h 36m
- Median to resolution
- 5h 36m
- p95 to resolution
- 11h 39m
- Naming an upstream provider
- 0%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A zone, region or data center became unavailable | 1 |
| Orchestration, control plane, service discovery or load balancing failed | 1 |
| A schema or data migration caused the failure | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
US-1 - Network Maintenance
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Hyperstack services and resources running in the US1 region are experiencing a degradation in connectivity due a data center outage. The ETA for restoration is September 2nd 3:00 AM UTC [9PM MDT].
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Emergency Maintenance - Hyperstack Platform and API Services
Affected: Hyperstack Portal, Hyperstack API
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are performing an emergency maintenance on the Hyperstack platform and API Services to address a platform issue. During this window, the Hyperstack console and public API may be unavailable or return errors. Existing running workloads (VMs, containers, and Kubernetes...
-
investigating
We are continuing to investigate this issue.
-
identified
The issue has been identified and a fix is being implemented.
-
monitoring
A fix has been applied and the services are returning to normal operation. The console will be accessible again in a while along with the API services.
-
resolved
Emergency maintenance is complete. The Hyperstack console and API Services have been restored and are operating normally. All components have returned to full service . If you continue to experience issues, please contact our support team at support@hyperstack.cloud Thank you...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Hyperstack Support Portal Migration
Affected: Hyperstack Portal
Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
As previously communicated via email we are currently in the process of transitioning the Hyperstack Support Solution. We will be temporarily disabling the Live Chat option but we will still be able to respond to queries over email via support@hyperstack.cloud. We apologies for...
-
resolved
The transition of the Hyperstack Support Solution is still in progress and we are adjusting this update for the Hyperstack Portal to be in a Maintenance window. The Live Chat option is still currently disable but we will still be able to respond to queries over email via...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Service degredation in CA1 Region
Affected: Canada 1 Datacentre, Hyperstack Portal, Infrahub Customer Portal, Hyperstack API
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating an issue in the Canada-1 region following emergency maintenance. This has resulted in some hosts not becoming active following reboot. This has also resulted in a reduced selection of deployments available. We are working to resolve this issue as...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
API Degradation affecting all Regions
Affected: Norway 1 Datacentre, Canada 1 Datacentre, Hyperstack Portal, Hyperstack API, US 1 DataCenter, Canada 2 Datacentre
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are investigating degraded API availability affecting the Hyperstack Console. Some API-dependent workflows may be degraded or unavailable.
-
resolved
Hotfix v1.51.5 has been successfully deployed for infrahub api. Server cpu load stable and apis are responding.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Hyperstack API Degradation
Affected: Hyperstack API
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are investigating degraded API availability affecting the Hyperstack Console. Some API-dependent workflows may be degraded or unavailable.
-
resolved
Hyperstack API availability has been restored.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Hyperstack Service Degradation
Affected: Hyperstack Portal, Infrahub Customer Portal, Hyperstack API
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Some customers may be unable to access the Hyperstack console or related services. We are working to restore access as quickly as possible.
-
monitoring
We have identified the issue and implemented a fix. Services have recovered, and customers should now be able to access the console normally. We will continue to monitor for stability.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Network in CANADA-1 and NORWAY-1 regions
Affected: Canada 1 Datacentre
Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
Our monitoring has identified network issues that are affecting these two regions. This is currently under investigation and work is underway by our team to return service to expected levels.
-
investigating
We are continuing to investigate this issue.
-
monitoring
The issue appeared to be related to our DC networking, which impacted our external connectivity from the data center. Our engineers have applied a fix, and the network is back in the datacenter. All the services are back and working fine. We will be actively monitoring for any...
-
resolved
We are happy to advise that all network-related issues have been fully remediated, and services are working normally. We apologize for any inconvenience caused during this time. If you continue to experience any connectivity problems with resources in any of our regions, please...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
NVIDIA Helm Repository Unreachable
Affected: Norway 1 Datacentre, Canada 1 Datacentre, Hyperstack Portal, Hyperstack API, US 1 DataCenter, Canada 2 Datacentre
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
Summary We are currently investigating an issue affecting cluster reconciliation. The NVIDIA Helm chart repository (https://helm.ngc.nvidia.com/nvidia) is returning errors and appears to be unreachable. Impact Clusters that depend on NVIDIA Helm charts may fail to reconcile to...
-
investigating
We are continuing to investigate this issue.
-
identified
Summary We have narrowed down the issue to just cluster creation. Reconciliation for existing clusters (incl adding/removing nodes/groups) aren't affected. Impact Clusters that depend on NVIDIA Helm charts may fail to create. Observed Error 502 Bad Gateway when attempting to...
-
resolved
Summary The underlying issue has been resolved. New clusters are now creating successfully and transitioning to an Active state as expected. Impact Cluster creation and reconciliation processes are now fully operational. Current Status All systems are functioning normally. We...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CANADA-1 (CA-1) Region Network Issues
Affected: Canada 1 Datacentre, Hyperstack Portal, Infrahub Customer Portal
Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently investigating a network connectivity issue affecting customer VM reachability from the public internet in CANADA-1 (CA-1) Region. Our on-site team is actively engaged and working to resolve the fault. What we know: • The issue has been isolated to the network...
-
monitoring
The issue appeared to be related to internet routing with our DC Network Partners, which impacted our external connectivity from the data center. They have provided the fix, and the network is back again in the datacenter. All the services are back and working fine. We will be...
-
resolved
The issue impacting external connectivity from the data center has been fully resolved for all customers at 07.00 UTC. The root cause was identified as an internet routing issue with our data center network partners. Our DC Network Partners have implemented the required fix, and...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CANADA-2 (CA-2) Region Network Issues
Affected: Canada 2 Datacentre
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently aware of a network-related issue affecting our CANADA-2 Data Centre. Our engineering teams are actively investigating the matter. We will provide further updates on the status and progress as soon as they become available.
-
identified
We are currently investigating a network connectivity issue affecting customer VM reachability from the public internet. Our on-site team is actively engaged and working to resolve the fault. What we know: • The issue has been isolated to the network layer at the data centre...
-
monitoring
Our engineering teams have successfully implemented a fix at the network layer, and full reachability from the public internet to customer VMs has been restored. Additionally, the ability to create new VMs with functional network interfaces is now fully operational. We are...
-
resolved
We are happy to advise that all network-related issues have been fully remediated in the region, and services have returned to normal. We apologise for any inconvenience caused during this time. If you continue to experience any connectivity problems with resources in the...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Intermittent Failures on Launching H100 VMs in CANADA-1 (CA1) Region
Affected: Canada 1 Datacentre
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating an issue where newly created H100 virtual machines in our CANADA-1 (CA-1) Region may fail to launch and report an error during creation. Other VM/Flavor types and regions are not impacted. Our engineering teams are actively working to identify the...
-
identified
We have identified the root cause of the intermittent H100 VM creation failures in the CANADA-1 (CA-1) region. Our engineering team is now working on the fix and will monitor recovery as remediation progresses. We will provide another update once normal H100 VM provisioning has...
-
resolved
The issue affecting the H100 VM creation intermittently in the CANADA-1 (CA-1) region has been resolved. Our engineering team has completed fixing the impacted hosts and validated that new H100 VMs are launching successfully again. All systems in the CANADA-1 (CA-1) Datacentre...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Intermittent failures to launch for on-demand Kubernetes clusters
Affected: Canada 1 Datacentre
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating intermittent failures to launch on-demand Kubernetes clusters in our CA1 region. When failing, the cluster build will respond with an error "Infrastructure reconciliation failed".
-
identified
The issue has been identified and a fix is being implemented.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Connectivity Issues in Canada (CA1)
Affected: Canada 1 Datacentre
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Investigating – Connectivity Issues in Canada (CA1) We are currently investigating an issue affecting newly created and restored virtual machines in our Canada (CA1) region, where some instances may not become reachable immediately after launch (e.g. SSH or network access). Our...
-
identified
The issue has been identified and service operations are beginning to recover. Our engineering teams are continuing to investigate the root cause and are working to fully remediate the problem as soon as possible.
-
identified
The root cause has been identified. Remediation work and validation are in progress and we are preparing to deploy a fix on the Hyperstack side.
-
resolved
After identifying and remediating the root cause, this problem has now been addressed. We have verified that everything in the Canada1 region has returned to normal operations.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA-1-Network Issues-Canada Data Centre
Affected: Canada 1 Datacentre
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently aware of a network-related issue affecting our Canada Data Centre. Our engineering teams are actively investigating the matter. We will provide further updates on the status and progress as soon as they become available.
-
identified
The issue has been identified as a control plane failure in CA1, and remediation efforts are underway.
-
identified
We are actively replacing the failed control plane to restore full system health. This action requires a momentary network reset. Some customers should expect a brief disconnection, during which all active network sessions will be interrupted for a few seconds. Service is...
-
monitoring
Engineering have been investigating intermittent connectivity interruptions in our Canada (CA1) region. Our engineering teams identified the issue as control-plane instability affecting routing services, which under certain conditions caused brief disruptions for a subset of...
-
monitoring
We have implemented additional stabilisation measures in our Canada (CA1) region and are currently seeing improved stability and healthy traffic levels. Our engineering teams have reduced routing instability and are continuing to monitor the region closely to ensure sustained...
-
resolved
We are happy to advise network-related issues have been remediated and services have returned to normal. We apologise for any inconvenience caused during this time.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Maintenance in US1
Affected: US 1 DataCenter
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage maintenance in our US1 region to replace faulty drives. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate your patience and...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
Our storage expansion in CA1 has finished. Thank you for your patience.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Brief Outage Detected on HyperStack and API Services
Affected: Hyperstack Portal, Hyperstack API
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Cloudflare.
-
investigating
We are currently aware of a service disruption affecting HyperStack and API Services. This issue appears to be related to a wider incident on the Cloudflare Global Network, which Cloudflare is actively investigating. Our team is monitoring the situation closely and working to...
-
monitoring
We have implemented mitigations for the issue impacting the Cloudflare Global Network by rerouting our API calls externally, and we are seeing recovery across HyperStack and API Services. Our systems are operating normally, and we are continuing to closely monitor them for...
-
resolved
The Cloudflare Global Network incident has been fully resolved, and our long-running tests have shown no issues with the system. Our HyperStack and API Services are fully recovered and are operating normally. We are flagging that the issues have been resolved based on system...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: The text does not say enough to tell. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
We have completed the storage expansion and all operations have returned to normal.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
NO1: Upstream Network Degradation
Affected: Norway 1 Datacentre
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
We have identified that our upstream ISPs in the NO1 datacentre are experiencing outages on certain network paths affecting connectivity to specific external locations. We are working with our ISPs to resolve this issue as soon as possible.
-
resolved
The upstream network links to Norway have been restored. Thank you for your patience.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
US-1 - Service Performance Degradation
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating reports of a service impairment impacting the US-1 region. Customers in this region may experience degraded performance or issues when deploying new resources. Our engineering team is actively working to identify the root cause. We will share...
-
resolved
The issue was identified and fixed.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
US-1 - Degraded Service Performance
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating reports of a potential service impairment affecting the US-1 region. Customers in this region may experience degraded services. Our engineering team is actively working to identify the root cause and will provide an update as soon as more...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
NO1: Upstream Network Degradation
Affected: Norway 1 Datacentre
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently experiencing degraded performance on one of our upstream ISP connections in the NO1 datacentre. While network traffic continues to operate normally through redundant paths, customers may experience reduced available bandwidth until the affected link is fully...
-
identified
We have further identified that one of our upstream ISPs in the NO1 datacentre is experiencing outages on certain network paths affecting connectivity to specific external locations. To maintain stability, we have temporarily disabled the affected ISP link and rerouted all...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Hyperstack Frontend–Backend Connectivity Issue
Affected: Hyperstack Portal
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We’re currently investigating an issue where the Hyperstack frontend is unable to communicate with backend services. All API endpoints are operational, but the dashboard and web interface may appear unavailable or unresponsive. Our engineers are actively investigating the...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Storage backend maintenance
Affected: Hyperstack API
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We will be performing maintenance on our storage backend. During this time there will be a slight performance impact which may cause storage-related operations to take longer to complete. Data resiliency will not be impacted.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Connectivity issues to Hyperstack VMs and Portal
Affected: Norway 1 Datacentre, Canada 1 Datacentre, Hyperstack Portal, Hyperstack API
Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
monitoring
We are aware of intermittent connectivity issues affecting access to Hyperstack VMs and the portal. - The issue appears to be related to internet routing, impacting certain regions (notably Asia). - This is not isolated to a specific ISP or customer environment. Our team is...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Electrical Issue
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We are currently experiencing an electrical issue that might affect some systems, more information to come.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CPU-only Instance Availability in CA1
Affected: Canada 1 Datacentre
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
We are currently experiencing a shortage of CPU-only nodes in the CA1 region. Customers may see errors when attempting to launch CPU-only instances. Our infrastructure team is actively working to bring additional capacity online to restore availability. GPU-backed instances are...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
NO1- Some users may experience outages due to electrical issues.
Affected: Norway 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We're experiencing intermittent electrical-related issues in NO1 region, a small selection of racks are currently identified as affected which in turn is causing unexpected reset of certain hosts. We're currently working with our on-site engineering team and are in the process...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 AI Studio Outage
Affected: Canada 1 Datacentre
Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
There will be a short outage to AI Studio while backend services are reconfigured.
-
identified
We are continuing to work on a fix for this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 Storage Node replacement
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
At 1600 UTC we will be shutting down a storage node to perform maintenance. During this time there may be a slight decrease in IO performance for storage volumes.
-
identified
We are continuing to work on a fix for this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Storage Degradation
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating a downed storage node. There may be a slight effect on performance as the cluster rebalances.
-
identified
We have identified an inaccessible storage node and are attempting remediation to access.
-
monitoring
The storage node has been brought back online and the cluster is now recovering. We will continue to monitor performance. While recovery is performed there may be a minor performance impact to storage in customer VMs.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
US1 Outage
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently investigating an outage affecting multiple nodes in the US1 region. We are currently working toward finding a resolution at this time.
-
identified
We have been informed by our US1 data centre partner that a catastrophic power outage has impacted some pods and compute nodes, affecting a subset of customers. Our engineering team is working around the clock to bring new compute nodes online. For customers being affected, our...
-
resolved
We have successfully restored capacity in the US1 region following the recent outage at our data centre. Additional compute nodes have been brought online, and stock levels have been replenished to meet on-demand customer requirements. Our engineering and support teams will...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Service Outage in CA1
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We're currently investigating a storage-related issue in our CA1 region. A subset of storage nodes is unavailable, which is affecting performance and availability for some virtual machines and operations that rely on block storage. - You may experience delays launching,...
-
identified
Our CA1 data centre experienced a power disruption. While critical infrastructure responded as expected, some backend systems remain affected and we are actively working with our partners to restore full service.
-
monitoring
All services are now restored in CA1. However, clients might still expected IOs degradation when our storage nodes are being rebalanced.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
We have completed the Ceph expansion and all operations have returned to normal.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
NO1 Network Connectivity Issue
Affected: Norway 1 Datacentre
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating a network connectivity issue affecting the NO1 region. Our team is working to identify the root cause and restore connectivity as quickly as possible. Updates will follow as more information becomes available.
-
identified
The issue has been identified and a fix is being implemented.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
API Service Degradation - Instance Creation Impacted
Affected: Canada 1 Datacentre, Hyperstack API
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently experiencing an API service issue affecting instance creation and network configuration changes. Our engineering team has identified the root cause and is actively working on remediation. Existing instances and network configurations remain unaffected.
-
identified
The issue has been identified and a fix is being implemented.
-
identified
We are continuing to work on a fix for this issue.
-
identified
The fix was successfully tested and we are actively rolling it out in our Canada1 region. VM creation and management will be recovered by GPU types, starting with the RTX A6000.
-
resolved
The issue impacting instance creation and network configuration changes in the CA1 region has been resolved. As of now, all GPU types have been restored to normal operation, and VM creation and management are functioning as expected. We appreciate your patience while our...
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
The storage expansion has finished. Thank you for your patience.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in NO1
Affected: Norway 1 Datacentre
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our NO1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
The expansion has finished. Thank you for your patience.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
The expansion has finished. Thank you for your patience.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Potential Impact on Service Stability Due to Extreme Heat in CA-1
Affected: Canada 1 Datacentre
Failure mode: A zone, region or data center became unavailable. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
NexGen wishes to inform you that Montreal is currently experiencing an extreme heatwave, with temperatures reaching up to 46°C. This unprecedented heat is placing significant strain on our data centre infrastructure in CA-1 region, potentially affecting system stability. We are...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Disruption with Hyperstack services in US1
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We are currently experiencing a disruption with Hyperstack services in the US1 region. Existing workloads continue to run without interruption; however, API calls to this region are temporarily unavailable. Our team is investigating and will provide updates as they become...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Temporary Storage Performance Degradation
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently addressing an issue within our CA1 region storage cluster. One of the storage nodes has experienced a failure and is being investigated. While the system is designed to self-heal and redistribute data automatically, you may notice temporary performance...
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Disruption with Hyperstack services in US1
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We are currently experiencing a disruption with Hyperstack services in the US1 region. Existing workloads continue to run without interruption; however, API calls to this region are temporarily unavailable. Our team is investigating and will provide updates as they become...
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 Region External Network Degradation
Affected: Canada 1 Datacentre
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
CA1 region experienced a complete network outage lasting approximately 5 minutes due to an upstream issue with our internet service provider. The disruption impacted all external connectivity, including access to public services and remote management. Internal services remained...
-
monitoring
We are observing ongoing degraded performance on a small number of routes. Most network traffic remains stable and unaffected. We are continuing to monitor the situation closely and are working with our upstream provider to ensure full resolution.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Ceph Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
The storage expansion in CA1 has finished. Thank you for your understanding.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
Storage Expansion in CA1
Affected: Canada 1 Datacentre
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
We are currently performing a storage expansion in our CA1 region to support future growth and improve performance. Impact: There is no expected disruption to running workloads. However, clients may experience minor performance variations during data rebalancing. We appreciate...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
US-1 Service Outage
Affected: US 1 DataCenter
Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We are currently investigating a service outage affecting the US-1 Data Center. Please note that this incident does not impact any running workloads, but you may experience issues when launching new services. We sincerely apologise for the inconvenience and will provide further...
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Temporary Storage Performance Degradation
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We are currently addressing an issue within our CA1 region storage cluster. One of the storage nodes has experienced a failure and is in the process of being removed from the cluster. While the system is designed to self-heal and automatically redistribute data, you may...
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
The cluster has now been rebalanced, and full data redundancy is restored.
Excerpts are shortened. Read the full incident on Hyperstack's status page.
CA1 - Temporary Storage Performance Degradation
Affected: Canada 1 Datacentre
Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently addressing an issue within our CA1 region storage cluster. One of the storage nodes has experienced a failure and is being removed from the cluster. While the system is designed to self-heal and redistribute data automatically, you may notice temporary...
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
All maintenance on the storage cluster has completed successfully and all active monitoring checks are returning healthy results.
Excerpts are shortened. Read the full incident on Hyperstack's status page.