Apollo GraphQL
Apps and SaaS. Source: Apollo GraphQL status page.
- Incidents, last 90 days
- 5
- Median to mitigation
- 26m
- Median to resolution
- 1h 24m
- p95 to resolution
- 4h 37m
- Naming an upstream provider
- 20%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A configuration change caused the failure | 2 |
| The text does not say enough to tell | 2 |
| A provider or third-party service this vendor depends on failed | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
Graph Artifacts Pipeline Facing Issues
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We're seeing issues deploying new graph artifacts builds. We're currently investigating the root cause
-
monitoring
We've identified an issue with our cloud configuration. We've corrected that and are now seeing recovery
-
resolved
Issue with the graph artifacts pipeline has been resolved. No data has been lost during the incident only delayed while we resolved the issue. We apologize for this disruption
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Service Disruptions Due to Google Cloud Platform Issues
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS, Google Cloud.
Started during Google Cloud: Multiple products in us-central1-b are experiencing network service degradation..
-
investigating
Due to an ongoing outage on Google Cloud Platform (GCP), we're currently experiencing service disruptions across multiple Apollo services. We're actively investigating and will share updates as more information becomes available.
-
investigating
The AWS Uplink endpoint remains available. GraphOS Routers and Gateways using the default multi-cloud configuration can continue to start up and fetch their configuration during this outage, but new schemas cannot be published.
-
identified
Due to an ongoing outage on Google Cloud Platform (GCP), we're currently experiencing service disruptions across multiple Apollo services. We're actively investigating and will share updates as more information becomes available.
-
monitoring
Our third party vendor has confirmed the issue was isolated to a portion of the us-central1-b zone; remaining zones in the region were unaffected. Most Apollo products have recovered, and our engineering team is validating the remaining impacted services. We'll continue...
-
monitoring
Our third party vendor has confirmed the issue was isolated to a portion of the us-central1-b zone; remaining zones in the region were unaffected. Most Apollo products have recovered, and our engineering team is validating the remaining impacted services. We'll continue...
-
resolved
Our third party vendor has confirmed the issue is now resolved, and all Apollo products have recovered.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Organization and user modifications are being incorrectly rate limited
Failure mode: A configuration change caused the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
Some organization and user modifications (user invites, user permission changes, OAuth changes) are failing intermittently. We are working on a fix.
-
monitoring
We are no longer seeing any request failures, but we will continue to monitor until a root cause fix has been deployed.
-
resolved
The fix has been deployed. These failed operations would have appeared as rate limit errors, but the rate limit was being incorrectly applied. We have made changes to ensure the rate limit works as expected.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Registry workflow latency
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Registry service workflow latency has spiked, affecting mulitple internal services serving long-running workflows.
-
monitoring
The issue has been resolved and we're continuing to monitor.
-
resolved
The issue has been resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Embedded Explorer and Embedded Sandbox temporarily down
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
Embedded Explorer and Embedded Sandbox are temporarily down. The issue has been identified and a fix is being deployed.
-
resolved
The issue has been fixed.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Proposals elevated error rate
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We’re currently investigating a rise in error rates for our Proposals service.
-
investigating
We are still investigating elevated error rates for Proposals. We appreciate your patience.
-
identified
We’ve identified the issue and are working on resolution.
-
monitoring
We have applied a fix and have begun to see recovery in our metrics. We will continue to monitor and update the status page once it’s resolved.
-
resolved
The error rate for Proposals has returned to normal, no further impact expected.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Field and coordinate usage metric ingestion
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are investigating suspected data loss for field and coordinate usage metrics for all customers.
-
monitoring
Field and coordinate usage metric ingestion is functional and lag is trending downwards. We estimate 8-10 hours for full recovery.
-
monitoring
The backlog of object field, input object field, and enum value metrics are now being processed but we expect the backlog to take 8-10 hours to be fully processed. For a period of time between 4 am and 9pm on June 27 UTC we expect there to be data loss for input object field and...
-
resolved
The backlock of schema coordinate (object field, input object field, and enum value) metrics has now cleared and there is no more metric ingestion delay. The metrics for input object fields, enum values, and the operation and client breakdown of object fields is inaccurate for...
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Delays in metric ingestion
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating a case where metric ingestion is delayed for some customers. There is no loss of data.
-
investigating
We have identified the cause though customers continue to experience metric ingestion delays while we identify the appropriate mitigation. Data backfill after remediation is still expected after mitigation is applied. We appreciate your patience.
-
identified
We are incrementally applying mitigations which are showing positive results. We feel we are confidently iterating toward recovery. At this time, no data is believed to be lost and metric backfill will continue with the oldest metrics being populated first, though we currently...
-
monitoring
We are confidently seeing recovery. Some metric delay will continue for the next couple hours, though many metric types have already recovered. We will keep this status page open in Monitoring state and update it when we have eliminated our metric lag. Thank you again for your...
-
resolved
The metric ingestion backlog has been fully processed and all metrics have resumed normal ingestion. We do not believe any data was lost as a result of this incident. All metric types should now be operating as expected. Thank you for your patience throughout this incident.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Router Licensing impacting multiple customers
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Some Router deployments are experiencing issues coming online due to a message around licensing. We are investigating.
-
investigating
We are continuing to investigate.
-
monitoring
We have seen recovery for about 50 minutes now and we are continuing to monitor the recovery. If you find that you are impacted by a licensing message on Router startup, please open a ticket with our support team.
-
resolved
After extended monitoring, we have confirmed that the issue is resolved. Post-incident action items will continue after the end of the incident. As with all incidents, we will be conducting an internal post-mortem. We appreciate your patience.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Issue Accessing “Submit a Request” and “My Requests” in Support Portal
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We’re currently investigating an issue where the “Submit a Request” and “My Requests” buttons are not appearing properly in the Apollo Support Portal. As a temporary workaround, tickets can still be created and accessed through the “Submit a Request” button available in the...
-
investigating
We’re continuing to investigate the issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal. At this time, we are working with our external provider to identify the root cause and restore normal functionality.
-
monitoring
We’ve implemented changes to address the issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal. At this time, the portal appears to be functioning normally, and we are actively monitoring the situation to ensure...
-
resolved
The issue affecting the visibility of the “Submit a Request” and “My Requests” buttons in the Apollo Support Portal has been resolved. Portal functionality has been restored, and users should now be able to access these options normally. Thank you for your patience while we...
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Support ticket creation unavailable from Studio UI
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Opening a ticket using the Contact Support button on the [studio.apollographql.com](http://studio.apollographql.com "studio.apollographql.com") site itself is currently failing. Support tickets can still be created through our...
-
monitoring
Support ticket opening functionality has been restored to Studio’s “Contact Support” functionality in the UI. We have mitigated the failures and will monitor before marking as resolved.
-
resolved
We have observed sustained restored functionality in our Studio-initiated support ticket form and are marking this as fully resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Studio Performance Deteriorated
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We're seeing issues with StudioUI and are investigating to find the root cause
-
investigating
We're continuing to investigate to find the root cause and resolve the issue
-
identified
We have identified a possible root cause, and we are testing changes to confirm
-
monitoring
We confirmed with internal testing that our fix is working as expected, and customers should be seeing recovery. We're continuing to monitor to confirm.
-
resolved
We have resolved the issue. Customers should see full functionality.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Platform availability for Studio SSO users
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We’re investigating the availability of our platform after a recent deploy.
-
monitoring
We have restored functionality to impacted users SSO access and are monitoring.
-
resolved
SSO login is now fixed.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Schema proposal diff views returning errors
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are aware of an issue affecting GraphOS Studio where users may encounter errors when attempting to view the Changes or Schema Reference tabs within graph proposal. We are investigating and will give an update soon.
-
monitoring
We have identified the root cause of the failures affecting the Changes and References tabs in Schema Proposals. A fix has been deployed and we continue to monitor and investigate.
-
resolved
Proposals errors have been clear for ~2 hours since the rollback reached prod. Marking resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Transactional email delays
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We’re monitoring an interruption in the transmission of transactional emails (e.g., email verification, welcome emails, proposals approvals). We are investigating.
-
monitoring
We’ve identified the root cause of the delays, applied mitigations and recent delivery attempts are succeeding. Some previously attempted deliveries may take time to be retried.
-
resolved
Retries of delayed transaction emails have been successful. If you have not received an email you were expecting for account reasons, we suggest retrying the workflow which triggered the email.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Delays with GraphOS metrics availability and slow loading times for insights pages
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
resolved
We've identified an issue where our metrics ingestion pipeline was falling behind for several hours. Systems have now fully recovered and no data was lost, however some customers may have experienced delays viewing insights data in Studio and / or slower page load times during...
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Degredaded performance across several Apollo GraphOS services
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We're seeing increases in latency across a few different Apollo services. You may be seeing latency issues with proposals, checks, and studio.
-
investigating
Platform API is also seeing degraded performance, which we are investigating
-
investigating
We're still investigating the root cause. We're seeing some recovery, but not to full recovery
-
investigating
We're continuing to investigate the root cause of the issue. Some customers are reporting significant improvements in performance.
-
monitoring
We've implemented a fix and are monitoring the system.
-
resolved
The issue has been resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Schema Publishing Encountering Issues
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
There was a period of degraded performance on schema publishing. The root cause has been identified and service restored.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Elevated Error Rates for Cloud Routers and Studio UI
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We're experiencing elevates error rates due to an upstream DNS provider major incident.
-
monitoring
Our DNS provider has provided an update: "The issue has been identified and a fix is being implemented." We will be monitoring for recovery of our systems.
-
monitoring
We're continuing the monitor the affected services while the upstream provider remediates the issue.
-
monitoring
At this time, our monitors indicate that only Cloud Routers hosted on Apollo infrastructure are impacted as a result of an upstream service provider incident. Studio UI has already recovered. Beyond these two impacted services, there has been no impact to APIs (including Uplink).
-
monitoring
The upstream provider has indicated recovery on their status page (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). At the same time, we have seen our own monitors return to non-triggered states. We will continue to monitor for the next 15 minutes and then declare this...
-
resolved
We are marking this incident resolved on the Apollo side. For any continuing matters, please consult our support portal: https://support.apollographql.com/
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Cloud Router elevated error rates
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
resolved
Cloud Routers are experiencing elevated errors due to an upstream external provider incident.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Federation version minor version build failures
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We’re investigating a potential issue with at minimum Federation versions 2.9, 2.10 and 2.11 that causes builds to fail.
-
investigating
We've rolled back federation versions 2.10 and 2.11 to the previous minor version, which should resolve the issue for those federation versions. We're continuing to investigate potential issues with federation version 2.9.
-
investigating
We've rolled back federation version 2.9 to its previous minor version, which should resolve the issue for that federation version. We're still working on confirming that the issue is completely resolved.
-
resolved
We've confirmed that the issue with federation versions 2.9, 2.10, and 2.11 is resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Schema Publishing Encountering Issues
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
We're investigating issues in the schema publishing pipeline. There is degraded performance right now. We are investigating the root cause.
-
monitoring
We've identified the root cause of the issue as a third party dependency of Apollo attempting to recover from the AWS outage that has impacted many. The third party is working on remediating the issue and we are continuing to monitor.
-
resolved
No schema publishing or checks issues have been observed for couple of hours now. We are continuing to monitor our services in relation to the AWS outage today on the other open incident.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
AWS us-east-1 Outage Impacting Apollo Services
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS, Google Cloud.
-
monitoring
AWS has reported continued degradation after initial signs of recovery. We are seeing some minor impacts to the AWS Uplink service, which we are continuing to monitor. As we've previously stated we recommend customers to allowlist AWS and GCP Uplink endpoints which Apollo...
-
monitoring
Apollo Products hosted in AWS may be running into performance issues as problems at the Cloud Hosting Provider are remediated. Please use the "contact support" button in Studio if you require further bespoke assistance from Apollo.
-
monitoring
We are continuing to monitor for any further issues.
-
monitoring
We are continuing to monitor for any further issues.
-
monitoring
All systems appear operational but some of our 3rd party dependencies have not fully recovered. Will continue to monitor.
-
monitoring
Systems remain stable and operational on our side, though we are still monitoring dependent software closely before we mark this resolved. We will post another update within 60 minutes.
-
resolved
We will mark this as resolved as issues impacting our service have been resolved either through mitigation or upstream provider resolution. While some of our dependent services have indicated there is on-going stabilization of their own systems (following on the AWS us-east-1...
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
AWS us-east-1 Outage Impacting Multiple Apollo Services
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS, Google Cloud.
-
investigating
Outage in AWS us-east-1 is impacting multiple Apollo products including Uplink (AWS), Cloud Routers and customer's ability to open supports tickets. By default Routers are configured to use GCP and AWS for Uplink, however if you have configured your router to use AWS exclusively...
-
monitoring
AWS is reporting signs of recovery on their status page and we are seeing recovery in several of our triggered monitors for systems which rely on AWS, however, we continue to monitor the overall situation as AWS processes a backlog of events in us-east-1. As noted in the...
-
resolved
As a result of the AWS us-east-1 system recovery, our own systems have also recovered, including AWS Uplink and AWS-hosted Cloud Routers. At this time we will mark this issue as Resolved, though we will continue to monitor for any possible reoccurrence. As noted in a previous...
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.
Schema proposal changes are unable to be saved
Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
identified
A change to proposal permissions has blocked proposal revisions from being saved. We are in the process of reverting this change.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Apollo GraphQL's status page.