Dagster+

Affected: API, Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. monitoring 2026-10-07 20:26 UTC

   Between 3:15 PM and 3:45 PM ET (12:15–12:45 PM PT, 19:15–19:45 UTC) on October 7, a backend issue delayed communication between Dagster+ and customer agents. During this window, some schedule, sensor and declarative automation ticks failed with "Unable to reach the user code...

2. resolved 2026-10-09 17:26 UTC

   We've deployed several fixes and performance optimizations that have resolved this issue, and should prevent it in the future.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-08-29 17:30 UTC

   Impact: Dagster+ hybrid agents connecting to the US region via AWS PrivateLink experienced connection loss. Duration: ~30 minutes (17:35 – 18:05 UTC / 13:35 – 14:05 ET on 2026-08-29) Summary: As part of a planned infrastructure migration, we rotated the network load balancers...

2. postmortem 2026-08-31 18:15 UTC

   **What happened** During a planned network migration on 2026-08-29 17:35 UTC, we disconnected existing PrivateLink consumer VPC endpoints as part of the cutover. Hybrid customer agents connecting via PrivateLink lost their connections and were not able to reconnect until the new...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2026-04-29 18:16 UTC

   We observed an increase in Dagster+ serverless runs failing to start between 10:50AM and 11:00AM PST. Our cloud provider indicated an incident on their end which we have linked to these issues. Failure rate has since gone back down, we will post more information as we learn more.

2. resolved 2026-04-29 19:28 UTC

   Our cloud provider has confirmed that there was a service degradation between 10:48 AM and 11:01 AM PDT causing an increase in task launch error rates. We are implementing additional retry behavior and fault tolerance to prevent similar issues in the future from causing runs to...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-02-24 00:33 UTC

   A configuration change caused an delay in scheduling runs and sensors. We identified and fixed the issue. The issue was identified at 12:15pm PST and was resolved at 1:30pm PST.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: CDN or edge layer failed or served bad content. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-11-18 14:04 UTC

   We're investigating an issue where some users are seeing a version of the old Dagster+ UI. We've identified the root cause of the problem and are investigating a fix.

2. resolved 2025-11-18 14:53 UTC

   This issue has now been resolved - all users should now be seeing the new Dagster+ UI again.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2025-07-01 14:00 UTC

   We observed some DNS resolution related API latency and errors from 10:10pm-10:35pm EST. We've identified the issue and have released a fix.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2025-06-27 18:56 UTC

   We observed some API latency and errors from 2:15pm-2:45pm EST. We've identified the issue and have released a fix.

2. resolved 2025-06-27 18:58 UTC

   This issue has now been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: Google Cloud.

1. monitoring 2025-06-12 18:59 UTC

   GCP is reporting degraded status for a number of services: https://status.cloud.google.com Dagster+ is largely unaffected. Users using Google to authenticate to Dagster+ may see issues with sign in and authentication. We'll continue to monitor and update with any other Dagster+...

2. resolved 2025-06-13 01:08 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-04-24 18:11 UTC

   We're investigating an uptick in Dagster+ UI errors beginning at 10:40PT.

2. monitoring 2025-04-24 18:19 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-04-24 18:31 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: CDN or edge layer failed or served bad content. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-03-27 10:51 UTC

   We detected errors for some users when accessing the catalog Dagster+ UI. Using incognito mode in the browser solves it. We are currently investigating the problem to provide fast mitigation.

2. identified 2025-03-27 11:53 UTC

   Issue was identified and a fix is currently being worked on.

3. monitoring 2025-03-27 12:24 UTC

   A fix was deployed and we are monitoring the situation.

4. resolved 2025-03-27 14:21 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2025-03-23 19:30 UTC

   Between 11:30 AM PST and 12:15 PM PST on Sunday, March 23rd, some Dagster+ Serverless runs failed to start with a “run worker failed” error. We are working with our cloud provider to investigate the root cause.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-02-27 09:27 UTC

   We've received reports from multiple users having trouble accessing the Dagster Cloud UI. Our engineering team is currently investigating.

2. investigating 2025-02-27 10:25 UTC

   We are continuing to investigate this issue.

3. investigating 2025-02-27 10:51 UTC

   We have rolled out a fix and are monitoring service restoration. The Dagster+ UI is available again. STDERR and STDOUT tabs on runs are temporarily disabled.

4. monitoring 2025-02-27 10:52 UTC

   We have rolled out a fix and are monitoring service restoration. The Dagster+ UI is available again. STDERR and STDOUT tabs on runs are temporarily disabled.

5. resolved 2025-02-27 11:29 UTC

   The Dagster+ UI is restored. Agents, runs, and automations are not affected. We will continue to work to restore STDOUT and STDERR in the UI during US business hours. In the meantime, compute logs will continue to upload but will not be accessible via the UI. [UPDATE 1:10 pm...

6. postmortem 2025-03-07 21:12 UTC

   In addition to [structured event logs](https://docs.dagster.io/guides/monitor/logging/#structured-event-logs) that appear in the Dagster UI, Dagster supports [raw compute logs](https://docs.dagster.io/guides/monitor/logging/#raw-compute-logs) to capture STDERR and STDOUT. During...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2025-02-14 15:00 UTC

   "Between 7:05AM PST and 7:25AM PST there were elevated errors while accessing the Dagster+ UI. The cause of the errors has been identified and fixed and the problem should now be resolved"

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API, Dagster Cloud UI

Failure mode: A configuration change caused the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2025-02-07 15:37 UTC

   Some runs containing ops tagged with concurrency keys that have not been configured explicitly or with a default limit are getting blocked. Separately, there are some UI issues affecting the concurrency key settings page. We have identified fixes for these and are in the process...

2. monitoring 2025-02-07 15:56 UTC

   The fixes has been rolled out and previously blocked runs appear to be progressing. We are continuing to monitor for any issues. The previous blocked runs should progress with their step execution, but may not reach a terminal state and may need to be manually terminated.

3. resolved 2025-02-07 16:11 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-02-05 23:44 UTC

   We are reopening the previous incident, which only partially resolved the issue. We're investigating a more comprehensive fix.

2. identified 2025-02-06 00:18 UTC

   We're rolling out the more comprehensive fix right now.

3. monitoring 2025-02-06 00:27 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2025-02-06 00:42 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-02-05 21:25 UTC

   A small number of users are experiencing pages failing to load in the Dagster+ UI. We're in the process of rolling out a fix. As a workaround, you can resolve the problem by doing the following: 1. Close all open tabs to Dagster+. 2. Open up a single tab to Dagster+ and navigate...

2. identified 2025-02-05 21:26 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2025-02-05 23:08 UTC

   The fix has been rolled out for everyone. We are continuing to monitor.

4. resolved 2025-02-05 23:17 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2024-10-24 19:55 UTC

   Some users are seeing degradations loading the Dagster+ UI. We are investigating and working on a fix.

2. identified 2024-10-24 20:12 UTC

   We've identified the issue and are working on a fix.

3. monitoring 2024-10-24 20:58 UTC

   We have rolled out a fix.

4. resolved 2024-10-24 21:05 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2024-10-01 01:08 UTC

   Following the earlier incident, a configuration change is required that requires a brief outage. We expect the downtime for less than 2 minutes, and we do not expect any runs to fail but just pause. The outage will occur in the next hour.

2. resolved 2024-10-01 01:12 UTC

   This configuration change has been completed.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API, Dagster Cloud UI

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2024-09-30 23:36 UTC

   We are currently experiencing an issue with Dagster Cloud's storage engine that is preventing access to the service. We are immediately investigating the issue and will post continued updates here.

2. investigating 2024-09-30 23:44 UTC

   We are in the process of rolling out a change that is intended to restore access. We will continue to post updates here as we work to restore service.

3. identified 2024-09-30 23:45 UTC

   We are in the process of rolling out a change that is intended to restore access. We will continue to post updates here as we work to restore service.

4. monitoring 2024-10-01 00:00 UTC

   The fix has been implemented and we are monitoring to confirm that service has been restored.

5. resolved 2024-10-01 00:25 UTC

   This incident has been resolved. Some runs may have been delayed or failed during the outage. We do not expect any further interruption of service.

6. postmortem 2024-10-01 17:57 UTC

   During this weekend's maintenance window, we conducted a major update of our storage layer to enable continued scale and resiliency. The major update was completed over the weekend without a problem. Yesterday, the team worked through a set of follow-up tasks to finalize the...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. monitoring 2024-08-26 22:56 UTC

   We are investigating a partial outage which may rarely cause runs to intermittently fail. Failed runs should succeed upon retry.

2. resolved 2024-08-27 03:04 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2024-08-20 02:30 UTC

   An unexpected autoscaling event in our storage engine caused a small number of requests to our API to time out between 2:12 UTC and 2:19 UTC. This may have resulted in some runs that were running during this time period to fail. We've resolved the issue and we are continuing to...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2024-08-20 02:12 UTC

   Some users experienced errors in schedule, sensor, and auto-materialization evaluations between 2:12 and 4:22 related to an earlier API outage. This issue has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2024-08-01 19:31 UTC

   We've identified an issue with the run timeline UI overfetching runs. We're working on rolling out a fix.

2. monitoring 2024-08-01 21:08 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2024-08-01 21:19 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2024-06-03 21:56 UTC

   A configuration change to our load balancer has resulted in the Dagster Cloud UI being temporarily unavailable. We are in the process of rolling back the configuration change now.

2. monitoring 2024-06-03 21:58 UTC

   The configuration change has been rolled back and we are monitoring to ensure that access to the Dagster Cloud UI has been restored.

3. resolved 2024-06-03 22:01 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-05-14 18:53 UTC

   Some Dagster Plus Serverless runs are failing to start with an "Agent detected that run worker failed." error. We are working with our cloud provider to investigate the root cause of the problem as well as adding automatic retries for the underlying error that is causing the...

2. investigating 2024-05-14 19:34 UTC

   We are continuing to investigate the root cause of this issue with our cloud provider. We are rolling out a change to add additional automatic retries to runs that are affected by this issue.

3. identified 2024-05-14 19:57 UTC

   Our cloud provider has identified the root cause of the problem and is in the process of rolling back the change that caused it. We'll continue to provide updates as we work to resolve the issue.

4. resolved 2024-05-14 20:16 UTC

   The fix for this problem is now live.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-05-14 12:46 UTC

   Some failed runs are displaying an error message when they are retried in the Dagster Plus UI. We believe we have identified the source of the problem and are investigating a solution.

2. monitoring 2024-05-14 15:21 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2024-05-14 15:54 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API, Dagster Cloud UI

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2024-04-17 13:20 UTC

   In preparation for our Dagster+ launch, we erroneously began redirecting dagster.cloud to dagster.plus. We are rolling back this change now. If you continue to have trouble logging in, you may need to clear your browser's cache.

2. monitoring 2024-04-17 13:53 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2024-04-17 13:53 UTC

   In anticipation of today's Dagster+ launch event, we added a second domain to our app: dagster.plus. Last night, we erroneously started redirecting certain requests from dagster.cloud to dagster.plus. As a result, organizations who explicitly allow-list dagster.cloud in their...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-04-11 17:02 UTC

   Starting earlier this week, a small number of runs in Dagster Cloud started failing with an Internal Server Error exception due to a timeout in our storage engine. We've identified the source of the problem and are in the process of rolling out a fix.

2. identified 2024-04-11 17:04 UTC

   We are continuing to work on a fix for this issue.

3. monitoring 2024-04-11 17:54 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2024-04-11 21:25 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2024-04-01 10:20 UTC

   Increased latency on our storage engine caused a small number of requests to our API to time out at 10:20 UTC and again at 10:47 UTC. This may have resulted in some runs that were running during this time period to fail. The timeouts have now been resolved and we’re in the...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2024-02-26 17:44 UTC

   We experienced an outage while rotating secrets in our production environment. The outage lasted from 17:28 UTC - 17:39 UTC. We are back receiving traffic and are monitoring the recovery

2. resolved 2024-02-26 18:05 UTC

   We have confirmed that the incident is resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API, Dagster Cloud UI

Failure mode: A code release introduced the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2024-02-07 23:43 UTC

   As a result of our maintenance window, Dagster Cloud was down from 23:36 to 23:39UTC. This is now resolved; all systems should be operating normally.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2024-02-07 20:00 UTC

   From 20:01-20:13 UTC, we saw a small number of duplicate run launches related to our maintenance window. This issue is now resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2024-02-01 18:56 UTC

   We experienced a brief outage storing events from 18:41.55 UTC - 18:42.51 UTC. We recovered from the underlying issue but a number of runs may have unexpectedly failed.

2. resolved 2024-02-01 18:56 UTC

   The underlying incident has been resolved. We're still investigating the breadth of the impact and will message with more information shortly.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2023-09-18 18:07 UTC

   We are currently investigating issues with some Serverless runs failing to launch.

2. investigating 2023-09-18 18:39 UTC

   Our underlying cloud provider is having some networking issues which are likely the root cause of the Serverless run launch failures. We are continuing to investigate these errors.

3. investigating 2023-09-18 19:24 UTC

   We are continuing to monitor the networking issues with our underlying cloud provider which are causing Serverless run launch failures.

4. investigating 2023-09-18 20:06 UTC

   We are continuing to experience networking issues with our underlying cloud provider. We have begun implementing a workaround to failover to an availability zone that is not experiencing these issues.

5. investigating 2023-09-18 21:44 UTC

   We are continuing to experience networking issues with our underlying cloud provider. We have started to rollout a workaround to mitigate these networking issues.

6. monitoring 2023-09-18 23:53 UTC

   We've been rolling out changes across Serverless that have been stabilizing network issues. Also, our underlying cloud provider (AWS) has been providing updates that expect full resolution in the next hour.

7. monitoring 2023-09-19 01:40 UTC

   AWS has not been able to solve the network issues that they identified in a number of AZs. We are continuing to monitor their progress on the underlying issue. In the meantime, we have been rolling out a workaround to fall back on to stable AZs. This workaround has been applied...

8. resolved 2023-09-19 04:03 UTC

   We've rolled out our fallback change and the underlying network issue also appears to be resolved.

9. postmortem 2023-09-19 12:30 UTC

   Dagster Serverless places containers across three different availability zones for redundancy. Typically, this hedges us against issues like capacity availability constraints. At 10:00 AM PST, AWS began experiencing elevated latencies for a networking subsystem in two of these...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2023-09-18 12:46 UTC

   Database maintenance by our cloud provider caused a 90 second service disruption between 12:46:10 UTC and 12:47:40 UTC while a primary database failed over to a secondary. Dagster operations during this window would have failed. All systems have resumed normal operation.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2023-09-06 19:30 UTC

   We have observed an increase in timeouts from our storage engine during run execution, resulting in some runs failing with an error like the following in the event log: dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server...

2. identified 2023-09-06 20:08 UTC

   The issue has been identified and we are in the process of testing a fix.

3. monitoring 2023-09-06 22:39 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2023-09-06 23:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2023-08-03 20:29 UTC

   We are currently investigating this issue.

2. monitoring 2023-08-03 21:09 UTC

   We are seeing runs successfully launch as of 20:50 UTC. We are continuing to monitor the situation.

3. resolved 2023-08-03 21:23 UTC

   AWS was experiencing increased API error rates and latencies for their EC2 APIs, which impacted AWS services making use of these APIs. This included AWS Fargate, which is used in Dagster Cloud Serverless. Their incident was resolved as of 21:18 UTC.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-07-07 17:43 UTC

   We're investigating reports of intermittent read timeouts to our agent API.

2. monitoring 2023-07-07 18:02 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2023-07-07 19:03 UTC

   A recent change to how we autoscale our webservers caused a small number of customer read timeouts (on the order of magnitude of a few dozen per hour) when connecting to our API endpoint. We've reverted these changes and the incident is now resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API, Dagster Cloud UI

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-06-14 17:44 UTC

   We are currently investigating an outage affecting dagster.cloud DNS routing

2. investigating 2023-06-14 17:51 UTC

   We are continuing to investigate this issue.

3. resolved 2023-06-14 18:00 UTC

   DNS records have been updated and are propagating. Users unable to access Dagster Cloud should clear their DNS cache.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-05-03 22:58 UTC

   We have reverted a recent change and believe this should resolve the issue. We will provide an update after further investigation.

2. investigating 2023-05-03 23:12 UTC

   The issue is confirmed as resolved, though we are continuing to investigate the issue.

3. resolved 2023-05-03 23:29 UTC

   This incident has been resolved. An operation to upgrade the nodes of our cluster overlapped with our weekly release. Interaction between the two operations caused the Dagster daemons that control run queuing, alerting, schedules, and sensors to be temporarily unavailable.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2023-04-26 17:29 UTC

   Dagster Cloud experienced two brief outages: one from 9:27:16 PDT -> 9:27:46 PDT, and one from 9:51:25 PDT -> 9:51:28 PDT. This may have caused issues with runs that were either starting or completing during these windows.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2023-04-05 19:30 UTC

   A network configuration change applied to Dagster Cloud caused run launches to fail for some serverless users from 7:45 PM to 8:45 PM UTC on 4/5/2023. The change has since been rolled back and run launching behavior has returned to normal.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

1. monitoring 2023-03-15 15:13 UTC

   Users of the Github Actions CICD pipeline for Dagster Cloud may be affected. Follow the GitHub incident here: https://www.githubstatus.com/incidents/ybnn77s3lyf8

2. resolved 2023-03-15 15:49 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2023-01-27 19:14 UTC

   We are investigating a failure at the graphql layer that is causing runs to fail.

2. investigating 2023-01-27 19:14 UTC

   We are continuing to investigate this issue.

3. identified 2023-01-27 19:40 UTC

   An error in the graphql layer is causing sensors and runs using the k8s_job_executor to fail

4. monitoring 2023-01-27 20:32 UTC

   A fix has been implemented and we are monitoring the results.

5. resolved 2023-01-27 20:50 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: Dagster Cloud UI

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-01-04 14:54 UTC

   Components in the Dagster Cloud UI that display usage information for billing purposes are unable to fetch data for display. We are currently investigating the issue.

2. resolved 2023-01-04 16:47 UTC

   The incident has been resolved. The errors were originating from an external incident with one of our service providers.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2022-11-02 19:04 UTC

   The issue has been identified and a fix is being implemented.

2. identified 2022-11-02 19:44 UTC

   We are continuing to work on a fix for this issue.

3. monitoring 2022-11-02 20:29 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2022-11-02 20:34 UTC

   We upgraded our agent request queue backend at 2022-11-02 19:04 UTC. During upgrades, Dagster Cloud's backend services read messages from the old queue and write messages onto the new queue. Once the old queue is empty, Dagster Cloud's services begin reading messages only from...

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2022-10-13 23:00 UTC

   We have identified and rolled back a change which prevented some Serverless agents from launching runs

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Affected: API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2022-10-06 23:33 UTC

   We are currently investigating an interruption in the Dagster Cloud scheduler

2. resolved 2022-10-06 23:44 UTC

   Scheduler has returned to normal operation

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2022-09-09 14:35 UTC

   We are currently investigating an issue where runs to fail to start for some users.

2. identified 2022-09-09 14:48 UTC

   We have identified the likely cause of the issue and are publishing a fix.

3. resolved 2022-09-09 15:25 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Dagster+'s status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2022-08-19 15:39 UTC

   We are currently investigating this issue.

Excerpts are shortened. Read the full incident on Dagster+'s status page.