DataRobot
AI. Source: DataRobot status page.
- Incidents, last 90 days
- 2
- Median to mitigation
- 1h 18m
- Median to resolution
- 1h 47m
- p95 to resolution
- 2h 24m
- Naming an upstream provider
- 0%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A code release introduced the failure | 1 |
| The text does not say enough to tell | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
Codespace Session Validation Failure in Generative AI Playground
Affected: Website, Website, Website, API, API, API, Predictions, Predictions, Prediction, AutoML, AutoML, AutoML, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Apps, AI Apps, AI Apps, MLOps, MLOps, MLOps, Pipeline, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Codespace session in the Generative AI Playground fails due to a validation error. Engineering is working on creating a fix.
-
identified
Engineering has implemented a fix and is currently working on deploying it across the Production clusters.
-
resolved
The fix is now deployed to all the Prod environments and Generative AI Playground is now working as expected.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issues with Vector Database Creation
Affected: Generative AI VDB Builder, Generative AI VDB Builder, Generative AI VDB Builder
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The DataRobot Engineering team observed issues while creating a DataRobot hosted Vector Database. Currently, this is affecting all the DataRobot MTS environments and engineering is investigating the root cause.
-
monitoring
Engineering team has applied the fix and Vector Database creation is operational now. The team is monitoring the situation.
-
resolved
Engineering team observed that the Vector Database creation is operational across all DataRobot MTS environments. The incident is Resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issues with prediction requests against custom model deployments
Affected: Website, API, Predictions, AutoML, AI Catalog and Data Ingest, AI Apps, MLOps, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating issues in MTSaaS within EU region impacting prediction requests against Custom Models deployments.
-
identified
The team root caused the issue causing HTTP 403 responses for Custom Model deployments in EU region. Mitigations were applied and the team is verifying the fix stabilized the service.
-
resolved
The issue is contained and the team has verified that predictions against Custom Models in all MTSaaS environments are healthy.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Batch Jobs queued up in US MTS
Affected: Website, API, Predictions, AutoML, AI Catalog and Data Ingest, AI Apps, MLOps, Pipeline, Notebooks, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Batch jobs that are currently in the queue are completed successfully; however, their completion status is not being updated correctly. As a temporary workaround, we are manually marking these jobs as completed while we work on implementing a solution.
-
identified
The affected jobs have been resolved and the issue is mitigated. Engineering is actively working on a permanent fix and testing is currently underway
-
monitoring
The remediation script has been deployed and Engineering is actively monitoring the situation. Batch jobs may take longer than usual to show as completed until a permanent fix is rolled out in the next production deployment.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
A fix has been deployed to production and the issue is now resolved. All systems are operating normally.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Delay in Feature Drift Statistics Processing
Affected: MLOps
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Feature drift statistics are currently experiencing a processing delay of approximately 1 hour. No data has been lost, and all metrics will reflect accurate values once processing catches up. Our team is actively working to resolve this.
-
resolved
Feature drift statistics processing has been restored to normal. All metrics are now up to date. No data was lost.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Customers Experiencing Errors with New Custom Model Creation.
Affected: API, AutoML, AI Catalog and Data Ingest
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are experiencing a service interruption with Custom Models functionality in US SAAS environment. Predictions to existing deployments are working fine, but users cannot create new custom models. Engineering is investigating the issue and will provide updates as we make further...
-
monitoring
Engineering has applied a fix in the US SAAS environment which resolved the issue. At the time of issue, some users might have experienced issues with Custom Apps, Data upload and custom model creation. The issue is contained.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
Custom Models, Custom Applications & Data upload services are back to Operational state in US SAAS Environment. Issue is Resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Widespread intermittent service issues for new workloads in US Production
Affected: AI Apps, MLOps, Notebooks
Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
monitoring
We are currently experiencing intermittent service issues in US Production, which are primarily affecting the launch of new workloads for Notebooks, Custom models, and Custom Applications. This issue does not impact existing workloads. This disruption is strongly correlated with...
-
identified
We are continuing to experience issues launching new workloads for Custom Models and Custom Applications in US Production. This is connected to an ongoing AWS outage. Our team is exploring multiple mitigation options.
-
identified
We are continuing to experience issues launching new workloads for Custom Models and Custom Applications in US Production. This is connected to an ongoing AWS outage. Our team is exploring multiple mitigation options.
-
monitoring
Engineering resolved the underlying issue with workload scheduling and is monitoring the cluster.
-
resolved
Engineering confirmed the issue is resolved and all services are restored.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Delay in processing actual messages
Affected: MLOps
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
Processing actual messages on JP MTS is delayed due to autoscaling malfunction. Engineering scaled up the deployment to alleviate the issue. Root cause mitigation in progress
-
resolved
Engineering has applied the required infrastructure configuration changes. The service is operating normally and no further user impact is observed. Engineering will continue monitoring cluster health to ensure stability. The incident is now marked as Contained.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Elevated Errors on Managed AI Cloud
Affected: AI Apps, Notebooks
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We're experiencing an elevated level of errors and are currently looking into the issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
monitoring
Engineering has applied changes to mitigate the elevated error rates. Services are now operating normally. We are continuing to monitor the system while investigating the cause of the issue.
-
resolved
Engineering has implemented the required fixes to resolve the elevated error rates. Services are now operating normally, and no further user impact has been observed. The team will continue to monitor the system to ensure stability. The incident is now considered contained.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Degraded Performance on DataRobot MTS due to Quay outage
Affected: Website, Website, Website, API, API, API, Predictions, Predictions, Prediction, AutoML, AutoML, AutoML, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Apps, AI Apps, AI Apps, MLOps, MLOps, MLOps, Pipeline, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform. Engineering is currently monitoring the situation.
-
identified
We are continuing to work on a fix for this issue.
-
resolved
Quay.io functionality has been restored and DataRobot environments are fully stabilized.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Performance Degradation on Managed AI Cloud
Affected: API
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are experiencing performance degradation on Managed AI Cloud.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Intermittent UI disruptions on Managed AI Cloud
Affected: Website, API
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Network issue related to Kubernetes in US cluster
Affected: Predictions, MLOps, Pipeline
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
DataRobot is experiencing network issue related to Kubernetes in US Cluster. This will have impact on model deployment and predictions. Engineering is investigating the root cause.
-
identified
Engineering has identified the root cause of the problem and a mitigation is put in place.
-
monitoring
The mitigation implemented by Engineering has improved the network issue. The team is continuing to monitor the environment to ensure full recovery.
-
resolved
The mitigation implemented by Engineering has resolved the Kubernetes network issue, and the incident is now contained.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Degraded Performance on the DataRobot MTS due to Quay outage
Affected: Website, Website, Website, API, API, API, Predictions, Predictions, Prediction, AutoML, AutoML, AutoML, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Apps, AI Apps, AI Apps, MLOps, MLOps, MLOps, Pipeline, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Notebooks, Generative AI VDB Builder, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
Our engineering team has found the the Quay outage currently happening is causing degraded performance across the DataRobot platform.
-
resolved
This incident is now resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
LLM blueprints deployments cannot be created
Affected: Generative AI LLM Playground
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
LLM blueprints deployments can not be created in JP MTS environment. Engineering is rolling back JP cluster to previous version to mitigate the issue.
-
resolved
Rollback of the JP cluster to the previous version is complete and the problem has been mitigated.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Agent Application Template Impacted After Moderations Library Upgrade.
Affected: AI Apps, AI Apps, AI Apps
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Agent application template is affected with the recent moderations library upgrade, fix is identified and mitigation is in progress.
-
resolved
New version of Agentic application template is released, the issue is resolved
Excerpts are shortened. Read the full incident on DataRobot's status page.
Degraded Performance on the DataRobot US MTS
Affected: API, AI Catalog and Data Ingest
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are observing issues on DataRobot US MTS environment. Users may experience degraded performance using APIs and data ingest services. The engineering team is currently investigating the root cause.
-
resolved
The incident has now been resolved. All services are now operational.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Problem connecting to DataRobot due to client browser caching
Affected: Website, Website, Website
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
monitoring
Some customers have reported issue connecting to DataRobot. Please do a hard refresh of your browser by clearing the cache and this should fix the problem. As always let us know if it continue to have issue connecting to DataRobot after clearing the cache.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Degraded Platform Performance Due to Docker Hub Outage
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
resolved
Between 16:50 UTC and 17:09 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Degraded Platform Performance Due to Docker Hub Outage
Affected: Predictions, Predictions, Prediction, Predictions, MLOps, MLOps, MLOps, MLOps
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
monitoring
Between 09:04 UTC and 09:17 UTC, one of external providers(DockerHub) had an outage that might have caused temporary delays in starting platform workloads. The issue has been resolved and normal operations have resumed. Engineering is continuing to monitor the system.
-
resolved
The engineering team has confirmed recovery across all MTS and STS environments. No further updates are expected.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Some Users Facing Issues In Accessing Notebooks/Codespaces in US/EU/JP MTS.
Affected: Notebooks, Notebooks
Failure mode: Login, tokens, certificates or credentials failed. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Some Users with GenAI licensing are facing issues in accessing Notebooks/Codespaces. Engineering team is checking into it.
-
identified
Engineering team has identified the issue and hotfix is being applied to all MTS envs
-
identified
We are continuing to work on a fix for this issue.
-
identified
Engineering team has verified that the fix has been applied to JP MTS production environment and Non GenAI users are now able to access notebooks in JP cluster. We are now applying the hotfix to US and EU clusters.
-
identified
Engineering team has verified that the fix has been applied to JP MTS production environment and Non GenAI users are now able to access notebooks in JP cluster. We are now applying the hotfix to US and EU clusters. The mitigation for Self Managed and STS deployments is in...
-
identified
Our team has deployed a fix for the issue. All the MTS environments are currently operational.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Some Users Are Unable To Login Into US,EU & JP MTS clusters.
Affected: Website, Website, Website
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Engineering is investigating reports of some users not being able to login in US,EU & JP MTS clusters. Please contact support@datarobot.com if you have any questions.
-
resolved
The issue is fixed. Access to the US, EU, and JP clusters is restored.
Excerpts are shortened. Read the full incident on DataRobot's status page.
The AutoML experiment creation workflow and visibility of certain tabs impacted in MTS environments
Affected: Website, Website, Website, AutoML, AutoML, AutoML
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The AutoML experiment creation workflow and the visibility of certain tabs in both the Classic and NextGen UIs are currently impacted for the customers with GenAI license. Engineering is actively investigating the issue.
-
resolved
The issue has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Notebooks are timing out occasionally on US Production.
Affected: Notebooks
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Notebooks are timing out occasionally on US Production, engineering team is investigating the issue.
-
monitoring
The issue of Notebooks timing out is not reproducible on US Prod. Engineering is monitoring the service and investigating the root cause.
-
resolved
The Engineering team hasn't observed any new occurrences during monitoring. All services remain fully operational.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Data Connectors Hanging on Creation
Affected: AI Catalog and Data Ingest, AI Catalog and Data Ingest, AI Catalog and Data Ingest
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Users in MTS Production are not able to create new Data Connections, the process times out and Data Connection cannot be saved. The issue was reported for Snowflake connection, however other connections may be affected too. The Engineering team is actively investigating the...
-
resolved
The Engineering team has successfully resolved the connection and processing timeouts affecting data operations in MTS Production. Services are now running stable.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Temporary Access Issue for Org Admins to Custom Applications
Affected: AI Apps, AI Apps, AI Apps
Failure mode: Login, tokens, certificates or credentials failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
After the recent permission update, Org Admins temporarily lost access to Custom Applications. A fix is being deployed and should be completed within 6 hours.
-
identified
The hotfix has been applied and access for Org Admins to Custom Applications has been restored.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
AWS services outage affects STS and MTS DataRobot environments
Affected: Website, Website, API, API, Predictions, AI Catalog and Data Ingest, AutoML, Generative AI LLM Playground, AI Catalog and Data Ingest, Generative AI VDB Builder, AI Apps, MLOps, Pipeline, Notebooks, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
investigating
Due to on-going AWS outage in us-east-1 region overall functionality of MTS and STS is affected. Users will experience degraded performance and intermittent connectivity issues. Engineering team is in contact with AWS team to find a resolution.
-
investigating
We are continuing to investigate this issue.
-
investigating
We are continuing to investigate this issue.
-
investigating
We are continuing to investigate this issue.
-
monitoring
Overall functionality of DataRobot is recovering. Engineering team is actively monitoring the services to ensure continued stability.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved. AWS is reporting systems as operational. Engineering will continue to monitor the situation.
Excerpts are shortened. Read the full incident on DataRobot's status page.
UI Issue Affecting Model Visibility and Project Management
Affected: AutoML, AutoML, AutoML
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We’ve identified an issue causing limited access to Projects in Classic UI for customers with GenAI Builder seat license assigned. The Engineering team is working on a fix.
-
identified
The Engineering team has released and validated the fix on JP AI Cloud. Deployment to EU and US AI Clouds is in progress.
-
identified
The Engineering team has released and validated the fix on EU AI Cloud. Deployment to US AI Cloud is in progress.
-
resolved
Engineering team has applied the fix to address UI issue affecting Model Visibility and Project Management. This problem has now been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issue With DockerHub Services Impacting MTS & STS DataRobot Clusters.
Affected: API, API, API, API, Predictions, Predictions, Prediction, AI Apps, AI Apps, AI Apps, Notebooks, Notebooks
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
At 23:09 UTC / 4:09 PM PST, an unexpected outage from DockerHub triggered internal alerts. The engineering team is currently assessing potential impact across environments. We will share further details and mitigation steps as soon as our investigation progresses.
-
monitoring
As of 01:09 UTC (Sept 25)/ 06:09 PM PST, systems have started recovering from the DockerHub outage. DataRobot internal services are back to normal. The engineering team will continue to monitor closely.
-
resolved
Issue is resolved and DataRobot MTS & STS services are back to normal.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Users in APAC region are experiencing network issues when accessing application endpoints in US MTS
Affected: Website
Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We've identified that some users in Japan are experiencing an intermittent login issue when trying to access the US MTS. The engineering team is currently investigating.
-
investigating
Users in APAC are experiencing network issues when accessing application endpoints in US MTS. Engineering is investigating potential ISP-related networking issues in APAC region. Affected customers are encouraged to try using a different network and/or browsers to mitigate the...
-
monitoring
Users can now access US MTS cluster from APAC region and issue recovered. The engineering team will continue monitoring this issue.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Serverless Prediction Servers are impacted when an a new deployment is created or an existing deployment is modified.
Affected: Prediction, MLOps
Failure mode: A configuration change caused the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We've identified an issue impacting serverless deployments when a new deployment is created or an existing deployment is modified, in Japan cluster. Prediction against existing deployments deployed to serveless prediction server that are not modified are not impacted....
-
resolved
Engineering has applied the fix to mitigate the issue impacting serverless deployments when a new deployment is created or an existing deployment is modified. The problem has now been contained.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Some of the customers are not receiving scheduled deployment reports.
Affected: MLOps, MLOps, MLOps
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Some of the customers are not receiving scheduled deployment reports. The data can still be observed in the application UI. The engineering is actively working on restoring the functionality. Please check the below-mentioned link for docs for generating report:...
-
identified
Engineering has identified the root cause and prepared a fix. The team has now started deploying the fix to MTS production.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Interface is down for all customer.
Affected: Website, API, Predictions, AutoML, AI Catalog and Data Ingest, AI Apps, MLOps, Pipeline, Notebooks, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
Starting 11:05 EST today, Interface is down for all customers in the US Prod due to internal error. Engineering has identified the problem and is in the process of applying the fix.
-
monitoring
The fix has been applied successfully and the interface has been restored. Engineering is currently monitoring the system.
-
resolved
The fix has resolved the interface failure issue. DataRobot App in US Prod is fully functioning as normal.
Excerpts are shortened. Read the full incident on DataRobot's status page.
DataRobot Experiencing Stability Issues
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
identified
External cloud providers are experiencing outages which is impacting the DataRobot platform and services.
-
monitoring
Our engineering team will continue to monitor the outages reported by external cloud providers.
-
resolved
Our engineering team confirmed that the outage with our external cloud provider has been resolved. We have been monitoring our functionalities and no specific functionality was affected.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Managed EU AI Cloud Degraded Performance
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
Managed EU AI Cloud had a degraded performance during 7:00 - 7:30 AM UTC. Some users could experience intermittent connection interruptions. Please reach out to support@datarobot.com if you have any questions.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Deployment drift and accuracy charts are broken
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Root cause is identified and team is working on the fix
-
identified
Team is working on implementing the fix
-
resolved
Issue is now resolved and the drift charts are working as expected
Excerpts are shortened. Read the full incident on DataRobot's status page.
Codespaces are not starting
Affected: Website, API
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Codespaces are not starting on EU MTS because of internal connection issues. The Engineering team is working on the fix.
-
resolved
The Engineering team has applied the fix and the issue has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Notebooks sessions getting terminated.
Affected: Notebooks, Notebooks
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Subset of running notebooks sessions getting terminated in all environments (US/EU/JP). Multiple customers are impacted. Engineering team investigating the root cause.
-
monitoring
Engineering has found the potential root cause and applied a fix. Engineering is now monitoring impacted services.
-
resolved
Engineering has successfully applied the fix, and all notebooks are now running as expected across all MTS environments.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Creation of new trial users is occasionally failing
Affected: Website, Website, Website
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
The Engineering team has identified the root cause and is woking on a fix.
-
identified
The Engineering team has applied the fix to Japan MTS and the Trial user creation has been restored in that cluster. The team is now proceeding to apply the fix to US and EU MTS.
-
resolved
The Engineering team has applied the fix to US and EU MTS and the Trial user creation has been restored across all MTS environments.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Users with language settings other than English in profile unable to launch or access AI apps on MTS
Affected: AI Apps, AI Apps, AI Apps
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Users with language settings other than English in profile unable to launch or access AI apps on MTS as the translation files are not loading correctly. As a workaround users can switch to English. We are actively investigating the issue and working on a fix.
-
identified
Engineering has identified the root cause of the issue and is currently preparing a fix.
-
identified
We are continuing to work on a fix for this issue.
-
identified
ETA for the deployment of the hotfix to the Japan (JP) environment is approximately 4 hours.
-
resolved
A fix has been applied to all production environments, no-code applications are operational in all languages.
Excerpts are shortened. Read the full incident on DataRobot's status page.
DataRobot STS degraded performance
Affected: Website, API, AI Catalog and Data Ingest, Generative AI LLM Playground, Generative AI VDB Builder
Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
STS customers might observe degraded performance caused by Regular Maintenance performed by DataRobot Engineering. Engineering team is working on the mitigation.
-
resolved
The Engineering team has identified the issue and is currently applying mitigation steps. If you continue to experience any issues, please contact DataRobot Support.
Excerpts are shortened. Read the full incident on DataRobot's status page.
AWS outage in ap-northeast-1 region
Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: AWS.
-
resolved
AWS had an outage in ap-northeast-1 region that has since been resolved. Outage times: 7.47am GMT to 9:30am GMT Engineering is working to restore any affected customers on STS.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Code Assistant functionality in Notebooks is unstable on US MTS
Affected: Notebooks
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Azure.
-
identified
Code Assistant API requests are currently taking longer than usual for US Production customers. This is a result of a partial outage on Azure. The Engineering team is monitoring the situation and will provide updates when functionality is restored.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issues during VDB creation in Multi-Tenant SaaS environments
Affected: Generative AI VDB Builder, Generative AI VDB Builder, Generative AI VDB Builder
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Engineering has observed that VDB creation began failing across all MTS environments following today's production deployment. The team is currently investigating the root cause and actively working on a fix.
-
resolved
The issue has been successfully mitigated in the MTS production environments. The incident is now contained, and VDB creation functionality has been fully restored.
Excerpts are shortened. Read the full incident on DataRobot's status page.
DataRobot SSO Issues
Affected: Website, Website, Website
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
SSO login has been broken, which may affect users in an attempt to log in. The Engineering team has prepared the fix. The approximate ETA for the hotfix release is 8 hours across all MTS Environments.
-
identified
We are continuing to work on a fix for this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issues while accessing Custom Applications on US Prod
Affected: AI Apps
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Some users on US SaaS are unable to access their Custom Applications. Engineering team is currently investigating, we will keep you updated.
-
resolved
The affected users can access the Custom Applications now. The incident has been mitigated.
Excerpts are shortened. Read the full incident on DataRobot's status page.
The Trial user provisioning is not working.
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The Trial user provisioning is not working on the DataRobot US production environment. The engineering team is currently investigating.
-
resolved
The Trial user provisioning is back on and working as expected in the DataRobot US production environment. The issue is resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Issues with creating Custom Models from LLM Playground
Affected: Generative AI LLM Playground
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
Japan MTS cluster is experiencing issues with creating custom models from LLM Playground. The engineering team is investigating.
-
monitoring
The issue is mitigated and users are able to create custom models again. The engineering team will continue to monitor the environment and prepare a permanent fix until the incident is contained. The estimate is ~ 2 hrs at the moment.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.
Delay updating the Deployment Monitoring Information.
Affected: MLOps
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Our team has identified an issue with our Deployment Monitoring Information. This is a process delay and no data loss is expected. Our team is currently investigating the root cause and is working on a fix. The following services are currently impacted Service Health, Data...
-
monitoring
Our team has identified the root cause and implemented the fix. Service Health and Accuracy no longer have a delay and are operating normally. The delay in Data Drift monitoring is improving, however the Engineering team expects it will take several hours to fully recover as the...
-
monitoring
The unprocessed message backlog continues to catch up. The engineering team is closely monitoring the process. We will provide an update once the processing of delayed messages is caught up.
-
resolved
This incident has been contained.
Excerpts are shortened. Read the full incident on DataRobot's status page.
The custom workload is failing to start
Affected: AI Apps, MLOps
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
The Custom Workload is failing to start in the US MTSaaS environment. Existing running Custom Apps aren't affected. The Engineering team is investigating the root cause.
-
resolved
Engineering has identified the root cause and fixed the issue. The Custom Workloads are starting normally on the US MTSaaS environment, and the incident is resolved.
Excerpts are shortened. Read the full incident on DataRobot's status page.