Vellum
AI. Source: Vellum status page.
- Incidents, last 90 days
- 1
- Median to mitigation
- 3h 48m
- Median to resolution
- 3h 48m
- p95 to resolution
- 3h 48m
- Naming an upstream provider
- 0%
- With a postmortem on the status page
- 0%
What failed, last 90 days
| A provider or third-party service this vendor depends on failed | 1 |
Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.
Incidents per month
All recorded incidents
Platform network instability
Affected: APIs, Vellum Assistant Platform, Web Application, Documents Endpoint, Predict Endpoint
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently monitoring a cloud provider outage causing intermittent networking issues
-
investigating
We are continuing to investigate this issue.
-
investigating
We are continuing to investigate this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Vector database downtime affecting search
Affected: Predict Endpoint
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased latency on predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased latency on predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Users unable to log in to app.vellum.ai
Affected: Web Application
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
Users are unable to log in to the Vellum webapp
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased Workflow Execution Latency
Affected: Predict Endpoint
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating increased latency on workflow executions
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum webapp outage
Affected: Web Application
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating the issue
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum webapp outage
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
The Vellum webapp experienced an outage for 6 minutes as a result of an infrastructure issue
Excerpts are shortened. Read the full incident on Vellum's status page.
API's and Web App down
Affected: APIs, Web Application, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
investigating
We are continuing to investigate this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Monitoring Outage
Affected: APIs
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
investigating
We are continuing to investigate this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum Web UI is Down
Affected: Web Application
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating why app.vellum.ai is hitting a Server Error and is unable to load. We expect all APIs to be functioning as usual.
-
monitoring
The Vellum web app is back up and we're monitoring the situation now.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum Frontend Was Down
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
The Vellum Frontend failed to load for 4 minutes from 5:11pm EST to 5:15pm EST. No backend services or APIs were affected
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum App and API is Down
Affected: APIs, Web Application, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
investigating
We are continuing to investigate this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Downstream Cloudflare Issues Detected
Affected: APIs, Web Application, Documents Endpoint, Predict Endpoint
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Cloudflare.
-
identified
Due to global Cloudflare outage. All other features are operational.
-
identified
We are continuing to work on a fix for this issue.
-
identified
Degraded login flow due to Cloudflare outage.
-
identified
Outage resolved
-
identified
Cloudfare downstream issues were detected on Documents Endpoint and Login Authentication
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Claude model request compilation errors
Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
resolved
For a brief period, some requests to a subset of Claude models failed due to the addition of a new feature which has been reverted
Excerpts are shortened. Read the full incident on Vellum's status page.
Users unable to create workflow sandboxes
Affected: APIs, Web Application, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are investigating this issue.
-
investigating
We are continuing to investigate this issue.
-
investigating
We are continuing to investigate this issue.
-
identified
We have identified the issue and are working on a fix.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Sandbox Saving Degraded
Affected: APIs
Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are investigating an issue with saving both prompts and workflows
-
investigating
The issue was identified and a fix was deployed
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Webapp downtime
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
Users were unable to access app.vellum.ai for a period of approximately 3 minutes starting at 17:15 UTC. The issue was isolated to the webapp, APIs and deployment traffic were unaffected.
Excerpts are shortened. Read the full incident on Vellum's status page.
Request Latencies to Workflows and Prompts are elevated
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating the issues
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased workflow execution errors
Affected: APIs, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating an issue with workflow execution errors
-
identified
We have identified the root cause of this issue and are implementing a fix
-
monitoring
We have implemented the fix and are monitoring workflow executions
-
resolved
We have confirmed this incident is resolved
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased Errors on Workflow Sandbox Executions
Affected: APIs, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
We are investigating an issue with workflow sandbox executions. As of now this is not effecting deployments
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
High latency on predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased error rate on workflows
Affected: APIs, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Elevated error rate on predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
GCP Outage affecting all Vellum Services
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
Names an upstream provider: Google Cloud.
-
investigating
We are currently investigating this issue.
-
monitoring
GCP has started to recover its services. We expect most of our services to be back online but are monitoring stability
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased error rate and latency on API, app failing to load
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
identified
The issue has been identified and a fix is being implemented.
-
investigating
We are currently investigating this issue.
-
investigating
We are aware of reports of application failing to load as we implement infrastructure level changes
-
investigating
App should be back online, but with degraded load times as we migrate traffic over to the new infra
-
investigating
The new fix is rolled out, we are continuing to monitor the situation
-
monitoring
API requests to the predict endpoint is experiencing increased latency as we roll out the remaining fixes
-
monitoring
We are continuing to monitor for any further issues.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
High load times for app functionality
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
identified
The issue has been identified and a fix is being implemented.
-
monitoring
A fix has been implemented and we are monitoring the results.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased latency on Predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
monitoring
We've rolled out a fix to reduce latency to normal levels
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased latency and 4XX responses on predict endpoint
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
investigating
We're continuing to investigate and actively work on resolving this incident.
-
monitoring
A fix has been implemented and we're closely monitoring.
-
monitoring
We have stabilized all requests made to https://predict.vellum.ai
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Workflows Failing w/ "Invalid API Key" Error
Affected: APIs, Predict Endpoint
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
We're seeing widespread issues with Workflows failing to run with an "Invalid API Key" error. The problem has identified and a fix is being worked on now.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Predict endpoint outage
Affected: Predict Endpoint
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
monitoring
Users are experiencing 403 errors on calls to predict.vellum.ai. [7:07pm ET] We expect this issue to be resolved and are monitoring
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Elevated Rate of 403 Responses
Affected: APIs
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We've gotten reports of an elevated rate of 403 responses in Vellum APIs. We're investigating now.
-
monitoring
A fix has been identified, rolled out, and is being closely monitored.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Code Execution nodes experiences elevated error rates
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
The incident has been resolved
Excerpts are shortened. Read the full incident on Vellum's status page.
Predict endpoint outage
Affected: Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Authentiation Errors on API endpoints
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
resolved
From 9:39-9:55AM ET API calls to Vellum using API key authentication raised 403 errors.
Excerpts are shortened. Read the full incident on Vellum's status page.
Web Application Not Loading & Inference Endpoints Hanging
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We are actively investigating reports of the Vellum web application not loading and inference endpoints hanging. We will be posting updates as we have them available here.
-
monitoring
We believe this to be resolved, but are closely monitoring the situation to ensure services remain healthy.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Partial downtime on predict endpoint
Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
resolved
Users experienced elevated error rates on the predict endpoint for 5 minutes (16:17-16:22 UTC) due to a database connectivity issue.
Excerpts are shortened. Read the full incident on Vellum's status page.
Increased Error Rates for Inference Endpoints
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
investigating
We're seeing an increase in error rates for inference endpoints, resulting in unexpected 500 responses. We're actively investigating this issue and will provide updates here.
-
resolved
This issue was resolved a few minutes later. There was an issue in our queueing service.
Excerpts are shortened. Read the full incident on Vellum's status page.
Partial outage on predict and documents endpoints
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
Elevated latencies and error rates on the predict and documents endpoints from 19:05-19:17 UTC
Excerpts are shortened. Read the full incident on Vellum's status page.
Users unable to log in due to Auth0 outage
Affected: APIs
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).
-
investigating
We are currently investigating this issue.
-
investigating
We are continuing to investigate this issue.
-
monitoring
Users are able to log in again
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.
Users unable to log in at https://app.vellum.ai
Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
resolved
From 10:09am EST - 10:23am EST, users were unable to log into the main Vellum platform. Requests to the API endpoints were unaffected.
Excerpts are shortened. Read the full incident on Vellum's status page.
Unable to Perform Prompt Requests
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
There were about 15 minutes beginning at 1:30pm EST in which we failed to make Prompt requests to model providers. The root cause was resolved and executions are behaving as they should once more.
Excerpts are shortened. Read the full incident on Vellum's status page.
Delayed Event Processing
Affected: APIs, Predict Endpoint
Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
identified
Event processing is currently delayed, resulting in recent monitoring data not appearing in monitoring UIs. We have identified the root cause and are rolling out a fix. Event processing should catch up and we'll provide updates here as they do.
-
resolved
The fix is live and all backed up events have been successfully processed.
Excerpts are shortened. Read the full incident on Vellum's status page.
Partial Outage of Vellum Services
Affected: APIs, Documents Endpoint, Predict Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
There were about 3 minutes of downtime of Vellum services beginning around 12:24pm EST December 23rd, 2024.
Excerpts are shortened. Read the full incident on Vellum's status page.
Partial Outage for Workflow Deployments
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
For about 15 minutes beginning 11:16am EST, API called to Workflow Deployments (i.e. `execute-workflow` and `execute-workflow-stream`) began incorrectly responding with 400 status codes. This issue has since been resolved and preventative measures are being put into place for...
Excerpts are shortened. Read the full incident on Vellum's status page.
Documents Outage
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
Document search and indexing were unavailable from 10:36AM-11:42AM CST
Excerpts are shortened. Read the full incident on Vellum's status page.
Vellum-Wide Outage
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
From 2:20-2:38pm PST Vellum had a global outage affecting the web application and all API endpoints.
Excerpts are shortened. Read the full incident on Vellum's status page.
Web Application Partial Outage
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).
-
resolved
The Vellum web application was down for about 2 minutes from 6:18pm - 6:20pm EST.
Excerpts are shortened. Read the full incident on Vellum's status page.
Elevated Connection Errors to OpenAI
Affected: Predict Endpoint
Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
Names an upstream provider: OpenAI.
-
identified
Vellums inference endpoints are experiencing an elevated rate of connection errors when making requests to OpenAI. A fix is going live now.
-
identified
We are continuing to work on a fix for this issue.
-
resolved
This was resolved at 6:31pm EST.
Excerpts are shortened. Read the full incident on Vellum's status page.
Document Processing and Search
Affected: Documents Endpoint
Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).
-
investigating
We are experiencing degraded stability for Document search and upload and for Document Index creation and deletion. We are actively working on a fix and should see lower error rates with time.
-
resolved
This incident has been resolved.
Excerpts are shortened. Read the full incident on Vellum's status page.