Splunk On-Call

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-10-15 07:46 UTC

   We are currently investigating this issue.

2. monitoring 2026-10-15 09:30 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-10-15 10:36 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Alert Ingestion, Web Client (Portal), Android Client - Mobile App, Alert Ingestion - Inbound email, iOS Client - Mobile App

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-10-08 16:10 UTC

   We are currently investigating this issue, further updates will be shared as they become available.

2. investigating 2026-10-08 16:21 UTC

   We believe that alert ingestion and subsequent incident creation and related paging notifications may be impacted as well. We are continuing to investigate this issue.

3. investigating 2026-10-08 16:25 UTC

   We are continuing to investigate this issue.

4. monitoring 2026-10-08 17:23 UTC

   A fix for the issue has been implemented and we are seeing recovery across all services. The platform is largely operational now, we are continuing to actively monitor services during this recovery

5. resolved 2026-10-08 19:17 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-14 19:53 UTC

   We are currently investigating a loading issue affecting the portal.victorops.com web interface. We will provide further details as soon as they are available. We apologize for any inconvenience! Please reach out to the Splunk On-Call Support team with any questions:...

2. investigating 2026-09-14 19:53 UTC

   We are continuing to investigate this issue.

3. monitoring 2026-09-14 23:00 UTC

   The issue has subsided. We are continuing to monitor results. Please contact our support team if you still notice any problems or have any questions:...

4. resolved 2026-09-18 00:43 UTC

   This incident has been Resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-08-12 06:30 UTC

   Splunk On-Call experienced loading issues affecting the portal.victorops.com web interface and the Android and iOS mobile applications. The disruption lasted approximately one hour and has been resolved. We apologize for any inconvenience this disruption may have caused.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - Custom Paging Policies

Failure mode: A configuration change caused the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-03-23 17:10 UTC

   Spunk On-Call Custom Paging Policies: We've identified an issue that inadvertently impacts Custom Paging Polices. Our Engineering team is working on a fix as soon as possible. In the meantime, please revert to a diversified Primary Paging policy in your User profile so as not to...

2. identified 2026-03-23 22:39 UTC

   We continue to work towards a fix for this issue. Please ensure your Primary Paging Policy is sufficient for receiving critical alerts until this issue is resolved. Helpful Paging Policy documentation: --------------- Splunk On-Call Custom Paging Policies:...

3. resolved 2026-03-24 17:14 UTC

   The Splunk On-Call Custom Paging Policy feature is back to a fully operational status. This incident has been resolved. Thank you for your patience while we resolved this issue. We apologize for any inconvenience! Please reach out to the Splunk On-Call Support team with any...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2025-12-05 09:00 UTC

   Splunk On-Call experienced an isolated network disruption between approximately 08:47 UTC - 09:12 UTC earlier today, December 5th. The functionality has since returned to operational. We apologize for any disruption or adverse impact this caused. Please contact Splunk On-Call...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Alert Ingestion, Web Client (Portal), Android Client - Mobile App, Alert Ingestion - Inbound email, iOS Client - Mobile App

Failure mode: CDN or edge layer failed or served bad content. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2025-11-18 12:31 UTC

   We are currently experiencing an isolated network disruption. The web client was temporarily unavailable, but has retunred to operational. Our Engineering team is working to identify the issue and implement a solution. In the meantime, please know that notification and web...

2. monitoring 2025-11-18 12:33 UTC

   We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts.

3. monitoring 2025-11-18 12:52 UTC

   Our Content Delivery Network continues to experience intermittent 500 errors impacting our Web Client. The Splunk On-Call Mobile App is currently unaffected.

4. monitoring 2025-11-18 13:09 UTC

   We continue to see internal server errors on our CDN. The Splunk On-Call web client is still being intermittently impacted with 500 errors. Again, the Splunk On-Call Mobile App is operational.

5. monitoring 2025-11-18 13:32 UTC

   We continue to see internal server errors on our CDN. The Splunk On-Call web client is still being intermittently impacted with 500 errors. Please Note: The Splunk On-Call Mobile App is operational. Contact Us:...

6. monitoring 2025-11-18 13:42 UTC

   We continue to see internal server errors on our CDN. The Splunk On-Call web client is still being intermittently impacted with 500 errors. **Mobile App functionality is now partially degraded** Contact Us:...

7. monitoring 2025-11-18 14:02 UTC

   We continue to see internal server errors on our CDN. The Splunk On-Call web client is still being intermittently impacted with 500 errors. **Mobile App functionality is now partially degraded** **Web Client and Mobile App Incident creation is now degraded** Contact Us:...

8. monitoring 2025-11-18 14:15 UTC

   Thank you for contacting Splunk Support. We are aware of an on-going issue which is affecting a number of Splunk services. We are actively investigating and working towards remediation. At this time, we are unable to provide and ETA for resolution though we remain committed to...

9. monitoring 2025-11-18 15:19 UTC

   Web Client and Mobile App are currently operational. Our CDN has implemented a fix, but Splunk On-Call remains in a monitoring state. Contact Us:...

10. resolved 2025-11-18 21:09 UTC

    Our services are currently operating normally. We are no longer observing elevated errors or latency across the network. We are moving this incident to a resolved state. Thank you for your patience and apologies for this disruption. Contact Us:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Alert Ingestion - Inbound email

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-11-13 15:35 UTC

   Splunk On-Call Email Alert Ingestion appears to be disrupted. Thanks for your patience while we investigate this issue. Contact Us:...

2. monitoring 2025-11-13 16:00 UTC

   We are now seeing successful delivery of email alerts in our system. We are moving status to monitoring while we investigate the cause for this disruption. Thanks for your patience while we continue to investigate. Updates to follow. Contact Us:...

3. resolved 2025-11-13 16:54 UTC

   We've identified this issue as stemming from a release by our Email Alert provider. Their release has been rolled back and now Email Alert delivery to On-Call is fully operational. Our apologies for this disruption. Contact Us:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal), Android Client - Mobile App, iOS Client - Mobile App

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-11-06 07:01 UTC

   Splunk On-Call is currently experiencing loading issues on the web UI and mobile applications (iOS and Android). We are currently investigating this issue with urgency. Alert ingestion and paging notifications are unaffected. We will update this page as our investigation...

2. monitoring 2025-11-06 07:20 UTC

   We experienced a CPU spike that caused loading issues in both the mobile applications and the web UI. To address the issue, we restarted our servers and, out of an abundance of caution, reverted a recent deployment. We are actively monitoring the systems to confirm that these...

3. resolved 2025-11-06 09:18 UTC

   This incident is now resolved. The Splunk On-Call web client and mobile applications are fully operational. We apologize for the inconvenience. Please reach out to the Splunk On-Call support team with any questions....

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS, Notifications - Phone, Notifications - WhatsApp

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2025-10-20 08:40 UTC

   Please bear with us while we troubleshoot this issue — updates to follow. Where possible, please adjust your personal paging policies to Push notification and email. If you are affected by this reporting feature, please reach out to On-Call Support. Contact Us:...

2. identified 2025-10-20 09:47 UTC

   The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status Please continue to diversify your paging policies to incorporate Push and email. ------------- Current AWS status: Increased Error...

3. identified 2025-10-20 10:42 UTC

   Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications. ----- AWS update: Increased Error Rates and Latencies Oct 20 3:35 AM PDT The underlying DNS issue has been fully...

4. monitoring 2025-10-20 11:01 UTC

   AWS fix has been implemented. We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed. Moving incident status to Monitoring. Where possible, to avoid single points of...

5. monitoring 2025-10-20 16:02 UTC

   We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance. We are working...

6. monitoring 2025-10-20 20:17 UTC

   Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status...

7. monitoring 2025-10-20 21:57 UTC

   We are continuing to monitor for any further issues.

8. resolved 2025-10-21 02:30 UTC

   We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery. Please contact Splunk On-Call support if you encounter any further problems. Contact Us:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Reporting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2025-10-03 15:11 UTC

   We are currently experiencing inaccurate reporting data in the On-Call Review report (OCR) within the web client. This is a known issue, and we're working on a fix as soon as possible. If you are affected by this reporting feature, please reach out to On-Call Support. Contact...

2. identified 2025-10-04 04:16 UTC

   Update: Please bear with us as we continue to work on a fix for this issue. If you are affected and require immediate On-Call Review reporting data, please reach out to Support via a support ticket or through in-product live chat. We'll gladly provide an accurate, manually...

3. identified 2025-10-07 00:25 UTC

   We continue to work on a fix for this issue. If you are affected and require immediate attention, please reach out to Support via a support ticket or through in-product live chat. We've upgraded this incident to Major, as we are currently unable to provide an accurate CSV for...

4. identified 2025-10-07 21:22 UTC

   Update: We continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and senior Support leadership continue to work towards providing you with accurate On-Call Review reporting. Please note that this is a complex problem involving...

5. identified 2025-10-08 22:23 UTC

   Update: Please bear with us as we continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and Senior Support leadership continue to work towards providing you with accurate On-Call Review reporting. ** Please note that this is a...

6. identified 2025-10-09 21:23 UTC

   Update: Please bear with us as we continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and Senior Support leadership continue to work towards providing you with accurate On-Call Review reporting. ** Please note that this is a...

7. identified 2025-10-10 22:12 UTC

   Update: Our Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible. ** Please note that this is a complex problem involving data from different sources. Our teams are working urgently and diligently, with accurate...

8. identified 2025-10-13 22:53 UTC

   Splunk On-Call Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible. ** Please note that this is a complex problem. Our teams are working urgently and diligently towards providing our customers with accurate On-Call-Review...

9. identified 2025-10-14 23:35 UTC

   Splunk On-Call Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible. ** Please note that this is a complex problem. Our teams are working urgently and diligently towards providing our customers with accurate On-Call-Review...

10. monitoring 2025-10-17 20:46 UTC

    We want to update you on the status of the On-Call Review report issue. Our team has implemented several fixes to address the root cause, and as a result, we are reducing the incident severity to "Minor" and updating the status to "Monitoring" while we complete final data...

11. monitoring 2025-10-31 22:21 UTC

    Issues with On-Call Reporting data still appear in the UI. Please know that our engineering team continues to work on these remaining problems. If this issue is impacting your access to On-Call Review reporting data and you need information, please contact our Support team by...

12. resolved 2026-01-09 21:30 UTC

    Issue Summary: An issue was introduced during an early-notification feature rollout, which prevented certain on-call and off-call entries from being saved to our primary database. This resulted in inaccurate reporting, with on-call hours appearing either higher or lower than...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: iOS Client - Mobile App

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-08-13 21:47 UTC

   Please bear with us while we troubleshoot this issue; updates will follow. Note, this issue DOES NOT affect the functionality of previously downloaded iOS apps. Android is unaffected. For questions or concerns regarding the iOS Mobile App availability, please reach out to Splunk...

2. resolved 2025-08-13 23:00 UTC

   This incident has been resolved. The iOS Mobile App for On-Call is once again available in the Apple App Store. Apologies for any inconvenience! Please send any questions regarding this issue to Splunk On-Call Support:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal), Android Client - Mobile App, iOS Client - Mobile App

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-06-04 07:55 UTC

   The Splunk On-Call portal is currently unavailable via web and mobile. We are actively working to resolve this issue as quickly as possible. If you have any immediate questions or concerns, our support team is standing by to respond: 1 (855) 775-8657...

2. identified 2025-06-04 08:27 UTC

   We've identified this issue related to a deploy inadvertently impacting our databases and are working to stabilize. The Web Client is coming back on line now. Thank you for your patience while we coming back on line.

3. monitoring 2025-06-04 08:36 UTC

   Update: A fix has been implemented and we are monitoring the issue. The On-Call timeline is still experiencing loading issues. We are seeing high alert-lag across the platform.

4. monitoring 2025-06-04 08:56 UTC

   We continue to monitor the situation. We are still experiencing high alert lag. The Web Client and Mobile Apps are now accessible. Thanks for your patience while we continue to work this issue to resolution.

5. resolved 2025-06-04 09:45 UTC

   This incident is now resolved. The Splunk On-Call web client and mobile applications are fully operational. We apologize for the inconvenience. Please reach out to the Splunk On-Call support team with any questions....

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Alert Ingestion - Inbound email

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-05-14 17:25 UTC

   Emails sent to the Generic Email Endpoint Integration are not being ingested by On-Call. These emails are not creating incidents or other alerts as expected. All other Integrations and On-Call incident operations are fully operational at this time. We are currently investigating...

2. identified 2025-05-14 17:59 UTC

   We have identified the source of this issue and are actively working to deploy a fix.

3. monitoring 2025-05-14 21:25 UTC

   A fix has been implemented and we are monitoring the results. Stand by for confirmation

4. resolved 2025-05-14 22:25 UTC

   This incident is now resolved. Alert Ingestion - Inbound email is now fully operational. We apologize for this inconvenience. If you continue to experience issues with your inbound email alerts, please reach out to On-Call Support as soon as possible!...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Single Sign On

Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-10-26 09:45 UTC

   Please bear with us while we investigate this issue—impacted components Single Sign On (SSO) Updates to follow. We apologize for this inconvenience. Please reach out to On-Call Support with any questions....

2. identified 2024-10-26 10:23 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2024-10-26 10:52 UTC

   A fix has been implemented and we are monitoring the solution. SSO is back to an operational state.

4. resolved 2024-10-26 11:03 UTC

   This incident has been resolved. Single Sign On (SSO) is fully operational. We apologize for this inconvenience. If you continue to experience issues with your SSO connections, please reach out to On-Call Support as soon as possible!...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: MS Teams - Add On

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-09-02 03:51 UTC

   We are investigating reports of issues with our MS Teams integration, specifically posts from On-Call to MS Teams. Please bear with us as we continue to look into this issue.

2. resolved 2024-09-02 05:44 UTC

   The issue has been identified and resolved; apologies for any inconvenience! Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-05-31 13:25 UTC

   Please bear with us while we investigate this issue—impacted pages: Users, Settings, Teams. Updates to follow. We apologize for this inconvenience. Please reach out to On-Call Support with any questions....

2. monitoring 2024-05-31 14:17 UTC

   A fix has been implemented and we are monitoring the results. We apologize for this inconvenience. If you continue to experience issues with page loading in the web client, don't hesitate to get in touch with On-Call Support:...

3. monitoring 2024-05-31 15:59 UTC

   Our recent fix has resolved much of the issue, but we will remain in a monitoring state for more testing. If you are experiencing loading issues when navigating to the Teams tab, refreshing the page from your browser should alleviate the issue. Again, we apologize for this...

4. resolved 2024-05-31 16:43 UTC

   This incident has been resolved. Page loading in the Web client has returned to normal functionality. All systems are operational. Thank you for your continued patience as we worked this issue to resolution. Please reach out to On-Call Support with any questions....

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal), Android Client - Mobile App, iOS Client - Mobile App, Alert Rules Engine

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-05-14 20:56 UTC

   Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline...

2. investigating 2024-05-14 20:57 UTC

   We are continuing to investigate this issue.

3. investigating 2024-05-15 22:03 UTC

   We continue to investigate this issue. Please note that only a subset of instances are affected by this issue and no disruption to incident creation and notifications out to on-call end-users have been detected. Apologies for any inconvenience. Please reach out to On-Call...

4. investigating 2024-05-16 22:35 UTC

   Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline...

5. monitoring 2024-05-17 19:59 UTC

   The issue has been identified as upstream from the On-Call platform but impacting our CPU utilization. Mitigation is underway and we are now monitoring the results. Our engineering teams continue to work on this issue with urgency. We will provide updates as soon as we learn...

6. resolved 2024-05-21 20:18 UTC

   This incident has been resolved. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal), Android Client - Mobile App, iOS Client - Mobile App

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-05-10 02:54 UTC

   Splunk On-Call is currently investigating an issue resulting in missing timeline events for approximately 8% of our customer base. These events display ingested alerts and incident actions such as paging and acknowledgments. The actual paging functionality is unaffected. As a...

2. monitoring 2024-05-10 04:25 UTC

   A fix has been implemented and we are monitoring the results

3. resolved 2024-05-10 14:58 UTC

   This incident has been resolved. Timeline events are no longer missing, all systems fully operational.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2024-05-04 21:59 UTC

   The issue has been identified. Our provider is working with carrier partners to resolve the issue. In the meantime, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid a single point of failure. Documentation is found...

2. monitoring 2024-05-05 00:59 UTC

   Update: This issue affects shortcode SMS to the domestic US (AMER). Our provider continues to work with carrier partners to resolve this issue. You may track the progress toward the resolution of this issue here: https://status.twilio.com/incidents/p6jvzrw06911

3. monitoring 2024-05-05 18:02 UTC

   The Splunk On-Call notification provider continues to experience SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. Our provider is working with the carrier partner to resolve the issue. Splunk On-Call will...

4. resolved 2024-05-06 13:33 UTC

   Our provider is no longer experiencing SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. This incident has been resolved. All systems are operational. If you're still experiencing issues with SMS please reach...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: MS Teams - Add On

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2024-03-01 18:51 UTC

   We are currently investigating a problem regarding the performance of our MS Team integration. Please bear with us while we troubleshoot this issue — updates to follow.

2. investigating 2024-03-01 18:52 UTC

   We are continuing to investigate this issue.

3. resolved 2024-03-01 19:32 UTC

   The issue has been identified and resolved! Apologies for any inconvenience! Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Reporting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-02-28 08:57 UTC

   We are currently experiencing lag in our report tooling. New Incidents will not show-up in any reports until the incident is resolved.

2. identified 2024-02-28 08:58 UTC

   We have identified the issue and are working towards a resolution.

3. identified 2024-02-28 16:28 UTC

   Apologies for the Reporting inconvenience! Current Status: - Data is up to date as of 6:00 AM (UTC) - No reporting data lost - Data replication is in progress and will be 100% available once complete Please direct any questions to Splunk On-Call Support:...

4. monitoring 2024-02-28 19:24 UTC

   UPDATE: Remediation complete; watching the latency decrease. Thanks again for your patience while we work this issue to resolution.

5. monitoring 2024-02-28 21:08 UTC

   UPDATE: Reporting back to Operational status. We continue to monitor our Reporting data and will be keeping this incident open for the time being. Thanks again for your patience while we work to completely resolve this incident.

6. resolved 2024-02-28 23:46 UTC

   This incident has been resolved. Reporting is fully operational. We apologize again for any inconvenience. Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-01-24 17:30 UTC

   Splunk On-Call is currently experiencing an isolated disruption to the Incident Page for large instances. We are actively working to resolve this issue as quickly as possible. Please use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any...

2. investigating 2024-01-24 17:31 UTC

   We are continuing to investigate this issue.

3. identified 2024-01-24 17:57 UTC

   The issue has been identified and a fix is being implemented. Please stay tuned for further updates.

4. resolved 2024-01-24 18:47 UTC

   This incident has been resolved. All systems are operational. If you're still experiencing an issue with loading large lists of on-call users in the Incident tab under "Team: All Teams" please refresh the page and give it a few seconds. If the problem continues, please reach out...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Single Sign On

Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-12-16 08:02 UTC

   We are currently investigating an issue with iOS mobile SSO log-in. For immediate access to Splunk On-Call please use the Web Client until we can identify the root cause of this issue. Again, this is only affecting iOS mobile. Thanks your your patience while we troubleshoot this...

2. monitoring 2023-12-16 08:45 UTC

   We've identified the issue and have implemented a fix. Moving this incident to a monitoring status. iOS SSO mobile login should now be available. Updates to follow. Apologies again for any inconvenience.

3. monitoring 2023-12-16 09:10 UTC

   We are continuing to monitor for any further issues.

4. monitoring 2023-12-16 09:13 UTC

   We continue to monitor the issue but have moved our iOS SSO Mobile login back to operational status. Thanks again for your patience while we continue to work the root cause of this issue to ground.

5. monitoring 2023-12-16 19:08 UTC

   We are still monitoring the status of this incident. Testing continues. All systems are operational.

6. resolved 2023-12-19 00:19 UTC

   This incident has been resolved. All On-Call systems are operational. Thank you for your patience while we completed our testing. For questions regarding this incident, please contact Splunk On-Call Support:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Knowledge Base

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-09-14 17:53 UTC

   We are currently investigating an issue with our Knowledge Base hosting service. The ability to contact Support is unaffected and fully operational. We will provide updates as soon as they're available.

2. identified 2023-09-14 18:42 UTC

   The issue has been identified by our KB host and they are working as quickly as possible towards resolution. Thanks for your patience while we/they restore functionality.

3. monitoring 2023-09-14 18:52 UTC

   A fix has been implemented by our host and the Knowledge Base (https://help.victorops.com/) is once again available. If you are still experiencing loading issues please clear your cache and refresh the page. Thank you!

4. monitoring 2023-09-14 18:54 UTC

   We are continuing to monitor for any further issues.

5. monitoring 2023-09-14 19:29 UTC

   Host Update - "Site availability is continuing to improve. We are actively monitoring and continue to see a full recovery in progress. Sep 14, 2023 - 12:12 PDT" ... Splunk On-Call will resolve this Statuspage incident when our host gives the all clear. For all intents and...

6. resolved 2023-09-14 21:04 UTC

   Host Update - "The original issue has been successfully resolved." Accordingly, we will be resolving this incident. Please reach out Splunk On-Call Support with any questions or concerns regarding this incident!...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2023-05-18 18:55 UTC

   Apologies for the inconvenience. We are investigating a loading issue on the Incidents Tab affecting the Team Incidents Dashboard. Please refer to the People Pane in the Timeline tab for this information. Updates to follow.

2. resolved 2023-05-18 19:09 UTC

   This incident has been resolved. Apologies for the inconvenience -- Team Incidents Dashboard is back to operational.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-05-10 15:19 UTC

   We are currently investigating Timeline and Mobile App loading issues. Please bear with us as we investigate this issue. Please stay tuned for updates.

2. investigating 2023-05-10 15:43 UTC

   We are continuing to investigate this issue.

3. monitoring 2023-05-10 15:58 UTC

   A fix has been implemented and we are monitoring the results. Timelines are back to operational. Thank you for your patience while we continue to work on this issue. Please stay tuned for further updates.

4. monitoring 2023-05-10 19:15 UTC

   We are continuing to monitor for any further issues.

5. monitoring 2023-05-10 19:20 UTC

   Current Status: We remain in a "monitoring" state for this issue. - Impacted customer Timelines have returned to fully operational - System testing continues We'll provide a more detailed update here as soon as the issue is fully resolved. Apologies again for this inconvenience...

6. resolved 2023-05-10 21:21 UTC

   Testing has been completed and this incident has been resolved. Duration of Impact: (109 min) 7:49 AM to 9:38 AM (MST) - This event was a visual impact on the Timeline in the UI. - No Alerts were dropped during this time - Detailed Alert information in the affected time range...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2023-05-04 01:27 UTC

   Currently experiencing issues with SSO login for all providers. Please bear with us while we work toward a resolution.

2. identified 2023-05-04 01:59 UTC

   We are working with our provider to regain access. Please bear with us while we work this issue to resolution. Our apologies for this inconvenience.

3. monitoring 2023-05-04 02:20 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2023-05-04 02:23 UTC

   This incident has been resolved. All systems are operational. Apologies again for this disruption.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal), MS Teams - Add On

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-01-14 23:04 UTC

   We are currently investigating disconnects and loading issues to the On-Call timeline that may affect the ability to ACK and RESOLVE active incidents. If you are experiencing any issues with incident actions in the Timeline, please reach out to On-Call Support via...

2. identified 2023-01-15 18:35 UTC

   This issue has been identified. MS Teams is impacted. Please bear with us while we implement a fix.

3. identified 2023-01-15 19:02 UTC

   We are continuing to work on a fix for this issue.

4. monitoring 2023-01-15 19:06 UTC

   The MS Teams integration is back to operational status. We continue to monitor Timeline performance and are working towards a resolution. Please reach out to Support via in-product Chat, the Support Portal, and our hotline (1-855) SPLUNK-S or (1-855) 775-8657 if you are...

5. monitoring 2023-01-17 21:29 UTC

   Update: Related to the Timeline issues experienced over the weekend, the MS Teams integration is in temporary maintenance. Please bear with us while we work towards a permanent fix.

6. monitoring 2023-01-17 21:50 UTC

   MS Teams Add-On Maintenance finished. Thanks for your patience while we completed this work

7. resolved 2023-01-17 22:07 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Chat integration - Omni-Channel (In-product Chat)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2022-12-05 20:15 UTC

   Apologies for this inconvenience! Currently, our in-product Live Chat feature is experiencing issues. We are investigating. For immediate Support, please use the Splunk Support Portal (https://login.splunk.com/) or the Call Center:...

2. resolved 2022-12-06 15:30 UTC

   Thanks for your patience while we worked this issue to the ground. Chat has returned to fully operational status.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: API

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2022-09-28 18:14 UTC

   Please be aware: we elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. For more information please visit: https://health.aws.amazon.com/health/status Updates to follow.

2. resolved 2022-09-28 21:14 UTC

   This incident has been resolved and the API service is operating normally. For further information on this incident please visit: For more information please visit: https://health.aws.amazon.com/health/status Apologies for this inconvenience.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2022-09-13 13:11 UTC

   We are actively researching a geographically isolated issue related to Phone Notification delivery challenges to the +86 (China) country code. We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve...

2. resolved 2022-09-13 22:21 UTC

   This incident has been resolved. Apologies for the inconvenience.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2022-08-09 03:46 UTC

   We are actively researching a geographically isolated issue related to Phone Notification delivery challenges into the +86 (China) country code. We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve...

2. investigating 2022-08-12 21:37 UTC

   As we continue to investigate the +86 (China) Phone notification issue with our telecommunications providers, we strongly encourage Splunk On-Call end-users to diversify their Personal Paging Policies to include Push, Email, and SMS as notification methods.

3. investigating 2022-08-19 22:05 UTC

   Thank you for your patience while we continue to work with our telecommunication provider to investigate phone notification service disruption to the (+86) China region. As mentioned, this disruption is limited to Phone notifications to the (+86) region. We strongly encourage...

4. identified 2022-08-22 15:29 UTC

   Working in collaboration with our current telecommunications provider for Phone notification service into +86/China, we're taking steps this week to significantly reduce (or) eliminate "zero-duration" calls from our inbound +86/China call volume. In doing so, we believe we can...

5. monitoring 2022-08-26 15:17 UTC

   Although we have not received formal notification back from our telecommunications provider on this front, we are currently reviewing our logs and seeing successful call delivery into the +86/China Country Code. IMPORTANT NOTE: In response to this situation, On-Call took...

6. resolved 2022-08-29 14:59 UTC

   RESOLVED: Successful phone call routing to carriers in China (country code +86) has resumed. The changes Splunk On-Call has implemented to avoid zero-second duration calls are outlined below and in our updated Knowledge Base article, found here:...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. identified 2021-12-15 16:01 UTC

   The AWS US-WEST-2 Region is currently experiencing Internet connectivity issues and are investigating at the moment. These connectivity issues are impacting our services but we will continue to monitor their status and update ours accordingly.

2. monitoring 2021-12-15 16:12 UTC

   AWS has updated their StatusPage to state that they have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. They have seen some improvement to Internet connectivity in the last few minutes but will...

3. monitoring 2021-12-15 16:27 UTC

   AWS has posted that they have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

4. resolved 2021-12-15 16:45 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Alert Ingestion - Inbound email

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2021-10-01 17:24 UTC

   We’re currently investigating an issue around email ingestion. We will be providing additional updates on our status page. Apologies for this inconvenience! If you have any immediate questions, please reach out to the Splunk On-Call Support team: spoc-support@splunk.com

2. identified 2021-10-01 18:16 UTC

   The issue has been identified and a fix is in progress

3. monitoring 2021-10-01 18:47 UTC

   A fix has been implemented and we are monitoring results.

4. monitoring 2021-10-01 19:05 UTC

   We are continuing to monitor for any further issues.

5. resolved 2021-10-01 20:09 UTC

   We are moving our Splunk On-Call status page to a Resolved state, in relation to this issue. We'll continue to monitor this situation. If you have any immediate questions in reference to your Splunk On-Call instance, please don't hesitate to reach out to our Support team at...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: API, Reporting

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2021-06-24 02:39 UTC

   Alert ingestion, incident creation, and notifications are unaffected and operating normally. The availability of newly occurring data in our reporting user interfaces as well as our reporting API is currently delayed beyond the expected amount. Recent events may not be available...

2. identified 2021-06-24 03:17 UTC

   The issue has been identified and a fix is being implemented. Apologies for the inconvenience and thank you for your patience.

3. monitoring 2021-06-24 05:00 UTC

   A fix has been implemented and we are now monitoring.

4. resolved 2021-06-24 05:04 UTC

   This incident has been resolved. Reporting UI and Reporting API are operational. Thanks for your patience while we worked this issue to resolution.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2021-03-11 17:30 UTC

   We’re currently monitoring an issue that has affected less than 1% of outbound notifications (all message delivery types: Phone, SMS, Push, and Email) during an isolated period of time. A fix is in progress and should be resolved soon. We will be providing additional updates on...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS, Notifications - Google Push, Notifications - Apple Push, Notifications - Phone

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2021-02-26 14:33 UTC

   We are actively investigating a service affecting issue with the message delivery and alerting associated to our platform. We are approaching this issue with the highest level of urgency and the appropriate parties are actively engaged in troubleshooting the issue. Updates to...

2. investigating 2021-02-26 14:33 UTC

   We are continuing to investigate this issue.

3. investigating 2021-02-26 14:46 UTC

   Update: The service affecting issues that we are actively investigating appear to be associated with a service disruption being experienced by one of our communications partners, Twilio: https://status.twilio.com/ We are continuing to pursue the investigation and resolution of...

4. investigating 2021-02-26 14:55 UTC

   Please be advised that message delivery and alerting associated with the Splunk On-Call platform are significantly delayed as a result of this issue. As we continue to investigate and work this issue to ground, we recommend that you directly evaluate any monitoring solutions you...

5. investigating 2021-02-26 15:30 UTC

   We are continuing to investigate this issue.

6. investigating 2021-02-26 15:38 UTC

   As we continue to investigate this issue, we recommend leveraging PUSH Notification in your Splunk On-Call personal paging policy settings. For guidance: https://help.victorops.com/knowledge-base/paging-policy-setup/ The service disruption we are continuing to troubleshoot is...

7. investigating 2021-02-26 15:56 UTC

   We are continuing to investigate this issue.

8. investigating 2021-02-26 15:58 UTC

   SMS message delivery is improving significantly (to) fully functional. Phone messaging is still delayed. We are continuing to investigate the issue. Updates to follow.

9. investigating 2021-02-26 16:34 UTC

   We are continuing to investigate this issue.

10. resolved 2021-02-26 16:38 UTC

    The issue has been resolved. All message delivery types (Phone, SMS, Push, and Email) and associated notifications are fully functional. We will be providing additional updates on our status page upon the completion of our internal review. If you have any immediate questions,...

11. postmortem 2021-02-26 22:53 UTC

    **Basic Timeline & Incident Overview**: Starting at approximately 6:18 AM \(Mountain\) on the morning of 02.26.21, the _Splunk On-Call_ \(SpOC\) platform began experiencing delays in the outbound delivery of notifications. These delays were directly related to a critical service...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Chat integration - Slack

Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2021-01-15 17:50 UTC

   Currently experiencing an issue with Slack integration re-authorization on the integrations page. Please do not attempt to reauthorize at this moment. We are investigating this issue and will provide updates as soon as possible.

2. identified 2021-01-15 18:17 UTC

   This Slack re-authentication issue has been identified. Please continue to NOT REAUTHENTICATE on the integrations page. If you are experiencing an issue with your Slack integration, please reach out to victorops-support@splunk.com for immediate help. Thanks for your patience...

3. investigating 2021-01-15 19:57 UTC

   We are continuing to investigate this issue.

4. identified 2021-01-15 20:01 UTC

   We continue to work on this identified issue and hope to have a fix implemented soon. Please reach out to victorops-support@splunk.com with any questions regarding your Slack integration. Thanks again for your patience while we work towards a resolution.

5. resolved 2021-01-16 00:11 UTC

   Thanks for your patience! A fix has been implemented for Slack. If you continue to experience any problems with your integration please do not hesitate to reach out us: victorops-support@splunk.com.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Billing

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2020-10-23 02:25 UTC

   We are monitoring slowdowns on the Splunk On-Call billing page. The page might be slow to load for some customers. Please stay tuned for updates.

2. monitoring 2020-10-23 03:26 UTC

   We are continuing to monitor for any further issues.

3. resolved 2020-10-23 03:26 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2020-08-30 13:37 UTC

   We are monitoring a performance issue associated with a third-party partner that may be creating intermittent performance challenges on the VictorOps platform. For more information, please reference the following status page: https://www.cloudflarestatus.com/ If you have any...

2. resolved 2020-08-30 14:58 UTC

   We are moving our VictorOps status page to a Resolved status in relation to this issue. We'll continue to monitor this situation via the following status page updates: https://www.cloudflarestatus.com/ If you have any immediate questions in reference to your VictorOps instance,...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2020-07-17 21:35 UTC

   We are experiencing upstream internet connectivity challenges impacting customers’ ability to log into VictorOps. Please stay tuned for further information on our status.victorops.com and Twitter.

2. identified 2020-07-17 21:39 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2020-07-17 21:40 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2020-07-17 23:03 UTC

   This incident has been resolved. For more detailed information on this incident, please reference the July 17th incident details listed on the following third-party status page: https://www.cloudflarestatus.com/ If you have any immediate questions in relation to your VictorOps...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS, Alert Ingestion, Notifications - Google Push, Alert Ingestion - Inbound email, Notifications - Apple Push, Notifications - Phone, Notifications - Email

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2020-04-19 23:22 UTC

   We are actively investigating a service disruption and will be posting updates as efficiently as possible. If you have any immediate questions, please contact the VictorOps Support Team: victorops-support@splunk.com

2. monitoring 2020-04-20 00:13 UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring 2020-04-20 00:58 UTC

   We are continuing to monitor for any further issues.

4. monitoring 2020-04-20 01:00 UTC

   We are continuing to monitor for any further issues.

5. resolved 2020-04-20 01:17 UTC

   After monitoring our remediation efforts surrounding this incident, we are moving this incident to a resolved status. Brief Incident Overview: From approximately 4:30 PM (Mountain) to 5:35 PM (Mountain) on Sunday (04.19.20), we experienced service disruption challenges that may...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Chat integration - Slack

Failure mode: A configuration change caused the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2020-04-15 15:08 UTC

   We are currently investigating an issue with our Slack integration. New customers attempting to configure their Slack integration may be encountering an error message during the initial set-up process. In addition, existing customers may encounter an error message if they revoke...

2. resolved 2020-04-16 19:26 UTC

   We have updated our Slack integration and confirmed the resolution of this issue. Our apologies for any unintended inconvenience this issue may have caused. If you have any questions or need any assistance, please don't hesitate to reach out to the VictorOps + Splunk Support...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2020-03-13 03:11 UTC

   We have noted intermittent Internet performance issues this afternoon and evening along the eastern seaboard of the United States. These sporadic and isolated issues have temporarily affected the ingestion and delivery performance of email and phone alerting and notifications....

2. resolved 2020-03-14 18:03 UTC

   We received notification that a planned maintenance event surrounding two third-party entities (CloudFlare and AWS) on Thursday, March 12th created some unforeseen, intermittent, network-level performance challenges that adversely affected VictorOps and other affiliated...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2020-03-04 17:50 UTC

   We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will...

2. resolved 2020-03-04 17:52 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS, Notifications - Google Push, Notifications - Apple Push, Notifications - Phone

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2020-02-26 18:36 UTC

   We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will...

2. identified 2020-02-26 18:45 UTC

   We've identified the issue and are working towards a resolution. We can confirm that off-call push notifications and test phone call notifications are degraded. Thanks for your patience while we sort this out and please stay tuned for further updates.

3. identified 2020-02-26 19:35 UTC

   We are continuing to work on a fix for this issue.

4. monitoring 2020-02-26 19:37 UTC

   A fix is being implemented and we will continue to update you as we monitor the results. To clarify, both Test and On and Off-call notifications are affected.

5. resolved 2020-02-26 20:23 UTC

   Apologies for any inconvenience. This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Web Client (Portal)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2019-12-18 18:11 UTC

   We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will...

2. monitoring 2019-12-18 18:31 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2019-12-18 18:56 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Notifications - SMS, Web Client (Portal), Notifications - Google Push, Android Client - Mobile App, iOS Client - Mobile App, Notifications - Apple Push

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2019-10-30 17:14 UTC

   VictorOps is currently experiencing an isolated disruption and we are actively working to resolve this issue as quickly as possible. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team...

2. monitoring 2019-10-30 17:16 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2019-10-30 17:36 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.

Affected: Webhooks

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2019-09-13 01:20 UTC

   We are actively investigating challenges with our JIRA Webhook Integration. We will continue to update our status page as additional information arrives. Apologies for any unintended inconvenience. Please contact the VictorOps + Splunk Support Team if you have any immediate...

2. identified 2019-09-13 03:13 UTC

   We have identified the issue (JIRA configuration change) and are actively working on a resolution. Please contact the VictorOps + Splunk Support Team if you have any immediate questions: victorops-support@splunk.com Additional updates to follow.

3. monitoring 2019-09-13 04:48 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2019-09-13 04:58 UTC

   Due to modifications Atlassian made to comply with GDPR, changes were needed to be implemented to accpet these modifications. We've been actively monitoring the implemented fix and the overall issue. Based on positive technical observations, we're moving this incident from...

Excerpts are shortened. Read the full incident on Splunk On-Call's status page.