VV1-1158
Veeva Vault Maintenance Release Notification
Veeva Vault Maintenance Release Notification
Veeva Vault Maintenance Release Notification
Veeva Vault Maintenance Notification
Veeva Vault Maintenance Release Notification
Veeva Vault PODs are currently experiencing a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Beginning at 6:06am PT Vault Technical Operations began receiving alerts regarding Vault authentication functions. At 6:16am PT Veeva Support received its first customer ticket indicating an inability to log in to Vault. The issue affects all PODs in the Vault fleet.
Diagnostic and corrective actions began immediately and continue to this point with highest priority. As of this time there is no firm ETA.
Vault technical teams are restarting affected systems and more PODs are coming online. The root cause has yet to be confirmed. The incident is being treated with highest priority. No ETA for full recovery is available as yet.
We are continuing to experience issues with Vault authentication servers. These issues are preventing some users from accessing their Vaults.
However, PODs are continuing to come online across the Vault fleet. We expect the fleet to have been fully restored shortly though a definitive ETA for full recovery is unavailable.
Investigation into Vault Auth server issues continues. Vault Authentication services were part of a Maintenance Release last Thursday night (Oct 5, 2023). The issue today may or may not be related to that update. Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack.
We are working to restore service as quickly as possible and this incident remains the highest priority.
All PODs as of 09:10am PT have been restarted and access to Vault function has resumed.
Investigation into the root cause continues. To reiterate, Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack. We continue to investigate if the recent Maintenance Release resulted in a delayed effect leading to this incident. The summary Trust entry at the conclusion of this incident will provide more detail into the suspected root cause.
Investigation into this incident remains the highest priority. A summary Trust entry will follow shortly disclosing what is known about the root cause.
The incident began at 06:06am PT when Vault Technical Operations received alerts regarding Authentication services. Customers soon reported issues with accessing the Vault login page and the Vault login itself beginning 06:16am PT.
The incident was resolved when, at 09:10am PT, all PODs across the Vault fleet were once more accessible.
Preliminary actions by Vault technical teams focused on addressing the interactions between Authentication services and the underlying database supporting those services. In time it became clear that a restart of Authentication services (and databases) was required to resolve the condition. Such restarts are lengthy procedures and were performed across all PODs and thus accounted for much of the time taken in the incident. As PODs once more became available, increasing numbers of customers were able to resume normal Vault functions until, eventually, all PODs were available.
While investigation into the suspected root cause continues, Vault technical and security teams have, with high confidence, ruled out a DDOS or other security attack as being the cause of the outage.
Focus is now placed on whether the Maintenance Release on Thursday Oct 5 may have led to this incident. A forthcoming Incident Report will detail the findings of the root cause analysis under way now.
Veeva Vault PODs are currently experiencing intermittent degraded performance. Veeva engineering teams are working to return the service to normal as quickly as possible.
Affected Veeva Vault PODs are now fully available with all services operating normally.
Many VeevaVault PODs had degraded performance where new connections were not working for period of time across many PODs during the past hour.
All systems UP and running
Veeva Vault Engineers are investigating increased error rates across Veeva Vault PODs which may cause intermittent degraded performance. Services impacted include Clinical AI. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault Engineers are continuing to investigate increased error rates across Veeva Vault PODs. Intermittent performance degradations are impacting Align, Clinical AI, and Renditions. Veeva engineering teams are working to return the service to normal as quickly as possible.
Updating incident to a partial outage impacting Align, Clinical AI, and Renditions.
Services are quickly recovering. Veeva Engineers are continuing to closely monitor as we continue toward full recovery. All services are now online.
Affected Veeva Vault PODs are now fully available with all services operating normally.