VV1-1140
Veeva Vault Maintenance Release Notification
Veeva Vault Maintenance Release Notification
Veeva Vault Release Notification
- Visit the About the 23R3 Release page to review the schedule and other release details.
- Read about What's New in 23R3 release notes for in-depth information about the new features.
- Reference the Latest Announcements for release-specific information.
A 60 minute extension is required for POD VV1-1140.
All other PODs have completed the upgrade and are now accessible.
Additional time added for VV1-1140 only. Engineering is investigating.
Scheduled Maintenance closed.
Veeva Vault Maintenance Release Notification
Veeva Vault Maintenance Notification
Veeva Vault Maintenance Release Notification
Veeva Vault PODs are currently experiencing a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.
Beginning at 6:06am PT Vault Technical Operations began receiving alerts regarding Vault authentication functions. At 6:16am PT Veeva Support received its first customer ticket indicating an inability to log in to Vault. The issue affects all PODs in the Vault fleet.
Diagnostic and corrective actions began immediately and continue to this point with highest priority. As of this time there is no firm ETA.
Vault technical teams are restarting affected systems and more PODs are coming online. The root cause has yet to be confirmed. The incident is being treated with highest priority. No ETA for full recovery is available as yet.
We are continuing to experience issues with Vault authentication servers. These issues are preventing some users from accessing their Vaults.
However, PODs are continuing to come online across the Vault fleet. We expect the fleet to have been fully restored shortly though a definitive ETA for full recovery is unavailable.
Investigation into Vault Auth server issues continues. Vault Authentication services were part of a Maintenance Release last Thursday night (Oct 5, 2023). The issue today may or may not be related to that update. Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack.
We are working to restore service as quickly as possible and this incident remains the highest priority.
All PODs as of 09:10am PT have been restarted and access to Vault function has resumed.
Investigation into the root cause continues. To reiterate, Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack. We continue to investigate if the recent Maintenance Release resulted in a delayed effect leading to this incident. The summary Trust entry at the conclusion of this incident will provide more detail into the suspected root cause.
Investigation into this incident remains the highest priority. A summary Trust entry will follow shortly disclosing what is known about the root cause.
The incident began at 06:06am PT when Vault Technical Operations received alerts regarding Authentication services. Customers soon reported issues with accessing the Vault login page and the Vault login itself beginning 06:16am PT.
The incident was resolved when, at 09:10am PT, all PODs across the Vault fleet were once more accessible.
Preliminary actions by Vault technical teams focused on addressing the interactions between Authentication services and the underlying database supporting those services. In time it became clear that a restart of Authentication services (and databases) was required to resolve the condition. Such restarts are lengthy procedures and were performed across all PODs and thus accounted for much of the time taken in the incident. As PODs once more became available, increasing numbers of customers were able to resume normal Vault functions until, eventually, all PODs were available.
While investigation into the suspected root cause continues, Vault technical and security teams have, with high confidence, ruled out a DDOS or other security attack as being the cause of the outage.
Focus is now placed on whether the Maintenance Release on Thursday Oct 5 may have led to this incident. A forthcoming Incident Report will detail the findings of the root cause analysis under way now.
Veeva Vault PODs are currently experiencing intermittent degraded performance. Veeva engineering teams are working to return the service to normal as quickly as possible.
Affected Veeva Vault PODs are now fully available with all services operating normally.
Veeva Vault Release Notification
- The Self-Serve Pre-Release program is now available for all production Vaults to support any pre-release testing efforts. If you don't have a production Vault, the ability to create a pre-release Vault has been enabled on your sandbox Vault.
- See Vault Help: 23R2 Announcements for more information.
- Additional information is on the Vault Help: About the 23R2 Release page.
- Click here to learn how to find your Vault's POD location.
Scheduled Maintenance closed.
Many VeevaVault PODs had degraded performance where new connections were not working for period of time across many PODs during the past hour.
All systems UP and running