Veeva Vault Maintenance Release Notification 

Veeva Vault Maintenance Release Notification 

Veeva Vault Maintenance Release Notification 

Veeva Vault Self-Serve Pre-Release Program

• The 23R3 Vault self-serve pre-release functionality has been enabled for all production Vaults. If you don't have a production Vault, pre-release capability has been enabled on your sandbox Vaults.   
• The 23R3 Pre-Release Program will conclude on January 15, 2024. 

• Visit the Vault Help: About the 23R3 Release page for important dates, announcements, and information about the Self-Serve Pre-Release Program.
• The What's New in 23R3 release notes are now available.  

Scheduled Maintenance closed.

Veeva Vault Maintenance Notification 

Veeva Vault Maintenance Release Notification 

Veeva Vault PODs are currently experiencing a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

Veeva Vault PODs continue to experience a service disruption. Veeva engineering teams are working to return the service to normal as quickly as possible.

Beginning at 6:06am PT Vault Technical Operations began receiving alerts regarding Vault authentication functions. At 6:16am PT Veeva Support received its first customer ticket indicating an inability to log in to Vault. The issue affects all PODs in the Vault fleet.

Diagnostic and corrective actions began immediately and continue to this point with highest priority. As of this time there is no firm ETA.

Vault technical teams are restarting affected systems and more PODs are coming online. The root cause has yet to be confirmed. The incident is being treated with highest priority. No ETA for full recovery is available as yet.

We are continuing to experience issues with Vault authentication servers. These issues are preventing some users from accessing their Vaults.

However, PODs are continuing to come online across the Vault fleet. We expect the fleet to have been fully restored shortly though a definitive ETA for full recovery is unavailable.

Investigation into Vault Auth server issues continues. Vault Authentication services were part of a Maintenance Release last Thursday night (Oct 5, 2023). The issue today may or may not be related to that update. Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack. 

We are working to restore service as quickly as possible and this incident remains the highest priority.

All PODs as of 09:10am PT have been restarted and access to Vault function has resumed.

Investigation into the root cause continues. To reiterate, Veeva technical and security teams indicate that this incident does not appear to be any type of cyber attack. We continue to investigate if the recent Maintenance Release resulted in a delayed effect leading to this incident. The summary Trust entry at the conclusion of this incident will provide more detail into the suspected root cause.

Investigation into this incident remains the highest priority. A summary Trust entry will follow shortly disclosing what is known about the root cause.

The incident began at 06:06am PT when Vault Technical Operations received alerts regarding Authentication services. Customers soon reported issues with accessing the Vault login page and the Vault login itself beginning 06:16am PT.

The incident was resolved when, at 09:10am PT, all PODs across the Vault fleet were once more accessible.

Preliminary actions by Vault technical teams focused on addressing the interactions between Authentication services and the underlying database supporting those services. In time it became clear that a restart of Authentication services (and databases) was required to resolve the condition. Such restarts are lengthy procedures and were performed across all PODs and thus accounted for much of the time taken in the incident. As PODs once more became available, increasing numbers of customers were able to resume normal Vault functions until, eventually, all PODs were available.

While investigation into the suspected root cause continues, Vault technical and security teams have, with high confidence, ruled out a DDOS or other security attack as being the cause of the outage.

Focus is now placed on whether the Maintenance Release on Thursday Oct 5 may have led to this incident. A forthcoming Incident Report will detail the findings of the root cause analysis under way now.

Veeva Vault PODs are currently experiencing intermittent degraded performance.  Veeva engineering teams are working to return the service to normal as quickly as possible.

Affected Veeva Vault PODs are now fully available with all services operating normally.

Many VeevaVault PODs had degraded performance where new connections were not working for period of time across many PODs during the past hour. 

All systems UP and running

Veeva Vault Self-Serve Pre-Release Program

• The 23R2 Vault self-serve pre-release functionality has been enabled for all production Vaults. If you don't have a production Vault, pre-release capability has been enabled on your sandbox Vaults.   
• The 23R2 Pre-Release Program will conclude on September 11, 2023. 

• Visit the Vault Help: About the 23R2 Release page for important dates, announcements, and information about the Self-Serve Pre-Release Program.
• The What's New in 23R2 release notes are now available.  

The listed pre-release PODs are scheduled to be upgraded on 8/2 at 9:30 am PT.  
Additional updates will be posted to indicate latest status and completion.

Upgrade on all PODs have started.

For VV1-24, all Vaults are in maintenance and is estimated to be completed ~11:00 am PT.

For the remaining PODs, the upgrade process is a phased approach.  Users will receive a notification (within Vault) when the upgrade is scheduled to start.  The downtime for each Vault is <30 minutes.  

Status of 23R2.0 upgrade for the Pre-Release PODs:

• Complete:  VV1-24, VV2-2130, VV3-3086
• In Progress*:  VV1-40, VV1-42, VV1-1156, VV2-2085

* Note: the upgrade process is a phased approach.  Users will receive a notification (within Vault) when the upgrade is scheduled to start.  The downtime for each Vault is <30 minutes.  

All of the mentioned pre-release PODs have been upgraded to 23R2.0.
If you have any concerns or issues, please contact Support.

Scheduled Maintenance closed.