|**Metadata**|**Description**  |
|--|--|
|Doc Title|  MVM v3: Update Regional Encryption Certificate|
|Navigation|[WIKI Home Page](https://dev.azure.com/mvmprodeus2/MVM/_wiki/wikis/documentation/1/documents-home#)|
|Tracking| Document Number: VPE-5512-035|
|Author| Graeme Thomson (gt163y) |
| Agreement Number | 24252.S.005 |

***
**Notices**

Copyright © 2025 Metaswitch Networks.  All rights reserved.

This manual is Confidential Information of Metaswitch Networks subject to the confidentiality terms
of the Agreement 01019223 as amended between AT&T and Metaswitch Networks.

It is issued on the understanding that no part of the product code or documentation (including this manual)
will be copied or distributed without prior agreement in writing from Metaswitch Networks and Alianza, Inc.

Metaswitch Networks and Alianza reserve the right to, without notice, modify or revise all or part of
this document and/or change product features or specifications and shall not be responsible for any
loss, cost, or damage, including consequential damage, caused by reliance on these materials.

Metaswitch and the Metaswitch logo are trademarks of Metaswitch Networks. Other brands and
products referenced herein are the trademarks or registered trademarks of their respective holders.

Product(s) and features documented in this manual handle various forms of data relating to your
users. You must comply with all laws and regulations applicable to your deployment, management,
and use of said product(s), and you should take all appropriate technical and organizational
measures to ensure you are handling this data appropriately according to any local legal and
regulatory obligations.

You are responsible for determining whether said product(s) or feature(s) is/are appropriate for
storage and processing of information subject to any specific law or regulation and for using said
product(s) or feature(s) in a manner consistent with your own legal and regulatory obligations. You
are also responsible for responding to any request from a third party regarding your use of said
product(s), such as a request to take down content under the U.S. Digital Millennium Copyright Act
or other applicable laws.


Metaswitch Networks
399 Main Street
Los Altos
CA 94022
<http://www.metaswitch.com>


***
***Table of Contents***
[[_TOC_]]

# 1. Document History

| **Issue** | **Issue Date** | **Author(s)** | **Identification** **of** **Changes** |
|-|-|-|-|
| 1| 06/10/2024| Gthomson|  initial draft |
| 2| 09/30/2024| Gthomson|  updates based on Ops feedback |

# 2. Versions

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  updates based on Ops feedback |

# 3. Integrated Solution Approach v1 (ISA v1)

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  updates based on Ops feedback |

# 4. MOP Impact Scope / General Information

## 4.1 Description

For encryption in transit to authenticate flows between regions, each individual
region must have a certificate which trusts the overall deployment root certificate.
The deployment root certificate was created as part of the initial R11.1 Deployment.

This MOP describes the process to update the regional certificate and associated private key
and apply the updated collateral to the SIs in the region.

## 4.2 Site Specific Description

| **Originator** | **Date** | **Time** |
|-|-|-|
| **Deployment Location(s)** | |
| **Description** | This MOP applies to the MVM V3 on Azure deployment, Release R11.5.4 | |

## 4.3 Service Impact

Service impact is not expected during this procedure.

## 4.4 Coordination

This MOP has no interactions outside of the MVM subscription.

# 5. Prerequisite/Dependencies/Entrance Criteria of MOP

This is a standalone MOP that describes the process to update the regional
encryption CA Certificate and the individual SI encryption certificates.

## 5.1 Required parameters

The following parameter values are required to run this MOP

| **Identifier** | **Description** |
|-|-|
| **CHANGE_ID** | Change ID, used as the prefix for any git branch created in the MOPs. |
| **CONFIG_VERSION** | The name of the config set directory containing the current SI configuration. Config sets are located in the `config` directory |
| **DEPLOYMENT_KEY_VAULT_NAME** | Name of the Azure Key Vault in which the root certificates for encryption in transit and the key for accessing Terraform backend storage are stored. |
| **DEPLOYMENT_KEY_VAULT_RG** | Name of the resource group that contains the key vault **DEPLOYMENT_KEY_VAULT_NAME** |
| **GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository. |
| **GIT_CONFIGURATION_URL** | URL of the configuration git repository. |
| **GIT_PASSWORD** | Password used to access the Azure DevOps repositories if you are using https to manage the local copy of the access the repository. |
| **KEY_VAULT_NAME** | Name of the Key Vault where secrets and certificates are stored for a single region. |
| **KEY_VAULT_RG** | Name of the resource group that contains the key vault **KEY_VAULT_NAME** |
| **MVM_FILESHARE** | Name of the fileshare containing the MVM release. |
| **ORGANIZATION_NAME** | Name of the Azure DevOps organization. |
| **PIPELINE_CONFIGURATION_NAME** | Name of the pipeline configuration file for the region without the .yml suffix (e.g. `vars_eus2_11500`). |
| **PROJECT** | Name of the Azure DevOps project. |
| **REGIONAL_PIPELINE_KEY_VAULT_NAME** | Name of the regional pipeline key vault |
| **REGIONAL_PIPELINE_KEY_VAULT_RG** | Name of the resource group that contains the key vault **REGIONAL_PIPELINE_KEY_VAULT_NAME** |
| **REGION_SHORTNAME** | The short (4-characters maximum) DNS label for the region |
| **SIMPL_SERVICE_NAME** | An identifier for the SIMPL instance |
| **SUBSCRIPTION_ID** | Azure subscription identifier.  |

## 5.2 Required files

No additional files are required to run this MOP.

# 6. Assumptions

The target audience for this procedure is the AT&T Engineer who will be performing the task. They will need to be familiar with Azure and have a working knowledge of the Azure CLI and Linux.

# 7. Material Requirements

## 7.1 Required Documents

## 7.2 Tools

| **Tool** | **Description** | **Quantity** |
|-|-|-|
| Laptop or Desktop PC | PC With at least 1G Memory and a network communications software application such as Procomm, Reflections or PuTTY | 1 |
| Azure connectivity PC | CloudShell Connectivity is required to the azure subscription. This can be accessed via [My Dashboard - Microsoft Azure](https://portal.azure.com/#cloudshell/) | |

# 8. Pre Maintenance Check, Precautions and Preparations

## 8.1 Precautions and Preparation

## 8.2 Precautions

> This procedure may cause a partial outage during implementation. Use executable script files to minimize down time and typing errors. Familiarize yourself with back-out procedures prior to starting the procedure.

| **Ask Yourself Principle** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Do I have the proper ID and appropriate building access permissions for the environment I am about to enter? | | |
| 2. Do I know why I'm doing this work? | | |
| 3. Have I identified and notified everybody - customers and internal groups - who will be directly affected by this work? | | |
| 4. Can I prevent or control service interruption? | | |
| 5. Is this the right time to do this work? | | |
| 6. Am I trained and qualified to do this work? | | |
| 7. Are the work orders, MOPs, and supporting documentation current and error-free? | | |
| 8. Do I have everything I need to quickly and safely restore service if something goes wrong? | | |
| 9. Have I walked through the procedure? | | |
| 10. Have I made sure the procedure includes proper closure including obtaining clearance and release for the appropriate work center? | | |


| **E911 Ask Yourself** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Does this work impact E911? | | |
| 2. Do I know how this work could impact 911/e911? | | |
| 3. Do I know what 911/e911 phase is required? | | |
| 4. Have I identified potential risks to 911/e911 and taken all measures to minimize? | | |
| 5. Does this work affect 15+ sites? | | |
| 6. Can I prevent or control service Interruptions to 911/e911? | | |
| 7. Is this the right time to do the work? | | |
| 8. Is the individual performing the work trained and qualified to do this work? | | |
| 9. Are MOPs and supporting documents current and error free? | | |
| 10. Does the MOP include a 911/e911 test plan? | | |
     

## 8.3 Pre-Maintenance Check Tools/System

Tier 2 needs to identify which tools they will use. This doesn't necessarily need to be included in the MOP but OPS needs to know which tools they will run.

(NEED TO USE STANDARD TOOLS) TIER 2


## 8.4 Pre-Maintenance Check Manual (Non-Automated Requirements)

These will be identify by the tier 3 MOP developer were required.

(MANDATORY CHECK REQUIRE FOR THE MOP) TIER 3


## 8.5 MOP Certification Environment

Examples:  PSL certified.  OR This MOP was paper certified by ATS engineers.

## 8.6 ATS Bulletin

**ATS Bulletin Check**
| **Step** | **Action** | **Results/Description** | **Timeline** |
|-|-|-|-|
| 1. | No Applicable bulletins | | |


## 8.7 Emergency Contacts

The following emergency contact numbers are to be used in the event provisioning support is required.

In the event a service interruption is encountered the AT&T Implementation Engineer will:
- Cease all work immediately.
- Notify the AT&T Voicemail TRC.
- Escalate to the next level of support.


| **Organization** | **Contact Name** | **Contact Number** |
|-|-|-|
| Voicemail TRC | SANRC | 877-662-7674, opt 3 |

# 9. Implementation

## 9.1 Preliminary Implementation
Pre-check tasks are completed the night of the cutover at least one hour prior to cutover activities.

1. Connect to the DevOps Portal
   1. Start a browser session to <https://dev.azure.com/>. This will be required to manage the pipelines
   1. Select the project associated with MVM v3
1. Connect to the Azure Portal
   1. Start a browser session to <https://portal.azure.com/>. This will be required to manage Azure resources
      and access the log analytics workspace (LAW)
   1. If prompted, complete the log in process
1. Connect to Azure Cloud Shell
   1. Start a CloudShell session by connecting a browser to <https://shell.azure.com/>
   1. If the menu at the top left indicates PowerShell select Bash from the menu and confirm at the prompt

      ![screenshot](images/powershell.jpg)
1. Upload any files and directories outlined in section 5.2 to your Cloud Shell account as they will be needed later


## 9.2 Implementation
The process to update the regional encryption Certificate Authority (CA) can be split into the following subtasks:

1. Generate the new CA certificate for the region
1. Restart SIMPL to pick up the new certificate
1. Restart the Application SIs in the region.

Each step is discussed in more detail below.

Roll-out of the new certificate can be monitored through the **Region Certificate Details** panel of
the **Encryption of Data in Transit** dashboard.
![screenshot](images/encryptiondashboard.jpg)

Having opened the dashboard select the region that is being upgraded and expand the
Certificates tab. The Region Certificate Details panel shows the sha256 key associated
with the SI.

Initially, this will be the key associated with the existing CA certificate.

As the rollout progresses this will change to the key associated with the
new CA certificate (CA_FINGERPRINT, extracted in section 9.2.1 below).

When the rollout is complete the key associated with each SI will be the new key.

### 9.2.1 Generate a new Regional CA Certificate

#### 9.2.1.1 Set the default subscription

1. Set the default subscription by running the command:

   ```
   az account set --subscription "{{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}"

   ```

#### 9.2.1.2 Create the SSL configuration file

1. Verify the OpenSSL version on your system is correct by running the following command:
   ```
   openssl version

   ```

   On Ubuntu 20.XX, the valid OpenSSL version is 1.1.1, or a 1.1.1"letter" release,  e.g.:
   ```
   > openssl version
   OpenSSL 1.1.1f  31 Mar 2020
   ```

   On RHEL 9, the valid OpenSSL version is 3.X.X, e.g.:
   ```
   > openssl version
   OpenSSL 3.2.2 4 Jun 2024 (Library: OpenSSL 3.2.2 4 Jun 2024)
   ```

1. Create the SSL configuration file

   Navigate to a directory in which you want to create the certificates: 
   ```
   mkdir ssl_encryption
   cd ssl_encryption
   ```

   Then create a new SSL config file `mvm_ssl.cfg` as follows:
   ```
   echo "[ req ]
   default_bits       = 4096
   encrypt_key        = no
   default_md         = sha256
   utf8               = yes
   distinguished_name = distinguished_name
   req_extensions     = v3_ca
   [ distinguished_name ]
   commonName         = ATT
   [ v3_ca ]
   subjectKeyIdentifier = hash
   basicConstraints     = critical, CA:true
   keyUsage = critical, digitalSignature, cRLSign, keyCertSign" > mvm_ssl.cfg
   ```

#### 9.2.1.3 Download the root certificate

1. Set the following environment variables:

   ```
   DEPLOYMENT_KEY_VAULT_NAME={{ DEPLOYMENT_KEY_VAULT_NAME | default('<DEPLOYMENT_KEY_VAULT_NAME>') }}
   DEPLOYMENT_KEY_VAULT_RG={{ DEPLOYMENT_KEY_VAULT_RG | default('<DEPLOYMENT_KEY_VAULT_RG>') }}
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')

   ```

1. Allow your current session to access the deployment key vault by running the following command:
   ```
   az keyvault network-rule add \
      --name ${DEPLOYMENT_KEY_VAULT_NAME} \
      --resource-group ${DEPLOYMENT_KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```
1. Download the root certificate and private key by running the commands
   ```
   az keyvault secret download \
       --name deployment-root-ca-certificate-cert \
       --vault-name ${DEPLOYMENT_KEY_VAULT_NAME} \
       --file deployment-root-ca-certificate_cert.pem

   az keyvault secret download \
     --name deployment-root-ca-certificate-key \
     --vault-name ${DEPLOYMENT_KEY_VAULT_NAME} \
     --file deployment-root-ca-certificate_key.pem

   ```

1. Remove the current session from the deployment key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${DEPLOYMENT_KEY_VAULT_NAME} \
      --resource-group ${DEPLOYMENT_KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```

#### 9.2.1.4 Create the regional encryption private key and certificate


1. Set the following environment variables:

   ```
   REGION_SHORTNAME={{ REGION_SHORTNAME | default('<REGION_SHORTNAME>') }}
   KEY_VAULT_NAME={{ KEY_VAULT_NAME | default('<KEY_VAULT_NAME>') }}
   KEY_VAULT_RG={{ KEY_VAULT_RG | default('<KEY_VAULT_RG>') }}
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')

   ```

1. Generate the regional CA certificate by running the following commands:

   Generate a signing request:
   ```
   openssl req -new \
   -config mvm_ssl.cfg \
   -newkey ec \
   -pkeyopt ec_paramgen_curve:prime256v1 \
   -keyout int-region-certificate_${REGION_SHORTNAME}_key.pem \
   -out int-region-certificate_${REGION_SHORTNAME}_csr.pem

   ```

   When prompted for a Distinguished Name, enter `ATT` after the text `ATT []:`

   Sign the request:
   ```
   openssl x509 \
   -extfile <(cat mvm_ssl.cfg \
   <(echo "authorityKeyIdentifier = keyid:always,issuer")) \
   -extensions v3_ca \
   -req -in "int-region-certificate_${REGION_SHORTNAME}_csr.pem" \
   -CA deployment-root-ca-certificate_cert.pem \
   -CAkey deployment-root-ca-certificate_key.pem \
   -out int-region-certificate_${REGION_SHORTNAME}_cert.pem  \
   -days 365 \
   -set_serial 0x"$(openssl rand -hex 19)"

   ```

   Concatenate the signed intermediate certificate, the deployment root CA certificate,
   and the new intermediate key together into the new region internal CA certificate:
   ```
   cat int-region-certificate_${REGION_SHORTNAME}_cert.pem \
    deployment-root-ca-certificate_cert.pem \
   > int-region-certificate_${REGION_SHORTNAME}_chain.pem

   cat int-region-certificate_${REGION_SHORTNAME}_chain.pem \
    int-region-certificate_${REGION_SHORTNAME}_key.pem \
    > region-internal-ca-certificate_${REGION_SHORTNAME}.pem

   ```

   Verify the final region internal CA certificate
   ```
   openssl verify -CAfile deployment-root-ca-certificate_cert.pem \
       region-internal-ca-certificate_${REGION_SHORTNAME}.pem

   ```

   If the certificate is valid you should see
   `region-internal-ca-certificate_{{ REGION_SHORTNAME | default('<REGION_SHORTNAME>') }}.pem: OK`
   printed to the terminal

1. Extract the sha256 fingerprint value of the new certificate. This will be
   needed to track roll-out to the region's SIs in a later step.

   Run the following command to calculate this value:
   ```
   export CA_FINGERPRINT=$(openssl x509 -fingerprint -sha256 \
   -in int-region-certificate_${REGION_SHORTNAME}_cert.pem | grep -i SHA256)

   echo $CA_FINGERPRINT

   ```

   Record the value of CA_FINGERPRINT. This will be required to validate that the
   new certificate is being used when the SIs are recreated later in the MOP

1. Allow your current session to access the regional key vault by running the following command:
   ```
   az keyvault network-rule add \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```

1. Upload the CA certificate by running the command:
   ```
   az keyvault certificate import \
    --vault-name ${KEY_VAULT_NAME} \
    --name region-internal-ca-certificate \
    --file region-internal-ca-certificate_${REGION_SHORTNAME}.pem

   ```

1. Verify that the regional CA certificate has been uploaded successfully by listing the secrets in Azure Key Vault:
   ```
   az keyvault certificate list --vault-name ${KEY_VAULT_NAME}

   ```

   Verify the output includes `region-internal-ca-certificate` with a timestamp which matches the previous step

1. Remove the local copies of the certificates and key as they are no longer required
   ```
   rm *.pem

   ```

1. Remove the current session from the deployment key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${DEPLOYMENT_KEY_VAULT_NAME} \
      --resource-group ${DEPLOYMENT_KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```

### 9.2.2 Apply the new CA Certificate to SIMPL

The CA certificate is applied to SIMPL by deleting the SIMPL cluster and then recreating it

#### 9.2.2.1 Set the default subscription
1. Set the default subscription by running the command:

   ```
   az account set --subscription "${SUBSCRIPTION_ID}"
   ```

#### 9.2.2.2 List the Application SIs SIMPL is managing
This allows you to verify that the restarted SIMPL is managing the same set of application SIs

1. Set the following environment variables:

   ```
   REGIONAL_PIPELINE_KEY_VAULT_NAME={{ REGIONAL_PIPELINE_KEY_VAULT_NAME | default('<REGIONAL_PIPELINE_KEY_VAULT_NAME>') }}
   REGIONAL_PIPELINE_KEY_VAULT_RG={{ REGIONAL_PIPELINE_KEY_VAULT_RG | default('<REGIONAL_PIPELINE_KEY_VAULT_RG>') }}
   SIMPL_SERVICE_NAME={{ SIMPL_SERVICE_NAME | default('<SIMPL_SERVICE_NAME>') }}
   SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')
   ```

1. Allow your current session to access the key vault by running the following commands:
   ```
   az keyvault network-rule add \
      --name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Extract the SIMPL Resource Group by running the following command:
   ```
   SIMPL_SERVICE_RG=$(az keyvault secret show \
      --name simpl-service-rg \
      --vault-name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --query 'value' -o tsv)

   echo $SIMPL_SERVICE_RG
   ```

1. Remove the current session from the key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. List the Application SIs being managed by this SIMPL:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get si"

   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get dc"
   ```

   Save the results of these commands, for comparison after SIMPL is restarted.

#### 9.2.2.3 Delete the SIMPL cluster
These commands are run from the DevOps Portal created in section 9.1

1. Select pipelines from the Left hand menu

1. Select pipelines from the sub menu

1. Select "All" from the resultant page

1. Expand pipelines

1. Select the pipeline `simpldestroy`

1. Select Run pipeline

1. In the resultant Run pipeline menu
   - Set the name of the release containing the SIMPL definitions to `{{ MVM_FILESHARE | default('<MVM_FILESHARE>') }}`
   - Set the name of the config file to `{{ PIPELINE_CONFIGURATION_NAME | default('<PIPELINE_CONFIGURATION_NAME>') }}`
   - Select Run

     Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.

     The progress of the pipeline can be followed by clicking on the jobs list.

     Once the pipeline has completed, success, or failure, is reported to screen

#### 9.2.2.4  Recreate the SIMPL cluster
These commands are run from the DevOps Portal created in section 9.1

1. Select pipelines from the Left hand menu

1. Select pipelines from the sub menu

1. Select "All" from the resultant page

1. Expand pipelines

1. Select the pipeline `simplapply`

1. Select Run pipeline

1. In the resultant Run pipeline menu
   - Set the name of the release containing the SIMPL definitions to `{{ MVM_FILESHARE | default('<MVM_FILESHARE>') }}`
   - Set the name of the config file to `{{ PIPELINE_CONFIGURATION_NAME | default('<PIPELINE_CONFIGURATION_NAME>') }}`
   - Leave `Enable grace mode for image verification` and `Enabled caching of image signatures` at their default values.
   - Select Run

     Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.

     The progress of the pipeline can be followed by clicking on the jobs list.

     Once the pipeline has completed, success, or failure, is reported to screen

#### 9.2.2.5 Verify that the SIMPL Cluster is still managing all the configuration

1. Set the following environment variables:

   ```
   REGIONAL_PIPELINE_KEY_VAULT_NAME={{ REGIONAL_PIPELINE_KEY_VAULT_NAME | default('<REGIONAL_PIPELINE_KEY_VAULT_NAME>') }}
   REGIONAL_PIPELINE_KEY_VAULT_RG={{ REGIONAL_PIPELINE_KEY_VAULT_RG | default('<REGIONAL_PIPELINE_KEY_VAULT_RG>') }}
   SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')
   ```

1. Allow your current session to access the key vault by running the following commands:
   ```
   az keyvault network-rule add \
      --name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Extract the SIMPL Resource Group by running the following command:
   ```
   SIMPL_SERVICE_RG=$(az keyvault secret show \
      --name simpl-service-rg \
      --vault-name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --query 'value' -o tsv)

   echo $SIMPL_SERVICE_RG
   ```

1. Remove the current session from the key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
      --resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Verify that the pods start successfully by running the command:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_RG}\
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get pods -A"
   ```

   You should see pods come up in the following namespaces:
   - connaisseur
   - csi-driver
   - default
   - gitops
   - istio-system
   - kube-system
   - simon
   - simpl

   If any of the pods enter a failed state (anything other than `Init`, `PodInitializing` or `Running`), see the Troubleshooting section in the Deployment Guide for troubleshooting guidance.

   >Depending on your Azure configuration, you might be prompted to re-login to Azure (including MFA) the first time the kubectl command executes. This is normal and to be expected

1. Get the application SIs now being managed by the SIMPL, by running the following commands:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get si"

   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get dc"
   ```

   Verify that the list of SIs returned above are the same as those extracted before respinning SIMPL.

#### 9.2.2.6 Verification

1. Verify that the SHA 256 Key in the **Region Certificate Details** Grafana panel matches
   the CA_FINGERPRINT value extracted in section 9.2.1

### 9.2.3 Generate a new Configuration Set
The new CA certificate has been loaded into the key vault. This step creates a
configuration set that references the new CA Certificate version.

#### 9.2.3.1 Set the default subscription
1. Set the default subscription by running the command:

   ```
   az account set --subscription "{{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}"

   ```

#### 9.2.3.2 Prepare the configuration Git repository

This is the Git repository that holds the code for mvm

1. Set the following environment variables:

   ```
   BRANCH={{ CHANGE_ID | default('<CHANGE_ID>') }}_update_encryption_CA_certificate
   CONFIG_VERSION={{ CONFIG_VERSION | default('<CONFIG_VERSION>') }}
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}@dev.azure.com/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/_git/{{ GIT_CONFIGURATION_REPOSITORY | default('<GIT_CONFIGURATION_REPOSITORY>') }}

      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/{{ GIT_CONFIGURATION_REPOSITORY | default('<GIT_CONFIGURATION_REPOSITORY>') }}

      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository
will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo

   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .

   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}

   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step


1. Create a new config set with an appropriate description by running the command:

   ```
   mvm-config-manager prepare-new-config \
      --from-config-version ${DOWNLEVEL_CONFIG_VERSION} \
      --description "Update Encryption CA certificate"
   ```

   This will create create a new config version and commit (but not push) the code.

   On success this will return a message of the form

   `Created new config version <UPLEVEL_CONFIG_VERSION> ...`

   Record the value of **UPLEVEL_CONFIG_VERSION**; it will be required later in this procedure.
   Set it as an environment variable as well:
   ```
   UPLEVEL_CONFIG_VERSION=<UPLEVEL_CONFIG_VERSION>
   ```



1. Allow your current session to access the key vault by running the following command:
   ```
   KEY_VAULT_NAME={{ KEY_VAULT_NAME | default('<KEY_VAULT_NAME>') }}
   KEY_VAULT_RG={{ KEY_VAULT_RG | default('<KEY_VAULT_RG>') }}
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')

   az keyvault network-rule add \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```

1. Download the secrets by running the command:

   ```
   ./scripts/secret_versions.sh \
      -p config/${UPLEVEL_CONFIG_VERSION}

   ```

   An SI uses the version of secrets as given by the contents of `secret_versions.vars`.
   This is an important file which helps ensure that SIs do not start using newer
   versions of secrets until they have been confirmed as safe via canarying.

1. Remove the current session from the key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}

   ```

1. Add any newly created files by running the command:
   ```
   git add -A

   ```

1. Commit the change to the local branch by running the command:
   ```
   git commit -a -m "Update secret versions"

   ```

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}


   ```

1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo


   ```

   (We have finished with the local copy of the repository)


### 9.2.4 Apply the new CA Certificate to the Application SIs
The new CA Certificate is applied to the application SIs by redeploying the SIs in the
following order:
- Simon
- SAS
- ESC
- CCM
- CMM

The SI should be replaced by following the appropriate upgrade MOP(s) as shown below.

|**SI**|**MOPs**|
|--|--|
|Simon | r1154upgradesimon.md |
|SAS   | r1154upgradesas.md & r1154deletesas.md |
|ESC   | r1154upgradeesc.md |
|CCM   | r1154upgradeccm.md |
|CMM   | r1154upgradecmm.md |


In addition to the validation process described in the appropriate upgrade MOP you
should also verify that the SHA 256 Key in the Region Certificate Details panel
matches the CA_FINGERPRINT value extracted in section 9.2.1

## 9.3 Test Plan

There is no explicit test plan associated with this MOP. You should refer to the
test plan associated with the appropriate SI upgrade MOP


## 9.4 Backout Procedure

There is no back-out process associated with this MOP. If the certificate is wrong
the MOP should be rerun to replace it with a new one.

# 10. Post checks

[System healthchecks]

# 11. Risk Assessment Score

1 - TBD

# 12. Execute MOP clean up if required

# 13. End of Document MOP

# 14. Service Assurance/Monitoring

# A. Appendix and Tables

# B. Approvers

# C. Peer Reviewers

# D. References for Other Documents

# E. Additional Appendices (If required)