|**Metadata**|**Description**  |
|--|--|
|Doc Title|  MVM v3: Upgrade the SIMPL cluster|
|Navigation|[WIKI Home Page](https://dev.azure.com/mvmprodeus2/MVM/_wiki/wikis/documentation/1/documents-home#)|
|Tracking| Document Number: VPE-5512-011 |
|Author| Graeme Thomson (gt163y) |
| Agreement Number | 24252.S.005 |

***
**Notices**

Copyright © 2025 Metaswitch Networks.  All rights reserved.

This manual is Confidential Information of Metaswitch Networks subject to the confidentiality terms
of the Agreement 01019223 as amended between AT&T and Metaswitch Networks.

It is issued on the understanding that no part of the product code or documentation (including this manual)
will be copied or distributed without prior agreement in writing from Metaswitch Networks and Microsoft.

Metaswitch Networks and Microsoft reserve the right to, without notice, modify or revise all or part of
this document and/or change product features or specifications and shall not be responsible for any
loss, cost, or damage, including consequential damage, caused by reliance on these materials.

Metaswitch and the Metaswitch logo are trademarks of Metaswitch Networks. Other brands and
products referenced herein are the trademarks or registered trademarks of their respective holders.

Product(s) and features documented in this manual handle various forms of data relating to your
users. You must comply with all laws and regulations applicable to your deployment, management,
and use of said product(s), and you should take all appropriate technical and organizational
measures to ensure you are handling this data appropriately according to any local legal and
regulatory obligations.

You are responsible for determining whether said product(s) or feature(s) is/are appropriate for
storage and processing of information subject to any specific law or regulation and for using said
product(s) or feature(s) in a manner consistent with your own legal and regulatory obligations. You
are also responsible for responding to any request from a third party regarding your use of said
product(s), such as a request to take down content under the U.S. Digital Millennium Copyright Act
or other applicable laws.


Metaswitch Networks
399 Main Street
Los Altos
CA 94022
<http://www.metaswitch.com>


***
***Table of Contents***
[[_TOC_]]

# 1. Document History

| **Issue** | **Issue Date** | **Author(s)** | **Identification** **of** **Changes** |
|-|-|-|-|
| 1| 06/10/2024| Gthomson|  initial draft |
| 2| 09/30/2024| Gthomson|  updates based on Ops feedback |
| 3| 11/26/2024| Gthomson|  Extract SIMPL RG programatically |
| 4| 11/26/2024| Gthomson|  Update default_vars.yml using sed |
| 5| 06/03/2025| GCradden|  Construct UPLEVEL_SIMPL_SERVICE_RG and DOWNLEVEL_SIMPL_SERVICE_RG based on version |

# 2. Versions

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  updates based on Ops feedback |
| 3| Gthomson|  Extract SIMPL RG programatically |
| 4| Gthomson|  Update default_vars.yml using sed |
| 5| GCradden|  Construct UPLEVEL_SIMPL_SERVICE_RG and DOWNLEVEL_SIMPL_SERVICE_RG based on version |

# 3. Integrated Solution Approach v1 (ISA v1)

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  updates based on Ops feedback |
| 3| Gthomson|  Extract SIMPL RG programatically |
| 4| Gthomson|  Update default_vars.yml using sed |
| 5| GCradden|  Construct UPLEVEL_SIMPL_SERVICE_RG and DOWNLEVEL_SIMPL_SERVICE_RG based on version |

# 4. MOP Impact Scope / General Information

## 4.1 Description

The SIMPL cluster is the "bottom turtle" in a deployment: it is a Kubernetes cluster,
which manages the creation and destruction of application Service Instances.

It does this by synchronizing with the config repository that declares the set of
desired SI instances, and then running Kubernetes Jobs to make the Service Instances
match those definitions.

This section explains how to upgrade the SIMPL cluster and the deployment-level
configuration that the SIMPL cluster manages.

## 4.2 Site Specific Description

| **Originator** | **Date** | **Time** |
|-|-|-|
| **Deployment Location(s)** | |
| **Description** | This MOP applies to the MVM V3 on Azure deployment, Release R11.5.3 | |

## 4.3 Service Impact

Service impact is not expected during this procedure.

## 4.4 Coordination

This MOP has no interactions outside of the MVM subscription.

# 5. Prerequisite/Dependencies/Entrance Criteria of MOP

This MOP is one of several that need to be run to execute the process to upgrade an existing deployment to an 11.5.3 release/patch.

Please refer to the corresponding *R11.5.3 Release Upgrade Overview* document for guidance on the order in which to run these MOPs

## 5.1 Required parameters

The following parameter values are required to run this MOP

| **Identifier** | **Description** |
|-|-|
| **CURRENT_CONFIG_VERSION** | The name of the config set directory containing the current SI configuration. This is located in the `config` directory |
| **DNS_ZONE** | Name of the global DNS zone |
| **DOWNLEVEL_MVM_FILESHARE** | Name of the fileshare containing the current MVM release |
| **DOWNLEVEL_PIPELINE_CONFIGURATION_NAME** | Name of the downlevel pipeline configuration file for the region without the .yml suffix (e.g. `vars_eus2_11400`). |
| **GIT_AUTOMATION_REPOSITORY** | Name of the automation Azure DevOps repository. |
| **GIT_AUTOMATION_URL** | URL of the automation git repository.|
| **GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository. |
| **GIT_CONFIGURATION_URL** | URL of the configuration git repository. |
| **GIT_PASSWORD** | Password used to access the Azure DevOps repositories if you are using https to manage the local copy of the access the repository. |
| **ORGANIZATION_NAME** | Name of the Azure DevOps organization. |
| **PROJECT** | Name of the Azure DevOps project. |
| **SIMPL_SERVICE_NAME** | An identifier for the SIMPL instance |
| **SUBSCRIPTION_ID** | Azure subscription identifier.  |
| **UPLEVEL_MVM_FILESHARE** | Name of the fileshare containing the Uplevel MVM release. (**This is specified in the release note**) |
| **UPLEVEL_PIPELINE_CONFIGURATION_NAME** | Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. `vars_eus2_11500`). This will contain details of the uplevel SIMPL. |
| **UPLEVEL_SIMPL_UNIQUE_SUFFIX** | A unique identifier to distinguish this SIMPL cluster from the existing SIMPL clusters, e.g.11500 |
| **UPLEVEL_SIMPL_VERSION** | Version of the uplevel SIMPL, e.g. 15.0.5. (**This is specified in the release note**) |
| **DOWNLEVEL_SIMPL_VERSION** | Version of the downlevel SIMPL, e.g. 14.0.0. (**This is specified in the release note**) |

## 5.2 Required files

No additional files are required to run this MOP.

# 6. Assumptions

The target audience for this procedure is the AT&T Engineer who will be performing the task. They will need to be familiar with Azure and have a working knowledge of the Azure CLI and Linux.

# 7. Material Requirements

## 7.1 Required Documents

## 7.2 Tools

| **Tool** | **Description** | **Quantity** |
|-|-|-|
| Laptop or Desktop PC | PC With at least 1G Memory and a network communications software application such as Procomm, Reflections or PuTTY | 1 |
| Azure connectivity PC | CloudShell Connectivity is required to the azure subscription. This can be accessed via [My Dashboard - Microsoft Azure](https://portal.azure.com/#cloudshell/) | |

# 8. Pre Maintenance Check, Precautions and Preparations

## 8.1 Precautions and Preparation

## 8.2 Precautions

> This procedure may cause a partial outage during implementation. Use executable script files to minimize down time and typing errors. Familiarize yourself with back-out procedures prior to starting the procedure.

| **Ask Yourself Principle** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Do I have the proper ID and appropriate building access permissions for the environment I am about to enter? | | |
| 2. Do I know why I'm doing this work? | | |
| 3. Have I identified and notified everybody - customers and internal groups - who will be directly affected by this work? | | |
| 4. Can I prevent or control service interruption? | | |
| 5. Is this the right time to do this work? | | |
| 6. Am I trained and qualified to do this work? | | |
| 7. Are the work orders, MOPs, and supporting documentation current and error-free? | | |
| 8. Do I have everything I need to quickly and safely restore service if something goes wrong? | | |
| 9. Have I walked through the procedure? | | |
| 10. Have I made sure the procedure includes proper closure including obtaining clearance and release for the appropriate work center? | | |


| **E911 Ask Yourself** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Does this work impact E911? | | |
| 2. Do I know how this work could impact 911/e911? | | |
| 3. Do I know what 911/e911 phase is required? | | |
| 4. Have I identified potential risks to 911/e911 and taken all measures to minimize? | | |
| 5. Does this work affect 15+ sites? | | |
| 6. Can I prevent or control service Interruptions to 911/e911? | | |
| 7. Is this the right time to do the work? | | |
| 8. Is the individual performing the work trained and qualified to do this work? | | |
| 9. Are MOPs and supporting documents current and error free? | | |
| 10. Does the MOP include a 911/e911 test plan? | | |
     

## 8.3 Pre-Maintenance Check Tools/System

Tier 2 needs to identify which tools they will use. This doesn't necessarily need to be included in the MOP but OPS needs to know which tools they will run.

(NEED TO USE STANDARD TOOLS) TIER 2


## 8.4 Pre-Maintenance Check Manual (Non-Automated Requirements)

These will be identify by the tier 3 MOP developer were required.

(MANDATORY CHECK REQUIRE FOR THE MOP) TIER 3


## 8.5 MOP Certification Environment

Examples:  PSL certified.  OR This MOP was paper certified by ATS engineers.

## 8.6 ATS Bulletin

**ATS Bulletin Check**
| **Step** | **Action** | **Results/Description** | **Timeline** |
|-|-|-|-|
| 1. | No Applicable bulletins | | |


## 8.7 Emergency Contacts

The following emergency contact numbers are to be used in the event provisioning support is required.

In the event a service interruption is encountered the AT&T Implementation Engineer will:
- Cease all work immediately.
- Notify the AT&T Voicemail TRC.
- Escalate to the next level of support.


| **Organization** | **Contact Name** | **Contact Number** |
|-|-|-|
| Voicemail TRC | SANRC | 877-662-7674, opt 3 |

# 9. Implementation

## 9.1 Preliminary Implementation
Pre-check tasks are completed the night of the cutover at least one hour prior to cutover activities.

1. Connect to the DevOps Portal
   1. Start a browser session to <https://dev.azure.com/>. This will be required to manage the pipelines
   1. Select the project associated with MVM v3
1. Connect to the Azure Portal
   1. Start a browser session to <https://portal.azure.com/>. This will be required to manage Azure resources
      and access the log analytics workspace (LAW)
   1. If prompted, complete the log in process
1. Connect to Azure Cloud Shell
   1. Start a CloudShell session by connecting a browser to <https://shell.azure.com/>
   1. If the menu at the top left indicates PowerShell select Bash from the menu and confirm at the prompt

      ![screenshot](images/powershell.jpg)
1. Upload any files and directories outlined in section 5.2 to your Cloud Shell account as they will be needed later


## 9.2 Implementation

### 9.2.1 Set the default subscription to the MVM subscription

1. Set the default subscription by running the command:

   ```
   az account set --subscription "Azure subscription identifier for the MVM subscription."
   ```

### 9.2.2 Prepare the automation Git repository

This is the Git repository that holds the pipelines, Terraform scripts etc.

These commands are run from the CloudShell session created above

1. Set the following environment variables:

   ```
   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_upgrade_simpl
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_AUTOMATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the automation Azure DevOps repository.

      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_AUTOMATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the automation Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir automation_repo
   cd automation_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_AUTOMATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_AUTOMATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step


### 9.2.3 Create a new pipeline configuration file

> This is an optional step that is only required if the release note and/or upgrade overview document indicates that the pipeline configuration file needs to be changed as part of the upgrade process.
>
> This will be the case if the upgrade process does ***NOT*** include an update of the pipeline infrastructure. In that case the uplevel pipeline configuration file will already have been created as part of that MOP.

1. Copy the existing pipeline configuration file to a new configuration file by running the command:

   ```
   cp pipelines/configuration/Name of the downlevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11400)..yml \
      pipelines/configuration/Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11500)..yml
   ```

1. Update the secrets monitoring configuration file by running the command:

   ```
   sed -i '/^  default_vars/c\  default_vars_file:  Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11500)..yml' pipelines/configuration/default_vars.yml

   ```

1. Add the new file to the repository by running the command:

   ```
   git add -A
   ```

1. Commit the change to the local branch by running the command:

   ```
   git commit -a -m "Create new pipeline configuration file"
   ```

### 9.2.4 Update the configuration to reflect the new SIMPL cluster

1. Update the pipeline configuration file by running the command: 

   ```
   sed -i '/^  simpl_unique_suffix:/c\  simpl_unique_suffix:  A unique identifier to distinguish this SIMPL cluster from the existing SIMPL clusters. May only contain alphanumeric characters and dashes, e.g.11500' pipelines/configuration/Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11500)..yml
   ```

1. Commit the change to the local branch by running the command:

   ```
   git commit -a -m "Update simpl configuration in pipeline configuration file"
   ```

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```
1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf automation_repo

   ```

   (We have finished with the local copy of the repository)

### 9.2.5 Create the uplevel SIMPL cluster

These commands are run from the DevOps Portal created in section 9.1

1. Select pipelines from the Left hand menu

1. Select pipelines from the sub menu

1. Select "All" from the resultant page

1. Expand pipelines

1. Select the pipeline `simplapply`

1. Select Run pipeline

1. In the resultant Run pipeline menu
   - Set the name of the release containing the SIMPL definitions to `Name of the fileshare containing the MVM release (This is specified in the release note)`
   - Set the name of the config file to `Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11500).`
   - Leave `Enable grace mode for image verification` and `Enabled caching of image signatures` at their default values.
   - Select Run

     Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.

     The progress of the pipeline can be followed by clicking on the jobs list.

     Once the pipeline has completed, success, or failure, is reported to screen

### 9.2.6 Add the uplevel SIMPL configuration to Git repository

Having created the Azure resources associated with the uplevel SIMPL we need to add
it to the configration repository so it can be managed by configuration manager

These commands are run from the CloudShell session created above

1. Set the following environment variables:

   ```
   CURRENT_CONFIG_VERSION=<CURRENT_CONFIG_VERSION>
   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_add_uplevel_simpl_configuration
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step

1. Update the SIMPL cluster configuration by running the following command:
   ```
   mvm-config-manager upgrade-simpl --config-version ${CURRENT_CONFIG_VERSION}
   ```

   If successful, a message similar to the one below is displayed on the screen

   `Applied config version 11.5.0-1-1 to SIMPL cluster simpl-21.3.5`

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```

1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo

   ```

   (We have finished with the local copy of the repository)


### 9.2.7 Verify the Uplevel SIMPL Cluster is functioning correctly
1. Set the following environment variables:

   ```
   SIMPL_SERVICE_NAME=An identifier for the SIMPL instance
   SUBSCRIPTION_ID=Azure subscription identifier for the MVM subscription.
   DOWNLEVEL_SIMPL_SERVICE_RG=${SIMPL_SERVICE_NAME}-Version of the downlevel SIMPL, e.g. 14.0.0. (This is specified in the release note)-rg
   DOWNLEVEL_SIMPL_SERVICE_RG=$(echo $DOWNLEVEL_SIMPL_SERVICE_RG | sed 's/\./-/g')
   UPLEVEL_SIMPL_SERVICE_RG=${SIMPL_SERVICE_NAME}-Version of the uplevel SIMPL, e.g. 15.0.5. (This is specified in the release note)-rg
   UPLEVEL_SIMPL_SERVICE_RG=$(echo $UPLEVEL_SIMPL_SERVICE_RG | sed 's/\./-/g')
   ```

1. Verify that the pods start successfully by running the command:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${UPLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get pods -A"
   ```

   You should see pods come up in the following namespaces:
   - connaisseur
   - csi-driver
   - default
   - gitops
   - istio-system
   - kube-system
   - simon
   - simpl

   If any of the pods enter a failed state (anything other than `Init`, `PodInitializing` or `Running`), see the Troubleshooting section in the Deployment Guide for troubleshooting guidance.

   
    If any pods get stuck in the `Init` state, and 
    ```
    az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${SIMPL_SERVICE_NAME}-rg \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl describe pod -n <namespace of initializing pod> <name of initializing pod>" 
    ```
    reports errors of the form `AADSTS700213: No matching federated identity record found for presented assertion subject`,
    there was an error with Entra ID initialization for the cluster. 
    Follow the backout procedure to destroy the service instance, then follow the MOP again to re-create the service instance.


   >Depending on your Azure configuration, you might be prompted to re-login to Azure (including MFA) the first time the kubectl command executes. This is normal and to be expected

1. Verify that there are two pods in the default namespace by running the command:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${UPLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get pods"
   ```

   You should see long running `service-instance-operator` and `deployment-config-operator` pods.

1. Verify that there are two pods in the simpl namespace by running the command:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${UPLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get pods -n simpl"
   ```

   You should see long running atm-endpoint-controller and dns-updater pods.


1. Get the down-level results for comparison purposes by running the following commands:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${DOWNLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get si"

   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${DOWNLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get dc"
    ```

1. Get the corresponding information from the up-level SIMPL cluster by running the following commands
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${UPLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get si"

   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${UPLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get dc"

1. Verify that the two outputs match:

   The age values will be different, but the names should be the same.

   If the output is different then stop and contact Support before proceeding with the back-out process

1. Log into Grafana

   Use the global URL `https://grafana-global.Name of the global DNS zone`

   Select the **SIMonMonitoring** dashboard and verify that the SI appears in the list of SIs in the Monitored Service Instances

   Thre should be two entries for SIMPL, one corresponding to the downlevel SIMPL
   and one to the uplevel SIMPL version as shown in the image below.

   ![screenshot](images/simplmonitor.jpg)


### 9.2.8 Delete the down-level SIMPL cluster

> **Note:** Before following the steps in this section, you must verify that the up-level SIMPL cluster is working correctly by following the steps in the verify stage above.
> After destroying the down-level SIMPL cluster, you will not be able to follow the backout steps to recover it.

Azure maintenance activity can result in the downlevel SIMPL being in a degraded state. In this degraded state, the delete operation may fail.

The following temporary workaround is recommended to ensure that the downlevel SIMPL is in a healthy state before proceeding with the deleting process.

1. Set the following environment variables:

   ```
   SIMPL_SERVICE_NAME=An identifier for the SIMPL instance
   SUBSCRIPTION_ID=Azure subscription identifier for the MVM subscription.
   DOWNLEVEL_SIMPL_SERVICE_RG=${SIMPL_SERVICE_NAME}-Version of the downlevel SIMPL, e.g. 14.0.0. (This is specified in the release note)-rg
   ```

1. Verify that all the nodes in the downlevel SIMPL are in a healthy state by running the command:
   ```
   az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${DOWNLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl get nodes -A"
   ```

   The `STATUS` of all of the nodes should be `Ready`.

   If any nodes have the `Ready,SchedulingDisabled` status the node must be deleted.

   To delete a node, run the following command:
   ```
    az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${DOWNLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
      --command "kubectl delete node <NODE_NAME>"
   ```

   where `<NODE_NAME>` is the name of the node with the `Ready,SchedulingDisabled` status.

   If a node is deleted, wait a few minutes and ensure that all nodes are in a `Ready` state before proceeding.

   For example, if the output of the `kubectl get nodes -A` command is:
   ```
   NAME                               STATUS                     ROLES    AGE     VERSION
   aks-nodepool-23948561-vmss000000   Ready,SchedulingDisabled   <none>   15d     v1.29.9
   aks-nodepool-23948561-vmss000001   Ready                      <none>   15d     v1.29.9
   ```

    The command to delete the node with the `Ready,SchedulingDisabled` status would be:
    ```
    az aks command invoke \
      --name ${SIMPL_SERVICE_NAME}-k8s \
      --resource-group ${DOWNLEVEL_SIMPL_SERVICE_RG} \
      --subscription "${SUBSCRIPTION_ID}" \
       --command "kubectl delete node aks-nodepool-23948561-vmss000000"
    ```

These commands are run from the DevOps Portal created in section 9.1

1. Select pipelines from the Left hand menu

1. Select pipelines from the sub menu

1. Select "All" from the resultant page

1. Expand pipelines

1. Select the pipeline `simpldestroy`

1. Select Run pipeline

1. In the resultant Run pipeline menu
   - Set the name of the release containing the SIMPL definitions to `Name of the fileshare containing the downlevel (current) MVM release`
   - Set the name of the config file to `Name of the downlevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11400).`
   - Select Run

     Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.

     The progress of the pipeline can be followed by clicking on the jobs list.

     Once the pipeline has completed, success, or failure, is reported to screen

### 9.2.9 Remove the down-level SIMPL from the configuration Git repository

This is the Git repository that holds the code and configuration for MVM.

These commands are run from the CloudShell session created above

1. Set the following environment variables:

   ```
   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_remove_downlevel_simpl_configuration
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step

1. Remove the down-level SIMPL directory by running the command
   ```
   DOWNLEVEL_SIMPL_VERSION=$(ls -d simpl-* | head -1 | sed 's/simpl-//')
   echo "delete SIMPL Version ${DOWNLEVEL_SIMPL_VERSION}"
   git rm -rf simpl-${DOWNLEVEL_SIMPL_VERSION}
   ```
1. Commit the change to the local branch by running the command:

   ```
   git commit -a -m "Remove downlevel simpl"
   ```

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```

1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo

   ```

   (We have finished with the local copy of the repository)

## 9.3 Test Plan

   There is no separate test plan, SIMPL is verified as part of the installation process

## 9.4 Backout Procedure

Backing out the SIMPL cluster creation is done by destroying the Uplevel SIMPL cluster

### 9.4.1 Delete the uplevel SIMPL cluster
These commands are run from the DevOps Portal created in section 9.1

1. Select pipelines from the Left hand menu

1. Select pipelines from the sub menu

1. Select "All" from the resultant page

1. Expand pipelines

1. Select the pipeline `simpldestroy`

1. Select Run pipeline

1. In the resultant Run pipeline menu
   - Set the name of the release containing the SIMPL definitions to `Name of the fileshare containing the MVM release (This is specified in the release note)`
   - Set the name of the config file to `Name of the uplevel pipeline configuration file for the region without the .yml suffix (e.g. vars_eus2_11500).`
   - Select Run

     Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.

     The progress of the pipeline can be followed by clicking on the jobs list.

     Once the pipeline has completed, success, or failure, is reported to screen

### 9.4.2 Remove the uplevel SIMPL from the configuration Git repository

This is the Git repository that holds the code and configuration for MVM.

These commands are run from the CloudShell session created above

1. Set the following environment variables:

   ```
   UPLEVEL_SIMPL_VERSION=Version of the uplevel SIMPL, e.g. 15.0.5. (This is specified in the release note)
   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_remove_uplevel_simpl_configuration
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step

1. Remove the uplevel SIMPL directory by running the command
   ```
   git rm -rf simpl-${UPLEVEL_SIMPL_VERSION}
   ```
1. Commit the change to the local branch by running the command:

   ```
   git commit -a -m "Remove uplevel simpl"
   ```

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```

1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo

   ```

   (We have finished with the local copy of the repository)

# 10. Post checks

[System healthchecks]

# 11. Risk Assessment Score

1 - TBD

# 12. Execute MOP clean up if required

# 13. End of Document MOP

# 14. Service Assurance/Monitoring

# A. Appendix and Tables

# B. Approvers

# C. Peer Reviewers

# D. References for Other Documents

# E. Additional Appendices (If required)