|**Metadata**|**Description**  |
|--|--|
|Doc Title|  MVM v3: Update Configuration Git Repository|
|Navigation|[WIKI Home Page](https://dev.azure.com/mvmprodeus2/MVM/_wiki/wikis/documentation/1/documents-home#)|
|Tracking| Document Number: VPE-5512-010|
|Author| Graeme Thomson (gt163y) |
| Agreement Number | 24252.S.005 |


***
**Notices**

Copyright © 2025 Metaswitch Networks.  All rights reserved.

This manual is Confidential Information of Metaswitch Networks subject to the confidentiality terms
of the Agreement 01019223 as amended between AT&T and Metaswitch Networks.

It is issued on the understanding that no part of the product code or documentation (including this manual)
will be copied or distributed without prior agreement in writing from Metaswitch Networks and Microsoft.

Metaswitch Networks and Microsoft reserve the right to, without notice, modify or revise all or part of
this document and/or change product features or specifications and shall not be responsible for any
loss, cost, or damage, including consequential damage, caused by reliance on these materials.

Metaswitch and the Metaswitch logo are trademarks of Metaswitch Networks. Other brands and
products referenced herein are the trademarks or registered trademarks of their respective holders.

Product(s) and features documented in this manual handle various forms of data relating to your
users. You must comply with all laws and regulations applicable to your deployment, management,
and use of said product(s), and you should take all appropriate technical and organizational
measures to ensure you are handling this data appropriately according to any local legal and
regulatory obligations.

You are responsible for determining whether said product(s) or feature(s) is/are appropriate for
storage and processing of information subject to any specific law or regulation and for using said
product(s) or feature(s) in a manner consistent with your own legal and regulatory obligations. You
are also responsible for responding to any request from a third party regarding your use of said
product(s), such as a request to take down content under the U.S. Digital Millennium Copyright Act
or other applicable laws.


Metaswitch Networks
399 Main Street
Los Altos
CA 94022
<http://www.metaswitch.com>


***
***Table of Contents***
[[_TOC_]]

# 1. Document History

| **Issue** | **Issue Date** | **Author(s)** | **Identification** **of** **Changes** |
|-|-|-|-|
| 1| 06/10/2024| Gthomson|  initial draft |
| 2| 09/10/2024| Gthomson|  Make OBER changes via inline command |
| 3| 09/15/2024| Gthomson|  Add declaration of DOWNLEVEL_MVM_VERSION |
| 4| 09/26/2024| Gthomson|  Explicitly set ACR_NOTARY_SIGNER, PERIMETA_EFXES & PERIMETA_VERSION |
| 5| 09/30/2024| Gthomson|  updates based on Ops feedback |
| 6| 11/26/2024| Gthomson|  Format cloud shell location in monospace format in section 5.2 |

# 2. Versions

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  Make OBER changes via inline command |
| 3| Gthomson|  Add declaration of DOWNLEVEL_MVM_VERSION |
| 4| Gthomson|  Explicitly set ACR_NOTARY_SIGNER, PERIMETA_EFXES & PERIMETA_VERSION |
| 5| Gthomson|  updates based on Ops feedback |
| 6| Gthomson|  Format cloud shell location in monospace format in section 5.2 |

# 3. Integrated Solution Approach v1 (ISA v1)

| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson|  initial draft |
| 2| Gthomson|  Make OBER changes via inline command |
| 3| Gthomson|  Add declaration of DOWNLEVEL_MVM_VERSION |
| 4| Gthomson|  Explicitly set ACR_NOTARY_SIGNER, PERIMETA_EFXES & PERIMETA_VERSION |
| 5| Gthomson|  updates based on Ops feedback |
| 6| Gthomson|  Format cloud shell location in monospace format in section 5.2 |

# 4. MOP Impact Scope / General Information

## 4.1 Description

Azure DevOps includes the MVM configuration Git repository that hosts configuration
for the set of application SIs, and the SIMPL cluster that controls them.

There is a separate repository for each region in the MVM deployment.

This mop describes the process to upgrade an existing configuration repository

## 4.2 Site Specific Description

| **Originator** | **Date** | **Time** |
|-|-|-|
| **Deployment Location(s)** | |
| **Description** | This MOP applies to the MVM V3 on Azure deployment, Release R11.5.3 | |

## 4.3 Service Impact

Service impact is not expected during this procedure.

## 4.4 Coordination

This MOP has no interactions outside of the MVM subscription.

# 5. Prerequisite/Dependencies/Entrance Criteria of MOP

This MOP is one of several that need to be run to execute the process to upgrade an existing deployment to an 11.5.3 release/patch.

Please refer to the corresponding *R11.5.3 Release Upgrade Overview* document for guidance on the order in which to run these MOPs

## 5.1 Required parameters

The following parameter values are required to run this MOP

| **Identifier** | **Description** |
|-|-|
| **CLOUD_SHELL_LOCATION** | The location of the files that were copied to the Cloud Shell account. |
| **DOWNLEVEL_CONFIG_VERSION** | The name of the config set directory containing the current SI configuration. Config sets are located in the `config` directory. |
| **DOWNLEVEL_MVM_VERSION** | Down-level Microsoft MVM release version, e.g. 11.5.0+1. This is a sub-directory of templates. The actual value to use is specified in the upgrade overview document that accompanies the release |
| **GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository. |
| **GIT_CONFIGURATION_URL** |  URL of the configuration git repository. |
| **GIT_PASSWORD** | Password used to access the Azure DevOps repositories if you are using https to manage the local copy of the access the repository. |
| **KEY_VAULT_NAME** | Name of the Key Vault where secrets and certificates are stored for a single region. |
| **KEY_VAULT_RG** | Name of the resource group that contains the key vault **KEY_VAULT_NAME** |
| **ORGANIZATION_NAME** | Name of the Azure DevOps organization. |
| **PROJECT** | Name of the Azure DevOps project. |
| **SUBSCRIPTION_ID** | Azure subscription identifier.  |
| **UPLEVEL_CONFIG_VERSION** | Configuration version returned by `mvm-config-manager prepare-new-config` |
| **UPLEVEL_MVM_FILESHARE** | Name of the fileshare containing the MVM release (**This is specified in the release note**) |
| **UPLEVEL_MVM_VERSION** | Up-level Microsoft MVM release version, e.g., 11.5.0+2. This is a sub-directory of templates. The actual value to use is specified in the upgrade overview document that accompanies the release |

## 5.2 Required files

The following file from the release point `Name of the fileshare containing the MVM release (This is specified in the release note)` must be uploaded to `The location of the files that were copied to the Cloud Shell account.` before starting this MOP:
-  `git/repo_template/main-The version number of the uplevel release, e.g. 11.5.0+1.tgz`.

# 6. Assumptions

The target audience for this procedure is the AT&T Engineer who will be performing the task. They will need to be familiar with Azure and have a working knowledge of the Azure CLI and Linux.

# 7. Material Requirements

## 7.1 Required Documents

## 7.2 Tools

| **Tool** | **Description** | **Quantity** |
|-|-|-|
| Laptop or Desktop PC | PC With at least 1G Memory and a network communications software application such as Procomm, Reflections or PuTTY | 1 |
| Azure connectivity PC | CloudShell Connectivity is required to the azure subscription. This can be accessed via [My Dashboard - Microsoft Azure](https://portal.azure.com/#cloudshell/) | |

# 8. Pre Maintenance Check, Precautions and Preparations

## 8.1 Precautions and Preparation

## 8.2 Precautions

> This procedure may cause a partial outage during implementation. Use executable script files to minimize down time and typing errors. Familiarize yourself with back-out procedures prior to starting the procedure.

| **Ask Yourself Principle** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Do I have the proper ID and appropriate building access permissions for the environment I am about to enter? | | |
| 2. Do I know why I'm doing this work? | | |
| 3. Have I identified and notified everybody - customers and internal groups - who will be directly affected by this work? | | |
| 4. Can I prevent or control service interruption? | | |
| 5. Is this the right time to do this work? | | |
| 6. Am I trained and qualified to do this work? | | |
| 7. Are the work orders, MOPs, and supporting documentation current and error-free? | | |
| 8. Do I have everything I need to quickly and safely restore service if something goes wrong? | | |
| 9. Have I walked through the procedure? | | |
| 10. Have I made sure the procedure includes proper closure including obtaining clearance and release for the appropriate work center? | | |


| **E911 Ask Yourself** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Does this work impact E911? | | |
| 2. Do I know how this work could impact 911/e911? | | |
| 3. Do I know what 911/e911 phase is required? | | |
| 4. Have I identified potential risks to 911/e911 and taken all measures to minimize? | | |
| 5. Does this work affect 15+ sites? | | |
| 6. Can I prevent or control service Interruptions to 911/e911? | | |
| 7. Is this the right time to do the work? | | |
| 8. Is the individual performing the work trained and qualified to do this work? | | |
| 9. Are MOPs and supporting documents current and error free? | | |
| 10. Does the MOP include a 911/e911 test plan? | | |
     

## 8.3 Pre-Maintenance Check Tools/System

Tier 2 needs to identify which tools they will use. This doesn't necessarily need to be included in the MOP but OPS needs to know which tools they will run.

(NEED TO USE STANDARD TOOLS) TIER 2


## 8.4 Pre-Maintenance Check Manual (Non-Automated Requirements)

These will be identify by the tier 3 MOP developer were required.

(MANDATORY CHECK REQUIRE FOR THE MOP) TIER 3


## 8.5 MOP Certification Environment

Examples:  PSL certified.  OR This MOP was paper certified by ATS engineers.

## 8.6 ATS Bulletin

**ATS Bulletin Check**
| **Step** | **Action** | **Results/Description** | **Timeline** |
|-|-|-|-|
| 1. | No Applicable bulletins | | |


## 8.7 Emergency Contacts

The following emergency contact numbers are to be used in the event provisioning support is required.

In the event a service interruption is encountered the AT&T Implementation Engineer will:
- Cease all work immediately.
- Notify the AT&T Voicemail TRC.
- Escalate to the next level of support.


| **Organization** | **Contact Name** | **Contact Number** |
|-|-|-|
| Voicemail TRC | SANRC | 877-662-7674, opt 3 |

# 9. Implementation

## 9.1 Preliminary Implementation
Pre-check tasks are completed the night of the cutover at least one hour prior to cutover activities.

1. Connect to the DevOps Portal
   1. Start a browser session to <https://dev.azure.com/>. This will be required to manage the pipelines
   1. Select the project associated with MVM v3
1. Connect to the Azure Portal
   1. Start a browser session to <https://portal.azure.com/>. This will be required to manage Azure resources
      and access the log analytics workspace (LAW)
   1. If prompted, complete the log in process
1. Connect to Azure Cloud Shell
   1. Start a CloudShell session by connecting a browser to <https://shell.azure.com/>
   1. If the menu at the top left indicates PowerShell select Bash from the menu and confirm at the prompt

      ![screenshot](images/powershell.jpg)
1. Upload any files and directories outlined in section 5.2 to your Cloud Shell account as they will be needed later


## 9.2 Implementation
### 9.2.1 Set the default subscription to the MVM subscription

1. Set the default subscription by running the command:

   ```
   az account set --subscription "Azure subscription identifier for the MVM subscription."
   ```

### 9.2.2 Prepare the configuration Git repository

This is the Git repository that holds the code and configuration for MVM.

These commands are run from the CloudShell session created above

1. Set the following environment variables:

   ```
   CLOUD_SHELL_LOCATION=The location of the files that were copied to the Cloud Shell account.
   DOWNLEVEL_CONFIG_VERSION=The name of the config set directory containing the downlevel SI configuration. Config sets are located in the `config` directory.
   DOWNLEVEL_MVM_VERSION=The version number of the downlevel release, e.g. 11.4.0+4
   KEY_VAULT_NAME=Name of the Key Vault where secrets and certificates are stored for a single region.
   KEY_VAULT_RG=Name of the resource group that contains the key vault KEY_VAULT_NAME
   UPLEVEL_MVM_VERSION=The version number of the uplevel release, e.g. 11.5.0+1

   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_update_configuration
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step

### 9.2.3 Add the release to the repository

1. Populate the configuration repository with the file delivered with the release by running the command:

   ```
   mvm-config-manager populate --source ${CLOUD_SHELL_LOCATION}/main-${UPLEVEL_MVM_VERSION}.tgz
   ```

   If successful, this will commit the code (but not push it) with a check-in message along the lines of

   ```
   [mvm-config-manager] Populate repo with templates for software version 11.5.0-rc.18.3 from ../../../11.51.0-rc.18.3/git/repo_template/main-11.5.0-rc.18.3.tgz
   ```

### 9.2.4 Create a backup of the supplied configuration set

1. Create a back-up copy of the template set by running the following commands:

   ```
   cd templates
   mkdir -p ../templates.bak
   cp -r ${UPLEVEL_MVM_VERSION} ../templates.bak/${UPLEVEL_MVM_VERSION}.bak
   cd ..
   git add -A
   git commit -a -m "Make backup of template set"
   ```

   We need to do this so we can make any manual changes to the supplied configuration set


### 9.2.5 Apply manual configuration changes

1. Disable OBER

   - Set the BHCA per subscriber to zero by running the following command:
     ```
     sed -i 's/bhca: 18/bhca: 0/' templates/${UPLEVEL_MVM_VERSION}/config/esc/mvm-esc.yaml
     ```

   - Reduce the healthy_threshold from .99 to .9 by running the following command:
     ```
     sed -i 's/healthy_threshold: 0.99/healthy_threshold: 0.9/' templates/${UPLEVEL_MVM_VERSION}/config/esc/mvm-esc.yaml
     ```

     This mitigates the issue of one bad call having an adverse magnified effect on an ESC when
     the average amount of traffic is low. This parameter is put in place in anticipation that
     OBER will, at some point, be re-enabled.

1. Commit the change to the local branch by running the command:

   ```
   git commit -a -m "Update OBER related parameters"
   ```

### 9.2.6 Create the initial uplevel configuration version


1. Create a new config set with an appropriate description by running the command:

   ```
   mvm-config-manager prepare-new-config \
      --from-config-version ${DOWNLEVEL_CONFIG_VERSION} \
      --description "initial config set for ${UPLEVEL_MVM_VERSION}"
   ```

   This will create create a new config version and commit (but not push) the code.

   On success this will return a message of the form

   `Created new config version <UPLEVEL_CONFIG_VERSION> ...`

   Record the value of **UPLEVEL_CONFIG_VERSION**; it will be required later in this procedure.
   Set it as an environment variable as well:
   ```
   UPLEVEL_CONFIG_VERSION=<UPLEVEL_CONFIG_VERSION>
   ```




   The output from previous step may have included the message
   `The following variables have no values set and must be filled in manually:`
   indicating that new variables have been added which could not be automatically
   populated from the previous config version. If this is the case fill in the
   listed variables in the specified files in `config/${UPLEVEL_CONFIG_VERSION}`

   New variables that are required will be tagged in the appropriate file with the line:

   `# @@@TODO: Fill in this variable`

   The value & format for the variable should be defined in the file itself and
   flagged in the upgrade overview document. Once the variable has been configured you should delete the line

   `# @@@TODO: Fill in this variable`

   >**References to variables associated with the MALT SI can be ignored, these do not need to be configured.**

1. Commit the changes by running the command:
   ```
   git commit -a -m "Add additional variables required in new version"
   ```

### 9.2.7 Update variables in the configuration set

This step updates variables that always need to be modified in the new configuration set

1. Set the Notary signer

   - Set ACR_NOTARY_SIGNER to the uplevel release by running the following command:
     ```
     sed -i '/^ACR_NOTARY_SIGNER=/c\ACR_NOTARY_SIGNER=Name of the fileshare containing the MVM release (This is specified in the release note)--' config/${UPLEVEL_CONFIG_VERSION}/region.vars
     ```

1. Set the Perimeta efix list

   - Set PERIMETA_EFIXES by running the following command:
     ```
     sed -i '/^PERIMETA_EFIXES=/c\PERIMETA_EFIXES="The list of efixes to install comma separated, specify as "" if no efixes. This is specified in the release note"' config/${UPLEVEL_CONFIG_VERSION}/esc/esc.vars
     ```

1. Set the Perimeta image version

   - Set PERIMETA_IMAGE_NAME by running the following command:
     ```
     sed -i '/^PERIMETA_IMAGE_NAME=/c\PERIMETA_IMAGE_NAME=The perimeta version, takes the form MSwVA-Perimeta-V5.4.10_SU56_P01-6GB_image. This is specified in the release note' config/${UPLEVEL_CONFIG_VERSION}/esc/esc.vars
     ```

1. Commit the changes to the local branch by running the command:

   ```
   git commit -a -m "Update fixed config set variables"
   ```

### 9.2.8 Update any additional variables in the configuration set

>This is an optional step that is only required if the release note and/or upgrade
>overview document indicates existing variables have to be changed as part of the release process

1. Edit the appropriate variable files as identified in the release documentation.

1. Commit the change to the local branch by running the command:
   ```
   git commit -a -m "Update existing variables"
   ```

### 9.2.9 Add any additional Perimeta configuration

>This is an optional step that is only required if the release note and/or upgrade
>overview document indicates that additional Perimeta configuration is required.

1. Edit the file `config/${UPLEVEL_CONFIG_VERSION}/esc/perimeta-config`

   Add the configuration identified in the release note / upgrade overview document and save the file

2. Commit the change to the local branch by running the command:
   ```
   git commit -a -m "Update Perimeta configuration"


### 9.2.10 Add any new secrets

>This is an optional step that is only required if the release note and/or upgrade
>overview document indicates new secrets have been added as part of the release

1. Allow your current session to access the key vault by running the following commands:
   ```
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')

   az keyvault network-rule add \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Create a file, `/tmp/secrets.txt`, to contain the new secrets.

   The release note and upgrade overview document will both provide details on
   the contents and format of this file.

1. Upload the secrets

   Run the following command to upload the secrets file to the key vault:
   ```
   ./scripts/upload_secrets.sh \
      --keyvault ${KEY_VAULT_NAME} \
      --secrets-file /tmp/secrets.txt
   ```

   The output will show a JSON description of the secret(s) that you have set.

1. Verify that the secrets have all been uploaded by running the following command:
   ```
   az keyvault secret list \
      --vault-name ${KEY_VAULT_NAME} \
      --query '[].name'
   ```

   Check that all the names in `/tmp/secrets.txt` appear in the output

1. Remove the secrets file as it is no longer required by running the command
   ```
   rm /tmp/secrets.txt
   ```

1. Remove the current session from the key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

### 9.2.11 Refresh the secrets

This step ensures that the upgraded configuration set is using the latest set of secrets

1. Allow your current session to access the key vault by running the following commands

   ```
   USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')

   az keyvault network-rule add \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Download the secrets by running the command:

   ```
   ./scripts/secret_versions.sh \
      -p config/${UPLEVEL_CONFIG_VERSION}
   ```

   An SI uses the version of secrets as given by the contents of `secret_versions.vars`.
   This is an important file which helps ensure that SIs do not start using newer
   versions of secrets until they have been confirmed as safe via canarying.

1. Remove the current session from the key vault access list by running the following command:
   ```
   az keyvault network-rule remove \
      --name ${KEY_VAULT_NAME} \
      --resource-group ${KEY_VAULT_RG} \
      --ip-address ${USER_IP}
   ```

1. Add any newly created files by running the command:
   ```
   git add -A
   ```

1. Commit the change to the local branch by running the command:
   ```
   git commit -a -m "Update secret versions"
   ```

### 9.2.12 Upgrade the migration Terraform file

1. Extract the example Terraform file by running the following command
   ```
   tar -zxvf ${CLOUD_SHELL_LOCATION}/main-${UPLEVEL_MVM_VERSION}.tgz \
        postgresql/terraform.tfvars.example --transform \
        "s#terraform.tfvars.example#""terraform.tfvars.example.${UPLEVEL_MVM_VERSION}#"
   ```

2. Check to see if there are any differences in the example file by comparing it with the previous version:
   ```
   diff postgresql/terraform.tfvars.example.${UPLEVEL_MVM_VERSION} \
        postgresql/terraform.tfvars.example.${DOWNLEVEL_MVM_VERSION}
   ```

   If there are, then apply the same changes to `postgresql/terraform.tfvars`.

   These will be picked up the next time the migration SQL databases are upgraded

1. Add any newly created files by running the command:
   ```
   git add -A
   ```

1. Commit the change to the local branch by running the command
   ```
   git commit -a -m "Update migration Psql file"
   ```

### 9.2.13 Apply the configuration to the Service Instances (SIs)
1. Apply the configuration to all of the non-active SIs by running the command:
   ```
   mvm-config-manager apply-config --config-version ${UPLEVEL_CONFIG_VERSION}
   ```

   Success or failure is reported to screen together with a list of SIs that have had the configuration applied to them, e.g.
   ```
   > mvm-config-manager apply-config
       Applied config version 11.5.0+1-0 to all SIs.

       Upgraded SIs:

       c01z1
       c01z2
       c01z3
       c02z2
       c02z3
       e01z1
       e01z3
       g01z1
       g01z2
       g01z3
       g02z2
       g02z3
       m01z1
       m01z3
       s01z1
       s01z3

   ```

### 9.2.14 Update the remote repository
1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```

1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo

   ```

   (We have finished with the local copy of the repository)


## 9.3 Test Plan

   There is no specific test plan associated with this procedure. If all commands complete successfully then the operator can proceed to the next MOP.

## 9.4 Backout Procedure

The back-out procedure is to delete the configuration version created above

### 9.4.1 Delete the new configuration

1. Set the following environment variables:

   ```
   BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_rollback_configuration
   UPLEVEL_MVM_VERSION=The version number of the uplevel release, e.g. 11.5.0+1
   ```

   Export the correct form of the URL to access the git repository
   -  If using https to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
      ```

   -  If using ssh to interact with the git repository
      ```
      GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
      ```

1. Change to an appropriate working directory in Cloud shell. Your Git repository will live in a subdirectory off of this path.

   ```
   cd ~
   mkdir configuration_repo
   cd configuration_repo
   ```

1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
   ```
   git clone ${GIT_CONFIGURATION_URL} .
   ```
   (note the trailing whitespace and period after the URL)

   -  If using HTTPS:
      - When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
   - If using SSH:
      - You will not be prompted for a password.

   This will create a local copy of the repository in the current working directory.

1. Create a new working branch by running the command
   ```
   git checkout -b ${BRANCH}
   ```

   The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step

1. Remove the newly created configuration version by running the command
   ```
   mvm-config-manager delete-old-config-versions -software-version ${UPLEVEL_MVM_VERSION}
   ```

1. Push the change to the DevOps repository by running the command:

   ```
   git push --set-upstream origin ${BRANCH}

   ```
1. Merge the change into the main branch via the 'pull request' mechanism

1. Tidy up by running the command:

   ```
   cd ~
   rm -rf configuration_repo

   ```

   (We have finished with the local copy of the repository)


# 10. Post checks

[System healthchecks]

# 11. Risk Assessment Score

1 - TBD

# 12. Execute MOP clean up if required

# 13. End of Document MOP

# 14. Service Assurance/Monitoring

# A. Appendix and Tables

# B. Approvers

# C. Peer Reviewers

# D. References for Other Documents

# E. Additional Appendices (If required)