r1153upgradeesc.md
markdown_main
rendered_scus/r1153upgradeesc.md
Rendered Markdown
2026 lines
|**Metadata**|**Description** |
|--|--|
|Doc Title| MVM v3: Upgrade ESC SI|
|Navigation|[WIKI Home Page](https://dev.azure.com/mvmprodeus2/MVM/_wiki/wikis/documentation/1/documents-home#)|
|Tracking| Document Number: VPE-5512-017|
|Author| Graeme Thomson (gt163y) |
| Agreement Number | 24252.S.005 |
***
**Notices**
Copyright © 2025 Metaswitch Networks. All rights reserved.
This manual is Confidential Information of Metaswitch Networks subject to the confidentiality terms
of the Agreement 01019223 as amended between AT&T and Metaswitch Networks.
It is issued on the understanding that no part of the product code or documentation (including this manual)
will be copied or distributed without prior agreement in writing from Metaswitch Networks and Microsoft.
Metaswitch Networks and Microsoft reserve the right to, without notice, modify or revise all or part of
this document and/or change product features or specifications and shall not be responsible for any
loss, cost, or damage, including consequential damage, caused by reliance on these materials.
Metaswitch and the Metaswitch logo are trademarks of Metaswitch Networks. Other brands and
products referenced herein are the trademarks or registered trademarks of their respective holders.
Product(s) and features documented in this manual handle various forms of data relating to your
users. You must comply with all laws and regulations applicable to your deployment, management,
and use of said product(s), and you should take all appropriate technical and organizational
measures to ensure you are handling this data appropriately according to any local legal and
regulatory obligations.
You are responsible for determining whether said product(s) or feature(s) is/are appropriate for
storage and processing of information subject to any specific law or regulation and for using said
product(s) or feature(s) in a manner consistent with your own legal and regulatory obligations. You
are also responsible for responding to any request from a third party regarding your use of said
product(s), such as a request to take down content under the U.S. Digital Millennium Copyright Act
or other applicable laws.
Metaswitch Networks
399 Main Street
Los Altos
CA 94022
<http://www.metaswitch.com>
***
***Table of Contents***
[[_TOC_]]
# 1. Document History
| **Issue** | **Issue Date** | **Author(s)** | **Identification** **of** **Changes** |
|-|-|-|-|
| 1| 06/10/2024| Gthomson| initial draft |
| 2| 09/30/2024| Gthomson| updates based on Ops feedback |
# 2. Versions
| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson| initial draft |
| 2| Gthomson| updates based on Ops feedback |
# 3. Integrated Solution Approach v1 (ISA v1)
| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson| initial draft |
| 2| Gthomson| updates based on Ops feedback |
# 4. MOP Impact Scope / General Information
## 4.1 Description
The ESC SI is the application SI that provides Voicemail and provisioning capability
This MOP describes the process to upgrade one or more ESC SIs.
The process to upgrade an ESC involves canarying a portion of ESC traffic to the uplevel ESC. This includes:
- TUI calls
- VVM traffic
- NWSAP provisioning
- Operators using CLI tools and MSM
- Timer callbacks
The following traffic is not canaried, and is automatically distributed onto the uplevel ESC:
- MO-SMS SMPP bind requests from MABs
- Inter-subscriber VPIM messages
When canarying the first ESC SI a cautious approach to moving traffic should be taken.
When the uplevel ESC SI is handling 100% of the corresponding downlevel ESC SI's traffic
without issue, the canary procedure may be performed more aggressively for the other
downlevel ESC SIs.
## 4.2 Site Specific Description
| **Originator** | **Date** | **Time** |
|-|-|-|
| **Deployment Location(s)** | |
| **Description** | This MOP applies to the MVM V3 on Azure deployment, Release R11.5.3 | |
## 4.3 Service Impact
Service impact is not expected during this procedure.
## 4.4 Coordination
This MOP has no interactions outside of the MVM subscription.
# 5. Prerequisite/Dependencies/Entrance Criteria of MOP
This MOP is one of several that need to be run to execute the process to upgrade an
existing deployment to an 11.5.3 release/patch.
Please refer to the corresponding *R11.5.3 Release Upgrade Overview* document for guidance
on the order in which to run these MOPs
## 5.1 Required parameters
The following parameter values are required to run this MOP
| **Identifier** | **Description** |
|-|-|
| **CONFIG_VERSION** | The name of the config set directory containing the current SI configuration. Config sets are located in the `config` directory |
| **DEPLOYMENT_GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository in the **deployment region**. |
| **DEPLOYMENT_GIT_CONFIGURATION_URL** | URL of the configuration git repository in the **deployment region**. |
| **DEPLOYMENT_ORGANIZATION_NAME** | Name of the Azure DevOps organization in the **deployment region**. |
| **DEPLOYMENT_PROJECT** | Name of the Azure DevOps project in the **deployment region**. |
| **DNS_ZONE** | Name of the global DNS zone |
| **DOMAIN_TLS_CERT** | PEM format TLS certificate or certificate chain (with the domain certificate first) for your domain, used for all HTTP and Custom VVM access |
| **ESC_SI_NAME** | Name of the ESC service instance |
| **ESC_SI_WEIGHT_IDENTIFIER** | The name of the weight identifier for an ESC SI. This is defined in the file `variables/si_weights.vars` |
| **GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository. |
| **GIT_CONFIGURATION_URL** | URL of the configuration git repository. |
| **GIT_PASSWORD** | Password used to access the Azure DevOps repositories if you are using https to manage the local copy of the access the repository. |
| **MVM_AUTH_APPLICATION_URI** | The application URI for this MVM deployment |
| **MVM_AUTH_CLIENT_ID** | The Application (client) ID of the registered Enterprise application for MVM |
| **ORGANIZATION_NAME** | Name of the Azure DevOps organization. |
| **PERIMETA_COUNT** | Number of Perimeta VMs to create in each ESC |
| **PERIMETA_DCR** | Name of the Data Collection Rule resource (DCR) used by Perimeta VMs to record logs |
| **PROJECT** | Name of the Azure DevOps project. |
| **REGION_LAW** | Name of the Log Analytics Workspace (LAW) associated with the region |
| **REGION_SHORTNAME** | The short (4-characters maximum) DNS label for the region |
| **SUBSCRIPTION_ID** | Azure subscription identifier. |
| **TENANT_ID** | Azure tenant identifier. |
| **TRUSTED_CA_BUNDLE** | The full path to a .pem file on the tooling VM containing the Root CA certificate used to sign the **DOMAIN_TLS_CERT**, which must be trusted by users of MVM APIs. This file must contain a single CA certificate. Multiple CA certs are not supported. |
## 5.2 Required files
No additional files are required to run this MOP.
# 6. Assumptions
The target audience for this procedure is the AT&T Engineer who will be performing the task. They will need to be familiar with Azure and have a working knowledge of the Azure CLI and Linux.
# 7. Material Requirements
## 7.1 Required Documents
## 7.2 Tools
| **Tool** | **Description** | **Quantity** |
|-|-|-|
| Laptop or Desktop PC | PC With at least 1G Memory and a network communications software application such as Procomm, Reflections or PuTTY | 1 |
| Azure connectivity PC | CloudShell Connectivity is required to the azure subscription. This can be accessed via [My Dashboard - Microsoft Azure](https://portal.azure.com/#cloudshell/) | |
# 8. Pre Maintenance Check, Precautions and Preparations
## 8.1 Precautions and Preparation
## 8.2 Precautions
> This procedure may cause a partial outage during implementation. Use executable script files to minimize down time and typing errors. Familiarize yourself with back-out procedures prior to starting the procedure.
| **Ask Yourself Principle** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Do I have the proper ID and appropriate building access permissions for the environment I am about to enter? | | |
| 2. Do I know why I'm doing this work? | | |
| 3. Have I identified and notified everybody - customers and internal groups - who will be directly affected by this work? | | |
| 4. Can I prevent or control service interruption? | | |
| 5. Is this the right time to do this work? | | |
| 6. Am I trained and qualified to do this work? | | |
| 7. Are the work orders, MOPs, and supporting documentation current and error-free? | | |
| 8. Do I have everything I need to quickly and safely restore service if something goes wrong? | | |
| 9. Have I walked through the procedure? | | |
| 10. Have I made sure the procedure includes proper closure including obtaining clearance and release for the appropriate work center? | | |
| **E911 Ask Yourself** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Does this work impact E911? | | |
| 2. Do I know how this work could impact 911/e911? | | |
| 3. Do I know what 911/e911 phase is required? | | |
| 4. Have I identified potential risks to 911/e911 and taken all measures to minimize? | | |
| 5. Does this work affect 15+ sites? | | |
| 6. Can I prevent or control service Interruptions to 911/e911? | | |
| 7. Is this the right time to do the work? | | |
| 8. Is the individual performing the work trained and qualified to do this work? | | |
| 9. Are MOPs and supporting documents current and error free? | | |
| 10. Does the MOP include a 911/e911 test plan? | | |
## 8.3 Pre-Maintenance Check Tools/System
Tier 2 needs to identify which tools they will use. This doesn't necessarily need to be included in the MOP but OPS needs to know which tools they will run.
(NEED TO USE STANDARD TOOLS) TIER 2
## 8.4 Pre-Maintenance Check Manual (Non-Automated Requirements)
These will be identify by the tier 3 MOP developer were required.
(MANDATORY CHECK REQUIRE FOR THE MOP) TIER 3
## 8.5 MOP Certification Environment
Examples: PSL certified. OR This MOP was paper certified by ATS engineers.
## 8.6 ATS Bulletin
**ATS Bulletin Check**
| **Step** | **Action** | **Results/Description** | **Timeline** |
|-|-|-|-|
| 1. | No Applicable bulletins | | |
## 8.7 Emergency Contacts
The following emergency contact numbers are to be used in the event provisioning support is required.
In the event a service interruption is encountered the AT&T Implementation Engineer will:
- Cease all work immediately.
- Notify the AT&T Voicemail TRC.
- Escalate to the next level of support.
| **Organization** | **Contact Name** | **Contact Number** |
|-|-|-|
| Voicemail TRC | SANRC | 877-662-7674, opt 3 |
# 9. Implementation
## 9.1 Preliminary Implementation
Pre-check tasks are completed the night of the cutover at least one hour prior to cutover activities.
1. Connect to the DevOps Portal
1. Start a browser session to <https://dev.azure.com/>. This will be required to manage the pipelines
1. Select the project associated with MVM v3
1. Connect to the Azure Portal
1. Start a browser session to <https://portal.azure.com/>. This will be required to manage Azure resources
and access the log analytics workspace (LAW)
1. If prompted, complete the log in process
1. Connect to Azure Cloud Shell
1. Start a CloudShell session by connecting a browser to <https://shell.azure.com/>
1. If the menu at the top left indicates PowerShell select Bash from the menu and confirm at the prompt
1. Upload any files and directories outlined in section 5.2 to your Cloud Shell account as they will be needed later
## 9.2 Implementation
***
**This procedure *may* require updates to the git configuration repository in two regions.**
Updates are always required to the configuration in the deployment region. These are
required to update the deployment wide settings, e.g. ESC weight and priority.
If the ESC is not in the deployment region, then changes that are specific to the ESC
(e.g. starting and stopping the ESC) are made to the repository in the region that
the ESC is located in.
***
### 9.2.1 Set the default subscription to the MVM subscription
1. Set the default subscription by running the command:
```
az account set --subscription "Azure subscription identifier for the MVM subscription."
```
### 9.2.2 Prepare the configuration git repository
This is the git repository that holds the code for mvm
1. Set the following environment variables:
```
CONFIG_VERSION=<CONFIG_VERSION>
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_upgrade_esc_si
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
```
- If using ssh to interact with the git repository
```
GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
```
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
### 9.2.3 Check the configuration for the new SIs
Before starting any additional SIs we need to ensure that the weight for the SI is 0.
1. Run the following command to display the current state of the deployment:
```
mvm-config-manager show-sis --si-type=esc --detailed
```
This returns detailed output on all ESC SIs. An example output is shown below
```
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| Config Version | SI Name | SI Type | AZ | Active | Weight | Thanos Compactor | Global GUIs |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z1 | ESC | 1 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z2 | ESC | 2 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z3 | ESC | 3 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.0+1-0 | e92z1 | ESC | 1 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.0+1-0 | e92z2 | ESC | 2 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.0+1-0 | e92z3 | ESC | 3 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
```
The ESC SIs that are not currently active are the SIs that will be upgraded. These
will be referred to as Uplevel SIs in the rest of this document.
The ESC SIs that are currently active are the SIs that will be replaced. These
will be referred to as Downlevel SIs in the rest of this document.
1. For any uplevel ESC SI that has a non-zero weighting run the following command:
```
ESC_SI_WEIGHT_IDENTIFIER=<ESC_SI_WEIGHT_IDENTIFIER>
mvm-config-manager change-si-weight \
--si-weight-identifier ${ESC_SI_WEIGHT_IDENTIFIER} \
--weight 0
```
### 9.2.4 Apply the latest configuration to the inactive (uplevel) SIs
1. Apply the configuration version to each inactive SI by running the command:
```
mvm-config-manager apply-config \
--si-type ESC
```
This will output a message confirming that the operation was successful, the config
version that was applied and a list of the SIs it was applied to, e.g.:
```
Applied config version 11.5.0+2-2 to SIs with type ESC.
Upgraded SIs:
e92z1
e92z2
e92z3
```
### 9.2.5 Add the SIs to the list of SIs managed by SIMPL
1. Update the SIMPL managed SI configuration by running the following commands
for each SI that you want to start:
```
UPLEVEL_ESC_NAME=<ESC SI to upgrade>
mvm-config-manager deploy-si --si-name ${UPLEVEL_ESC_NAME}
```
This will output a message confirming that the operation was successful e.g.
`Deployed SI ${UPLEVEL_ESC_NAME}.`
Additionally, if the command `mvm-config-manager show-sis --si-type=esc --detailed` is run
then any SI that is queued to start will have its active status changed from `false` to `true`
and its config version updated. For example, if we had deployed `e92z1` then the
corresponding output would be as shown below
```
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| Config Version | SI Name | SI Type | AZ | Active | Weight | Thanos Compactor | Global GUIs |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z1 | ESC | 1 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z2 | ESC | 2 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z3 | ESC | 3 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z1 | ESC | 1 | true | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z2 | ESC | 2 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z3 | ESC | 3 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
### 9.2.6 Enable automatic prioritization for the uplevel ESC SI
This step can be run before the ESC has fully spun up.
***
**This step is run against the configuration git repository in the Deployment region**
***
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_enable_ober
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization in the deployment region@dev.azure.com/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./_git/Name of the configuration Azure DevOps repository in the deployment region
```
- If using ssh to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./Name of the configuration Azure DevOps repository in the deployment region
```
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<DEPLOYMENT_GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${DEPLOYMENT_GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Check the status of the ESC by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | Auto |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | High (10) |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
Only proceed with the rest of this section if the Priority for the ESC is shown as `High (10)`
Using the example above, the ESC SI that we started was `e92z1` and so we
proceed with the rest of the section
1. Configure the ESC to use automatic prioritization by running the command:
```
mvm-config-manager reset-si-priority --si-name <ESC_SI_NAME>
```
1. Verify that the change has been applied by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | Auto |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | Auto |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
You should notice that the priority has changed to Auto for the affected ESC,
in this case `e92z1`
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
### 9.2.7 Monitor the SI creation
When the file is committed and pushed to the SIMPL DevOps repository, the resources
will be automatically pulled into SIMPL, which will trigger a Job to run Terraform
to create the Service Instance.
You can watch progress of this job from the Log Analytics Workspace (LAW) associated with the region
1. Connect to the LAW
- Through the portal, select the resource **Name of the Log Analytics Workspace (LAW) associated with the region**
- Select **Logs** from the menu
- From the resultant **Queries** page
- Enter `SIMPL jobs - Details` in the search box, this will match one query
- Select **Run**. This will load the query into the editor and run the query
The query shows the log entries that SIMPL has written for the apply / delete jobs
with the latest entry displayed first. The container name column contains the name
of the apply job, `apply-si-<SI NAME>`. This can take a few minutes before the first
log entry appears after the merge has been completed.
Sample outputs are shown below. Keep rerunning the query until the log entry
`Apply Complete!` is displayed. At this point the SIMPL job has completed.
If multiple deletes and adds are occurring at the same time then, for clarity, edit
the clause at the end of the query to enter the SI name before selecting Run.
For example, if the SI Name was x01z1 then the modified query would be as shown below
```
let startTimestamp = ago(7d);
(
KubePodInventory
| where TimeGenerated > startTimestamp
| where ContainerName has_cs "apply-si"
or ContainerName has_cs "delete-si"
or ContainerName has_cs "apply-dc"
or ContainerName has_cs "delete-dc"
| distinct ContainerID, ContainerName, ClusterName, ClusterId
)
| join
(
ContainerLog
| where TimeGenerated > startTimestamp
) on ContainerID
| project TimeGenerated, LogEntry, Container = split(ContainerName, "/")[1], ClusterName, ClusterId = split(ClusterId, "/")[4]
| project-rename ResourceGroup=ClusterId
| sort by TimeGenerated desc
// Uncomment this line to pick out the logs for a particular service instance.
| where Container has "x01z1"
```
**Query results for a running apply job**
**Query results for a completed apply job**
### 9.2.8 Enable uplevel Perimeta VM logging to the Log Analytics Workspace (LAW)
These commands are run from the Azure Portal session created in section 9.1
1. Enable Perimeta logging on the uplevel VMs
1. Browse to the DCR `Name of the Data Collection Rule resource (DCR) used by Perimeta VMs to record logs`
1. Click on **Resources** tab
1. Press **Add**
1. Select all of the uplevel Perimeta VMs
1. Select **confirm**
>The Perimeta VM names will be `perimeta-vm-<ESC_SI_NAME>-vm-0`
>through `perimeta-vm-<ESC_SI_NAME>-vm-"<PERIMETA_COUNT>-1`
***
Occasionally not all Perimeta VMs show up when the Resource Group associated with
the Perimeta VMs is expanded. This appears to be a timing-related issue (as they
do appear later).
If this does occur, then check the resource group itself and then hit apply.
All the VMs in the resource group will be added to the DCR
***
### 9.2.9 Verify ESC functionality
1. Follow the [**Test Plan**](#testplan) to verify ESC functionality.
### <a id=testpass></a>9.2.10 Cut over VPIM traffic
At this stage, the uplevel ESC SI is receiving all the TUI and VVM traffic but
no VPIM traffic.
The VPIM DNS MX record is deployment wide and only exists in `variables/deployment-config-v2.yaml`
in the **deployment** region
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_cutover_vpim
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization in the deployment region@dev.azure.com/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./_git/Name of the configuration Azure DevOps repository in the deployment region
```
- If using ssh to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./Name of the configuration Azure DevOps repository in the deployment region
```
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<DEPLOYMENT_GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${DEPLOYMENT_GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Update the DNS MX preference
Edit the file `variables/deployment-config-v2.yaml` and update the vpim DNS MX record
preferences so that the uplevel ESCs are preferred over the downlevel ESCs.
You should set the preference value for the uplevel ESC to the same as the other
active ESCs and set the value for the downlevel ESC to be several orders of
magnitude higher (so traffic should no longer be routed to it by the VPIM application)
1. Commit the change by running the command
```
git commit -a -m "<check in message>"
```
1. Apply the change to SIMPL by running the command:
```
mvm-config-manager upgrade-simpl --config-version ${CONFIG_VERSION}
```
If successful, a message similar to the one below is displayed on the screen
`Applied config version 11.5.0-1-1 to SIMPL cluster simpl-21.3.5`
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
### 9.2.11 Remove downlevel Perimeta VM logging from the Log Analytics Workspace (LAW)
These commands are run from the Azure Portal session created in section 9.1
1. Disable Perimeta logging on the downlevel VMs
1. Browse to the DCR `Name of the Data Collection Rule resource (DCR) used by Perimeta VMs to record logs`
1. Click on **Resources** tab
1. Select all of the downlevel Perimeta VMs
1. Select **delete**
1. Select **confirm**
>The Perimeta VM names will be `perimeta-vm-<DOWNLEVEL_ESC_SI_NAME>-vm-0`
>through `perimeta-vm-<DOWNLEVEL_ESC_SI_NAME>-vm-"<PERIMETA_COUNT>-1`
***
Occasionally not all Perimeta VMs show up when the Resource Group associated with
the Perimeta VMs is expanded. This appears to be a timing-related issue (as they
do appear later).
If this does occur, then check the resource group itself and then hit apply.
All the VMs in the resource group will be removed from the DCR
***
### 9.2.12 Destroy the downlevel ESC SI
At this stage, the downlevel ESC is not receiving any traffic and can be removed
>**This step must be carried out at least 4 hours after the previous step.
>Failure to do so may result in a gap in historical metrics**
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_remove_downlevel_esc
DOWNLEVEL_ESC_SI_NAME=<DOWNLEVEL_ESC_SI_NAME>
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
```
- If using ssh to interact with the git repository
```
GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
```
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Destroy the ESC SI by running the following command:
```
mvm-config-manager destroy-si --si-name ${DOWNLEVEL_ESC_SI_NAME}
```
The command will return output similar to
```
Destroyed SI ${DOWNLEVEL_ESC_SI_NAME}
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
### 9.2.13 Monitor the SI deletion
When the file is committed and pushed to the SIMPL DevOps repository, the resources
will be automatically pulled into SIMPL, which will trigger a Job to run Terraform
to delete the Service Instance.
You can watch progress of this job from the Log Analytics Workspace (LAW) associated with the region
1. Connect to the LAW
- Through the portal, select the resource **Name of the Log Analytics Workspace (LAW) associated with the region**
- Select **Logs** from the menu
- From the resultant **Queries** page
- Enter `SIMPL jobs - Details` in the search box, this will match one query
- Select **Run**. This will load the query into the editor and run the query
The query shows the log entries that SIMPL has written for the apply / delete jobs
with the latest entry displayed first. The container name column contains the name
of the apply job, `destroy-si-<SI NAME>`. This can take a few minutes before the first
log entry appears after the merge has been completed.
Sample outputs are shown below. Keep rerunning the query until the log entry
`Destroy Complete!` is displayed. At this point the SIMPL job has completed.
If multiple deletes and adds are occurring at the same time then, for clarity, edit
the clause at the end of the query to enter the SI name before selecting Run.
For example, if the SI Name was x01z1 then the modified query would be as shown below
```
let startTimestamp = ago(7d);
(
KubePodInventory
| where TimeGenerated > startTimestamp
| where ContainerName has_cs "apply-si"
or ContainerName has_cs "delete-si"
or ContainerName has_cs "apply-dc"
or ContainerName has_cs "delete-dc"
| distinct ContainerID, ContainerName, ClusterName, ClusterId
)
| join
(
ContainerLog
| where TimeGenerated > startTimestamp
) on ContainerID
| project TimeGenerated, LogEntry, Container = split(ContainerName, "/")[1], ClusterName, ClusterId = split(ClusterId, "/")[4]
| project-rename ResourceGroup=ClusterId
| sort by TimeGenerated desc
// Uncomment this line to pick out the logs for a particular service instance.
| where Container has "x01z1"
```
**Query results for a running destroy job**
**Query results for a completed destroy job**
### 9.2.14 Disable automatic prioritization for the downlevel ESC
***
**This step is run against the configuration git repository in the Deployment region**
***
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_disable_ober_monitoring_1
DOWNLEVEL_ESC_SI_NAME=<DOWNLEVEL_ESC_SI_NAME>
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization in the deployment region@dev.azure.com/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./_git/Name of the configuration Azure DevOps repository in the deployment region
```
- If using ssh to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./Name of the configuration Azure DevOps repository in the deployment region
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<DEPLOYMENT_GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${DEPLOYMENT_GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Check the status of the ESC by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | Auto |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | Auto |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
1. Disable automatic prioritization on the deleted ESC by running the command:
```
mvm-config-manager reset-si-priority \
--si-name <DOWNLEVEL_ESC_SI_NAME> \
--priority high
```
1. Verify that the change has been applied by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | High (10) |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | Auto |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
You should notice that the priority has changed to Auto for the affected ESC,
in this case `e91z1`
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
## <a id=testplan></a> 9.3 Test Plan
The testplan makes reference to two ESC SIs:
- **UPLEVEL_ESC_NAME**. The name of the ESC SI that is running the uplevel software
- **DOWNLEVEL_ESC_NAME**. The name of the ESC SI that is running the current, or downlevel, software
### 9.3.1 Functional verification
#### 9.3.1.1 Watch the pods come up
1. Set the following environment variables:
```
UPLEVEL_ESC_NAME=<UPLEVEL_ESC_NAME>
SUBSCRIPTION_ID=Azure subscription identifier for the MVM subscription.
```
1. Verify that all pods start successfully by running the command:
```
az aks command invoke \
--name ${UPLEVEL_ESC_NAME}-k8s \
--resource-group ${UPLEVEL_ESC_NAME}-rg \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get pods -A"
```
You should see pods come up in the following namespaces:
- `connaisseur`
- `csi-driver`
- `gitops`
- `istio-system`
- `kube-system`
- `mvm-health`
- `mvm`
- `platform`
- `simon`
If any of the pods enter a failed state (anything other than `Init`,
`PodInitializing` or `Running`), see the Troubleshooting section of the
*Deployment Guide* for troubleshooting guidance.
If any pods get stuck in the `Init` state, and
```
az aks command invoke \
--name ${UPLEVEL_ESC_NAME}-k8s \
--resource-group ${UPLEVEL_ESC_NAME}-rg \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl describe pod -n <namespace of initializing pod> <name of initializing pod>"
```
reports errors of the form `AADSTS700213: No matching federated identity record found for presented assertion subject`,
there was an error with Entra ID initialization for the cluster.
Follow the backout procedure to destroy the service instance, then follow the MOP again to re-create the service instance.
Depending on your Azure configuration, you might be prompted to re-login to
Azure (including MFA) the first time the `kubectl` command executes. This is
normal and to be expected
#### 9.3.1.2 Verify that the SI is being monitored by SIMon
1. Log into Grafana
Use the global URL `https://grafana-global.Name of the global DNS zone`
1. Select the **Azure Kubernetes Service health monitoring** dashboard
Verify that the SI **<UPLEVEL_ESC_NAME>** appears in the list of SIs in the Overview Health panel
#### 9.3.1.3 Verify there are no alerts associated with the SI
1. Log into Alerta
Use the global URL `https://alerta-global.Name of the global DNS zone`
Alerta will display any alerts raised in your deployment. There may be an
Azure Entra login prompt before Alerta can be accessed
1. Verify that there are no active alerts associated with **<UPLEVEL_ESC_NAME>**
#### 9.3.1.4 Configure and verify mvmmailbox works against the uplevel ESC
1. Connect to a tooling VM in the same region as the esc
1. Edit the file `/etc/hosts`
Add the following lines to the end of the file
```
<ESC_INGRESS_IP_ADDRESS> internal-prov.The short (4-characters maximum) DNS label for the region.Name of the global DNS zone
<ESC_INGRESS_IP_ADDRESS> internal-message-access.The short (4-characters maximum) DNS label for the region.Name of the global DNS zone
```
Save the file
1. Set the following environment variables:
```
export MSW_TRUSTED_CA_BUNDLE=The full path to a .pem file on the tooling VM containing the Root CA certificate used to sign the DOMAIN_TLS_CERT, which must be trusted by users of MVM APIs. This file must contain a single CA certificate. Multiple CA certs are not supported.
export MSW_MVMMBX_MPG_POOL="https://internal-prov.The short (4-characters maximum) DNS label for the region.Name of the global DNS zone"
export MSW_MVMMBX_RETRIEVAL_POOL="https://internal-message-access.The short (4-characters maximum) DNS label for the region.Name of the global DNS zone"
export MSW_AZURE_CLIENT_ID="The Application (client) ID of the registered Enterprise application for MVM"
export MSW_AZURE_DISCOVERY_URL="https://login.microsoftonline.com/Azure tenant identifier/v2.0"
export MSW_AZURE_API_URI=The application URI for this MVM deployment
```
This allows `mvmmailbox` to be run without specifying these parameters
1. Authorize `mvmmailbox` by running the following command:
```
mvmmailbox azure login
```
`mvmmailbox` will ask you to visit a Microsoft URL and enter a code, e.g.
```
$ mvmmailbox azure login
Open this URL in your browser:
https://microsoft.com/devicelogin
and enter the code: EUYWZWGCU
```
On visiting this URL, entering the code, and signing in, the tool will be signed in:
```
Successfully authorized
Credentials saved to "~/.config/mvmmailbox/.azurecredentials"
```
Export the credentials file to the environmental variable **MSW_AZURE_CREDENTIALS_PATH** by running the command:
```
MSW_AZURE_CREDENTIALS_PATH=<file from above>
```
Using the output above, this would be:
```
MSW_AZURE_CREDENTIALS_PATH="~/.config/mvmmailbox/.azurecredentials"
```
1. Verify the Azure credentials by running the following command
```
mvmmailbox azure check
```
This will return the following information
```
Client ID: MVM_AUTH_CLIENT_ID
Access token:
<JSON web token>
Refresh token:
<JSON web token>
Access token expires after: <time and date> (local: <time and date>)
Application Roles:
- metaswitch.mvm.mailbox.read
- metaswitch.subtopology.read
- ...
Displayed "<path to credentials file>"
```
Verify that the following information is correct:
- The Application roles returned should match the roles expected for your user type in Reference application roles on the Deployment Guide. If they do not, your user may not have been correctly assigned the right application roles, or the correct roles may not be present on the AD Application representing the deployment.
- Decode the returned access token (not the refresh token, which should be opaque to the user) using a tool such as [jwt](https://jwt.ms) - this will then enable you to check the following.
- The `iss` (issuer) field should match the `discover_doc_url` set in service configuration. If it does not, it is possible that your AD Application is misconfigured to use the wrong version of the access token API- ensure that the `accessTokenAcceptedVersion` in your application manifest is set to 2 (if not set, it will default to version 1).
- The `exp` (expiry) field should show a Unix timestamp later than the current timestamp at the time of sending the request. `mvmmailbox` should be handling refreshing the token - if this is not correct, contact Microsoft Support.
- The `aud` (audience) field should show an application ID that matches the `application_id` set in service configuration. If it does not, your user may have app roles configured against an incorrect application
1. Run the following command to check connectivity to the provision and retrieval APIs
```
mvmmailbox check connection
```
This command will attempt to connect to the provision and access APIs using the
cached Azure credentials. A successful connection test prints the following:
```
Successfully authorized against provisioning API
Successfully authorized against access API
Connection test complete
```
On failure, an error message is displayed showing which API or APIs authorization
has failed against. In this case, the first step to debugging this should be to re-run
the command as `mvmmailbox -vvv check connection` to show debug logs, including the
response from the underlying APIs. These responses should contain information
about what is incorrect about an authentication request.
If a 401 error is received, this indicates that the wrong authentication type may have been configured on MVM - check that the authentication section of the solution configuration is set up correctly, and ensure that you are not attempting to use any other authentication method (e.g. Radius) with mvmmailbox.
If a 403 error is received, this should also contain information about which field on the token used for authorization is incorrect (though not what the received or expected values are). Re-run mvmmailbox azure check, and double check the fields above.
The connection test can be performed by any user with the `metaswitch.mvm.mailbox.read` and the `metaswitch.subtopology.read` roles.
1. Retrieve details of a known mailbox by running the following command:
```
mvmmailbox show summary tel:+1xxxxxxxxxx
```
See Appendix A of MVM Subscriber Administration Guide for further information
on using mvmmailbox and sample output
1. Edit `/etc/hosts` and remove the two lines added above
#### 9.3.1.5 Verify VVM function
IMAP VVM functionality is available on the load balancer ports specified in the *MVM on Azure Solution Architecture*.
This can be initially tested by setting up a TLS session to **ESC_INGRESS_MT_INTERNET_EXT_IP_ADDRESS**
and the external port for the VVM type you plan to test
1. Set up a TLS session:
Your session will need to use either implicit TLS or STARTTLS encryption, or else
MVM will reject attempts to log in.
To start an implicit TLS session run the command:
```
openssl s_client -host $ESC_INGRESS_MT_INTERNET_EXT_IP_ADDRESS -port $IMAPS_PORT
```
To start a STARTTLS session run the command:
```
openssl s_client -host $ESC_INGRESS_MT_INTERNET_EXT_IP_ADDRESS -port $IMAP_PORT -starttls imap
```
2. Verify VVM function by running IMAP commands, for example:
```
001 LOGIN [SUBSCRIBER] [PIN]
002 SELECT "INBOX"
003 FETCH 1:1 (FLAGS UID BODY.PEEK[HEADER.FIELDS (Date From)])
004 FETCH 1 (BODY.PEEK[1])
005 LOGOUT
```
Note that you will need to have first configured a subscriber with VVM enabled,
both in the class of service and for that subscriber. This testing is limited
to the Custom IMAP interface, as the TUI PIN can be used as a fallback credential
for Custom IMAP (but not NiVVM).
#### 9.3.1.6 Verify that the ESC is processing synthetic traffic
ESC SIs generate synthetic calls that allow for monitoring of their health without
needing to run live traffic through them.
Before canarying any live traffic onto the ESC SI, you should verify that the synthetic
traffic is being handled successfully
1. Log into Grafana
Use the global URL `https://grafana-global.Name of the global DNS zone`
1. Select the **Outcome Based Routing** dashboard
1. Wait until the **Assessed Priority Tier** in the **ESCs by Automatic Prioritization Enabled & Priority Tier**
shows `High`
In the screenshot below, the ESC e02z1 is ready to canary traffic.
Follow the test plan [**Canarying Traffic**](#canary)
#### 9.3.1.7 Verify the ESC is logging to SAS
1. Log into SAS
Use the Regional URL `https://sas.The short (4-characters maximum) DNS label for the region.Name of the global DNS zone`
1. Find an event in SAS
MVM operations can be searched for using the telephone number of subscribers
involved, SIP URI or Call IDs.
Verify that the SAS trace looks ok
### <a id=canary></a> 9.3.2 Canarying Traffic
In this section of the test plan we move ESC traffic from the downlevel ESC,
**DOWNLEVEL_ESC_NAME**, to the uplevel ESC, **UPLEVEL_ESC_NAME**.
The aim of this procedure is to incrementally increase the weight of the uplevel SI
and decrease the weight of the downlevel SI until the uplevel SI has a weight
of `1000` and the downlevel SI has a weight of `0`. In between each incremental
increase we verify that the uplevel SI is functioning as expected.
***
**If at any point verification reveals that the uplevel SI is not functioning as
expected, back out of the procedure by immediately moving traffic back to the
downlevel SI. Once the problem with the uplevel SI is resolved, start again from
the first step in Verify SAS function**
***
***
**This procedure *may* require updates to the git configuration repository in two regions.**
Updates are always required to the configuration in the deployment region.
If the ESC is not in the deployment region, then the corresponding updates are
also required in the region that the ESC is located in.
***
#### 9.3.2.1 Move traffic to the uplevel ESC (deployment region)
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_move_sip_traffic_1
DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER=<DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER>
UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER=<UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER>
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization in the deployment region@dev.azure.com/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./_git/Name of the configuration Azure DevOps repository in the deployment region
```
- If using ssh to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./Name of the configuration Azure DevOps repository in the deployment region
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<DEPLOYMENT_GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${DEPLOYMENT_GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Move a small amount of sip traffic between the ESC SIs by updating the weighting for the SIs
Initially the current weights on these SIs are "0" and "1000" (for the uplevel
and downlevel SIs, respectively).
We recommend starting by moving 1% of traffic by updating the weights to be
`10` and `990`, respectively. This is achieved by running the following commands:
```
mvm-config-manager change-si-weight \
--si-weight-identifier ${UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER} \
--weight 10
mvm-config-manager change-si-weight \
--si-weight-identifier ${DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER} \
--weight 990
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
#### 9.3.2.2 Move traffic to the uplevel ESC (non-deployment region)
***
This is an optional step that is only required if the ESC SI is not located in the
deployment region
***
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_move_sip_traffic_2
DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER=<DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER>
DOWNLEVEL_WEIGHT_VALUE='value of weight applied to the downlevel ESC in the previous section`
UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER=<UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER>
UPLEVEL_WEIGHT_VALUE='value of weight applied to the uplevel ESC in the previous section`
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
```
- If using ssh to interact with the git repository
```
GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
```
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Apply the same changes to the ESC SI weights as was done in the deployment region
```
mvm-config-manager change-si-weight \
--si-weight-identifier ${UPLEVEL_ESC_SI_WEIGHT_IDENTIFIER} \
--weight ${UPLEVEL_WEIGHT_VALUE}
mvm-config-manager change-si-weight \
--si-weight-identifier ${DOWNLEVEL_ESC_SI_WEIGHT_IDENTIFIER} \
--weight {DOWNLEVEL_WEIGHT_VALUE}
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
#### 9.3.2.3 Verify the uplevel ESC is handling calls and not raising alarms
1. Place some TUI test calls
1. Log into Grafana
Use the global URL `https://grafana-global.Name of the global DNS zone`
1. Select the **ESC Upgrade** dashboard
You should see calls and VVM traffic begin to be handled by the up-level ESC SI.
The fraction of traffic handled by the up-level ESC should match the weighting you
set.
1. Log into Alerta
Use the global URL `https://alerta-global.Name of the global DNS zone`
Alerta will display any alerts raised in your deployment. There may be an
Azure Entra login prompt before Alerta can be accessed
Verify that there are no active alerts associated with **<UPLEVEL_ESC_NAME>**
#### 9.3.2.4 Move more traffic
Repeat the [**Canarying Traffic**](#canary) testplan moving increasing increments of traffic from the downlevel SI to the uplevel SI.
The recommend weightings are `5% (50), 25% (250), 50% (500), 75% (750)` and finally `100% (1000)`
for the uplevel SI.
### 9.3.3 Proceed with the upgrade
<---------------- PRE-RENDER START --------------->
https://dev.azure.com/mvmprodeus2/MVM/_git/documentation?path=/Labs-ANTS-DevOps/SMOPs/Keystone-Mops/GUI-Bastion-access-update.md&_a=preview
<---------------- PRE-RENDER END --------------->
1. If the test plan succeeded, proceed to [**Cut over VPIM traffic**](#testpass) to continue the upgrade process, otherwise proceed to [**Backout**](#backout)
## <a id=backout></a> 9.4 Backout Procedure
The backout process is to delete the ESC Service Instance that was just created.
### 9.4.1 Set the default subscription to the MVM subscription
1. Set the default subscription by running the command:
```
az account set --subscription "Azure subscription identifier for the MVM subscription."
```
### 9.4.2 Prepare the configuration git repository
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_destroy_esc
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization.@dev.azure.com/Name of the Azure DevOps organization./Name of the Azure DevOps project./_git/Name of the configuration Azure DevOps repository.
```
- If using ssh to interact with the git repository
```
GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization./Name of the Azure DevOps project./Name of the configuration Azure DevOps repository.
```
1. Change to an appropriate working directory in Cloud shell. Your git repository will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<GIT_CONFIGURATION_URL>**. The repository can be cloned using either ssh or https. In both cases you will run the following command:
```
git clone ${GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
### 9.4.3 Delete the ESC SI
1. Run the following command to display the current state of the deployment:
```
mvm-config-manager show-sis --si-type=esc --detailed
```
This returns detailed output on all ESC SIs. An example output is shown below
```
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| Config Version | SI Name | SI Type | AZ | Active | Weight | Thanos Compactor | Global GUIs |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z1 | ESC | 1 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z2 | ESC | 2 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.4.1+2-0 | e91z3 | ESC | 3 | true | 1000 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z1 | ESC | 1 | true | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z2 | ESC | 2 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
| 11.5.0+2-2 | e92z3 | ESC | 3 | false | 0 | | |
+----------------+---------+---------+----+--------+--------+------------------+-------------+
```
This shows that we have two ESC SIs running in AZ1, e91z1 (the downlevel SI)
and e92z1 (the uplevel SI).
We want to destroy the uplevel SI, e92z1.
1. Destroy the uplevel SI by running the following command:
```
mvm-config-manager destroy-si --si-name <UPLEVEL_ESC_NAME>
```
In the example above, `UPLEVEL_ESC_NAME` would be set to e92z1
The command will return output similar to
```
Destroyed SI e92z1
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
### 9.4.4 Monitor the SI deletion
When the file is committed and pushed to the SIMPL DevOps repository, the resources
will be automatically pulled into SIMPL, which will trigger a Job to run Terraform
to delete the Service Instance.
You can watch progress of this job from the Log Analytics Workspace (LAW) associated with the region
1. Connect to the LAW
- Through the portal, select the resource **Name of the Log Analytics Workspace (LAW) associated with the region**
- Select **Logs** from the menu
- From the resultant **Queries** page
- Enter `SIMPL jobs - Details` in the search box, this will match one query
- Select **Run**. This will load the query into the editor and run the query
The query shows the log entries that SIMPL has written for the apply / delete jobs
with the latest entry displayed first. The container name column contains the name
of the apply job, `destroy-si-<SI NAME>`. This can take a few minutes before the first
log entry appears after the merge has been completed.
Sample outputs are shown below. Keep rerunning the query until the log entry
`Destroy Complete!` is displayed. At this point the SIMPL job has completed.
If multiple deletes and adds are occurring at the same time then, for clarity, edit
the clause at the end of the query to enter the SI name before selecting Run.
For example, if the SI Name was x01z1 then the modified query would be as shown below
```
let startTimestamp = ago(7d);
(
KubePodInventory
| where TimeGenerated > startTimestamp
| where ContainerName has_cs "apply-si"
or ContainerName has_cs "delete-si"
or ContainerName has_cs "apply-dc"
or ContainerName has_cs "delete-dc"
| distinct ContainerID, ContainerName, ClusterName, ClusterId
)
| join
(
ContainerLog
| where TimeGenerated > startTimestamp
) on ContainerID
| project TimeGenerated, LogEntry, Container = split(ContainerName, "/")[1], ClusterName, ClusterId = split(ClusterId, "/")[4]
| project-rename ResourceGroup=ClusterId
| sort by TimeGenerated desc
// Uncomment this line to pick out the logs for a particular service instance.
| where Container has "x01z1"
```
**Query results for a running destroy job**
**Query results for a completed destroy job**
### 9.4.5 Disable automatic prioritization for the uplevel ESC
This is an optional step that only needs to be executed if automatic prioritization
was enabled during the upgrade process.
***
**This step is run against the configuration git repository in the Deployment region**
***
1. Set the following environment variables:
```
BRANCH=Change ID, used as the prefix for any git branch created in the MOPs_The version number of the uplevel release, e.g. 11.5.0+1_disable_ober_monitoring_2
UPLEVEL_ESC_SI_NAME=<UPLEVEL_ESC_SI_NAME>
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=https://Name of the Azure DevOps organization in the deployment region@dev.azure.com/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./_git/Name of the configuration Azure DevOps repository in the deployment region
```
- If using ssh to interact with the git repository
```
DEPLOYMENT_GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/Name of the Azure DevOps organization in the deployment region/Name of the Azure DevOps project in the deployment region./Name of the configuration Azure DevOps repository in the deployment region
1. Change to an appropriate working directory in Cloud shell. Your git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps git repository with **<DEPLOYMENT_GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${DEPLOYMENT_GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Check the status of the ESC by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | Auto |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | Auto |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
1. Disable automatic prioritization on the deleted ESC by running the command:
```
mvm-config-manager reset-si-priority \
--si-name ${UPLEVEL_ESC_SI_NAME} \
--priority high
```
1. Verify that the change has been applied by running the command:
```
mvm-config-manager show-si-priorities
```
A sample output is shown below.
```
+---------+-----------+
| SI Name | Priority |
+---------+-----------+
| e01z1 | Auto |
+---------+-----------+
| e01z2 | Auto |
+---------+-----------+
| e01z3 | Auto |
+---------+-----------+
| e02z1 | High (10) |
+---------+-----------+
| e02z2 | High (10) |
+---------+-----------+
| e02z3 | High (10) |
+---------+-----------+
| e91z1 | Auto |
+---------+-----------+
| e91z2 | Auto |
+---------+-----------+
| e91z3 | Auto |
+---------+-----------+
| e92z1 | High (10) |
+---------+-----------+
| e92z2 | High (10) |
+---------+-----------+
| e92z3 | High (10) |
+---------+-----------+
```
You should notice that the priority has changed to Auto for the affected ESC,
in this case `e921z1`
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
# 10. Post checks
[System healthchecks]
# 11. Risk Assessment Score
1 - TBD
# 12. Execute MOP clean up if required
# 13. End of Document MOP
# 14. Service Assurance/Monitoring
# A. Appendix and Tables
# B. Approvers
# C. Peer Reviewers
# D. References for Other Documents
# E. Additional Appendices (If required)