r1154learotatenotarykeys.j2
markdown_lea
templates/notary_rotation_lea/r1154learotatenotarykeys.j2
Jinja2 Template
1083 lines
|**Metadata**|**Description** |
|--|--|
|Doc Title| MVM v3 LEA: Rotate Notary Keys |
|Navigation|[WIKI Home Page](https://dev.azure.com/mvmprodeus2/MVM/_wiki/wikis/documentation/1/documents-home#)|
|Tracking| Document Number: VPE-5512-536 |
|Author| Graeme Thomson (gt163y) |
| Agreement Number | 24252.S.005 |
***
**Notices**
Copyright © 2025 Metaswitch Networks. All rights reserved.
This manual is Confidential Information of Metaswitch Networks subject to the confidentiality terms
of the Agreement 01019223 as amended between AT&T and Metaswitch Networks.
It is issued on the understanding that no part of the product code or documentation (including this manual)
will be copied or distributed without prior agreement in writing from Metaswitch Networks and Alianza, Inc.
Metaswitch Networks and Alianza reserve the right to, without notice, modify or revise all or part of
this document and/or change product features or specifications and shall not be responsible for any
loss, cost, or damage, including consequential damage, caused by reliance on these materials.
Metaswitch and the Metaswitch logo are trademarks of Metaswitch Networks. Other brands and
products referenced herein are the trademarks or registered trademarks of their respective holders.
Product(s) and features documented in this manual handle various forms of data relating to your
users. You must comply with all laws and regulations applicable to your deployment, management,
and use of said product(s), and you should take all appropriate technical and organizational
measures to ensure you are handling this data appropriately according to any local legal and
regulatory obligations.
You are responsible for determining whether said product(s) or feature(s) is/are appropriate for
storage and processing of information subject to any specific law or regulation and for using said
product(s) or feature(s) in a manner consistent with your own legal and regulatory obligations. You
are also responsible for responding to any request from a third party regarding your use of said
product(s), such as a request to take down content under the U.S. Digital Millennium Copyright Act
or other applicable laws.
Metaswitch Networks
399 Main Street
Los Altos
CA 94022
<http://www.metaswitch.com>
***
***Table of Contents***
[[_TOC_]]
# 1. Document History
| **Issue** | **Issue Date** | **Author(s)** | **Identification** **of** **Changes** |
|-|-|-|-|
| 1| 12/12/2024| Gthomson| initial draft |
# 2. Versions
| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson| initial draft |
# 3. Integrated Solution Approach v1 (ISA v1)
| **Version #** | **Editor** | **Comments** |
|-|-|-|
| 1| Gthomson| initial draft |
# 4. MOP Impact Scope / General Information
## 4.1 Description
The notary keys used for signing Docker images do not expire but it is possible to rotate them
if they are accidentally lost or exposed. If any of the ACR keys, signatures or images have
been compromised, or there is an issue with the keys or signatures, then all of the keys,
signatures and images must be removed from the ACR and recreated.
This MOP describes the process to remove the old notary keys from the deployment, and create
and apply new ones.
## 4.2 Site Specific Description
| **Originator** | **Date** | **Time** |
|-|-|-|
| **Deployment Location(s)** | |
| **Description** | This MOP applies to the MVM V3 on Azure deployment, Release R11.5.4 | |
## 4.3 Service Impact
Service impact is not expected during this procedure.
## 4.4 Coordination
This MOP has no interactions outside of the MVM subscription.
# 5. Prerequisite/Dependencies/Entrance Criteria of MOP
This is a standalone MOP that describes the process to update the notary rotation keys.
## 5.1 Required parameters
The following parameter values are required to run this MOP
| **Identifier** | **Description** |
|-|-|
| **ACR_NAME** | The name of the Azure Container Registry for this region. |
| **ACR_NOTARY_KEY_VAULT_NAME** | Name of the current key vault where the notary keys are stored. This will be changed during the MOP. |
| **AZURE_REGION** | The Azure region to create resources in. |
| **CHANGE_ID** | Change ID, used as the prefix for any git branch created in the MOPs. |
| **CONFIG_VERSION** | The name of the config set directory containing the current SI configuration in the configuration repository. Config sets are located in the `config` directory |
| **GIT_AUTOMATION_REPOSITORY** | Name of the automation Azure DevOps repository, containing the pipeline definitions. |
| **GIT_AUTOMATION_URL** | URL of the automation git repository. |
| **GIT_CONFIGURATION_REPOSITORY** | Name of the configuration Azure DevOps repository. |
| **GIT_CONFIGURATION_URL** | URL of the configuration git repository. |
| **GIT_PASSWORD** | Password used to access the Azure DevOps repositories if you are using https to manage the local copy of the access the repository. |
| **KEY_VAULT_NAME** | Name of the Key Vault where secrets and certificates are stored for a single region. |
| **KEY_VAULT_RG** | Name of the resource group that contains the key vault **KEY_VAULT_NAME** |
| **MVM_FILESHARE** | Name of the fileshare containing the MVM release. |
| **NOTARY_KEY_VAULT_1_NAME** | The name of the first notary key vault. Set on the first run through this MOP. |
| **NOTARY_KEY_VAULT_2_NAME** | The name of the second notary key vault. Set on the first run through this MOP. |
| **ORGANIZATION_NAME** | Name of the Azure DevOps organization. |
| **PIPELINE_CONFIGURATION_NAME** | Name of the pipeline configuration file for the region without the .yml suffix (e.g. `vars_eus2_11500`). |
| **PROJECT** | Name of the Azure DevOps project. |
| **REGIONAL_PIPELINE_KEY_VAULT_NAME** | Name of the regional pipeline key vault |
| **REGIONAL_PIPELINE_KEY_VAULT_RG** | Name of the resource group that contains the key vault **REGIONAL_PIPELINE_KEY_VAULT_NAME** |
| **REGION_AZURE_TAGS** | A set of string key/value pairs, used to tag region-level resources created in the main subscription in Azure. |
| **REGION_SHORTNAME** | The short (4-characters maximum) DNS label for the region |
| **SIMPL_SERVICE_NAME** | An identifier for the SIMPL instance |
| **SUBSCRIPTION_ID** | Azure subscription identifier. |
## 5.2 Required files
No additional files are required to run this MOP.
# 6. Assumptions
The target audience for this procedure is the AT&T Engineer who will be performing the task. They will need to be familiar with Azure and have a working knowledge of the Azure CLI and Linux.
# 7. Material Requirements
## 7.1 Required Documents
## 7.2 Tools
| **Tool** | **Description** | **Quantity** |
|-|-|-|
| Laptop or Desktop PC | PC With at least 1G Memory and a network communications software application such as Procomm, Reflections or PuTTY | 1 |
| Azure connectivity PC | CloudShell Connectivity is required to the azure subscription. This can be accessed via [My Dashboard - Microsoft Azure](https://portal.azure.com/#cloudshell/) | |
# 8. Pre Maintenance Check, Precautions and Preparations
## 8.1 Precautions and Preparation
## 8.2 Precautions
> This procedure may cause a partial outage during implementation. Use executable script files to minimize down time and typing errors. Familiarize yourself with back-out procedures prior to starting the procedure.
| **Ask Yourself Principle** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Do I have the proper ID and appropriate building access permissions for the environment I am about to enter? | | |
| 2. Do I know why I'm doing this work? | | |
| 3. Have I identified and notified everybody - customers and internal groups - who will be directly affected by this work? | | |
| 4. Can I prevent or control service interruption? | | |
| 5. Is this the right time to do this work? | | |
| 6. Am I trained and qualified to do this work? | | |
| 7. Are the work orders, MOPs, and supporting documentation current and error-free? | | |
| 8. Do I have everything I need to quickly and safely restore service if something goes wrong? | | |
| 9. Have I walked through the procedure? | | |
| 10. Have I made sure the procedure includes proper closure including obtaining clearance and release for the appropriate work center? | | |
| **E911 Ask Yourself** | **Yes** | **No** | **N/A** |
|-|-|-|-|
| 1. Does this work impact E911? | | |
| 2. Do I know how this work could impact 911/e911? | | |
| 3. Do I know what 911/e911 phase is required? | | |
| 4. Have I identified potential risks to 911/e911 and taken all measures to minimize? | | |
| 5. Does this work affect 15+ sites? | | |
| 6. Can I prevent or control service Interruptions to 911/e911? | | |
| 7. Is this the right time to do the work? | | |
| 8. Is the individual performing the work trained and qualified to do this work? | | |
| 9. Are MOPs and supporting documents current and error free? | | |
| 10. Does the MOP include a 911/e911 test plan? | | |
## 8.3 Pre-Maintenance Check Tools/System
Tier 2 needs to identify which tools they will use. This doesn't necessarily need to be included in the MOP but OPS needs to know which tools they will run.
(NEED TO USE STANDARD TOOLS) TIER 2
## 8.4 Pre-Maintenance Check Manual (Non-Automated Requirements)
These will be identify by the tier 3 MOP developer were required.
(MANDATORY CHECK REQUIRE FOR THE MOP) TIER 3
## 8.5 MOP Certification Environment
Examples: PSL certified. OR This MOP was paper certified by ATS engineers.
## 8.6 ATS Bulletin
**ATS Bulletin Check**
| **Step** | **Action** | **Results/Description** | **Timeline** |
|-|-|-|-|
| 1. | No Applicable bulletins | | |
## 8.7 Emergency Contacts
The following emergency contact numbers are to be used in the event provisioning support is required.
In the event a service interruption is encountered the AT&T Implementation Engineer will:
- Cease all work immediately.
- Notify the AT&T Voicemail TRC.
- Escalate to the next level of support.
| **Organization** | **Contact Name** | **Contact Number** |
|-|-|-|
| Voicemail TRC | SANRC | 877-662-7674, opt 3 |
# 9. Implementation
## 9.1 Preliminary Implementation
Pre-check tasks are completed the night of the cutover at least one hour prior to cutover activities.
1. Connect to the DevOps Portal
1. Start a browser session to <https://dev.azure.com/>. This will be required to manage the pipelines
1. Select the project associated with MVM v3
1. Connect to the Azure Portal
1. Start a browser session to <https://portal.azure.com/>. This will be required to manage Azure resources
and access the log analytics workspace (LAW)
1. If prompted, complete the log in process
1. Connect to Azure Cloud Shell
1. Start a CloudShell session by connecting a browser to <https://shell.azure.com/>
1. If the menu at the top left indicates PowerShell select Bash from the menu and confirm at the prompt
1. Upload any files and directories outlined in section 5.2 to your Cloud Shell account as they will be needed later
## 9.2 Implementation
The first time this MOP is run on a region, the keys will begin stored in the main regional key
vault, but it will be necessary to create a new Key Vault to store the refreshed secrets in.
This is because Key Vault deletion and purge policies won't allow us to overwrite keys in the
same vault.
Section 9.2.1 should only be run the first time this MOP is run for a region. It covers the
creation of two new region-scoped keyvaults, so that any subsequent runs of the MOP on the
same region can switch back and forth between them. In the unlikely event that you need to
rotate these secrets a second time within the Key Vault soft deletion retention period (90
days) you may need to run this part of the MOP again to create *another* new Key Vault.
Section 9.2.2 covers deletion, regeneration and application of notary keys, and should be
run every time.
### 9.2.1 First Time Setup (per-region)
If this MOP has been run before for this region, skip this step and go to 9.2.2.
An easy way to tell: look at parameter **ACR_NOTARY_KEY_VAULT_NAME**. If this is the same
as parameter **KEY_VAULT_NAME**, this MOP has *not* been run before and this section *will* be
necessary.
> Note: Some steps of this section require permissions that are typically only granted to
Azure administrators. If you do not have permission to assign RBAC roles, you will need to
contact someone with the Owner or Role Based Access Control Administrator role on the
subscription to help you.
>
> One step also requires permissions to edit ADO Git repository metadata: specifically a
variable group attached to the automation repository.
>
> There are more required permissions listed at the start of section 9.2.2. Consider
asking for these at the same time.
1. Set the following environment variables:
```
AZURE_REGION={{ AZURE_REGION | default('<AZURE_REGION>') }}
SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
REGION_AZURE_TAGS={{ REGION_AZURE_TAGS | default('<REGION_AZURE_TAGS>') }}
REGION_SHORTNAME={{ REGION_SHORTNAME | default('<REGION_SHORTNAME>') }}
KEY_VAULT_RG={{ KEY_VAULT_RG | default('<KEY_VAULT_RG>') }}
NOTARY_KEY_VAULT_1_NAME=mvm-$REGION_SHORTNAME-ntry-kv-1
NOTARY_KEY_VAULT_2_NAME=mvm-$REGION_SHORTNAME-ntry-kv-2
```
Record the values of NOTARY_KEY_VAULT_1_NAME and NOTARY_KEY_VAULT_2_NAME for reference in
future runs of this MOP.
1. Set the default subscription by running the command:
```
az account set --subscription "${SUBSCRIPTION_ID}"
```
1. Create the two new Azure Key Vaults in the resource group, including REGION_AZURE_TAGS
as tags. This command also enables purge protection for the vault, a critical safety
feature to protect your Key Vault from accidental or malicious deletion:
```
az keyvault create --name "$NOTARY_KEY_VAULT_1_NAME" \
--resource-group "$KEY_VAULT_RG" \
--location "$AZURE_REGION" \
--enable-purge-protection true \
--default-action Deny \
--tags $REGION_AZURE_TAGS
az keyvault create --name "$NOTARY_KEY_VAULT_2_NAME" \
--resource-group "$KEY_VAULT_RG" \
--location "$AZURE_REGION" \
--enable-purge-protection true \
--default-action Deny \
--tags $REGION_AZURE_TAGS
```
1. Add the Azure DevOps IP addresses. The addresses that Azure DevOps uses to access other
Azure resources are documented in
https://docs.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops&tabs=IP-V4#inbound-connections
> Attention: If your MVM deployment is NOT in the United States, then you need to find
the required IP addresses for your regions from the link above.
You need to add all of the addresses associated with the United States region to ensure
the Key Vaults can be accessed from Azure DevOps.
As of March 2024, the IP addresses are:
- 20.37.158.0/23
- 52.150.138.0/24
- 20.42.5.0/24
- 20.41.6.0/23
- 40.80.187.0/24
- 40.119.10.0/24
- 40.82.252.0/24
- 20.42.134.0/23
- 20.125.155.0/24
Add the relevant IP addresses by running the following commands:
```
export IP_LIST="20.37.158.0/23 \
52.150.138.0/24 \
20.42.5.0/24 \
20.41.6.0/23 \
40.80.187.0/24 \
40.119.10.0/24 \
40.82.252.0/24 \
20.42.134.0/23 \
20.125.155.0/24"
for IP in $IP_LIST; do \
az keyvault network-rule add \
--name "${NOTARY_KEY_VAULT_1_NAME}" \
--ip-address "${IP}" ; \
az keyvault network-rule add \
--name "${NOTARY_KEY_VAULT_2_NAME}" \
--ip-address "${IP}" ; \
done
```
1. Run the following commands to verify your changes:
```
az keyvault network-rule list \
--name $NOTARY_KEY_VAULT_1_NAME
az keyvault network-rule list \
--name $NOTARY_KEY_VAULT_2_NAME
```
The output of each should resemble the fragment below:
```
{
"bypass": "AzureServices",
"defaultAction": "Deny",
"ipRules": [
{
"value": "52.150.138.0/24"
},
{
"value": "20.37.158.0/23"
},
{
"value": "20.41.6.0/23"
},
{
"value": "20.42.134.0/23"
},
{
"value": "20.125.155.0/24"
},
{
"value": "20.42.5.0/24"
},
{
"value": "40.119.10.0/24"
},
{
"value": "40.80.187.0/24"
},
{
"value": "40.82.252.0/24"
}
],
"virtualNetworkRules": []
}
```
1. **(This step may need help from an administrator of the Automation Git repository)**
Some keys will need their names removing from the regional sensitive variable group
in the automation repository.
1. Select and expand the Pipelines menu for this project.
2. Select Library.
3. Select "mvm-{{ REGION_SHORTNAME | default('<REGION_SHORTNAME>') }}-vault".
4. Delete only the variables:
- root-prv
- root-pub
- root-passphrase
- repository-passphrase
- For the previous and current release (e.g. 11-3-0 and 11-3-1):
- RELEASE-pub (e.g. 11-3-0-pub and 11-3-1-pub)
- RELEASE-prv (e.g. 11-3-0-prv and 11-3-1-prv)
5. Select Save.
1. **(This step may need help from an administrator of the Azure subscription)**
The Service Principal that the pipelines use to access the Key Vaults must be configured
with the "Key Vault Secrets Officer" role to permit access to read and create secrets.
> Note: This part of the MOP requires permission to assign RBAC roles, and can only be
completed by someone with the Owner or Role Based Access Control Administrator role for
the Azure subscription.
1. **(This step may need help from an administrator of the Azure subscription)**
In addition to the permissions outlined in the next section, this first time of
running the MOP you'll need to be given Delete permissions on the main Region Keyvault
({{ KEY_VAULT_NAME | default('<KEY_VAULT_NAME>') }}) so that you
can remove the old secrets from there.
> Note: this key vault wasn't created using RBAC so assignment of a Key Vault Secrets
role will likely have no effect. Explicit Delete permissions must be assigned in the
Access Policies page of this vault. They can be revoked when the MOP is complete,
since subsequent runs will not require this permission.
### 9.2.2 Notary Key Rotation (per-region)
To perform this part of the MOP you will need the roles Contributor and AcrImageSigner
on the Container Registry {{ ACR_NAME | default('<ACR_NAME>') }}.
The Contributor role has broad permissions and should only be granted at the Container
Registry level. You will also need the roles Key Vault Secrets Officer, Key Vault
Certificates Officer and Key Vault Administrator on the Key Vaults <NOTARY_KEY_VAULT_1_NAME>
and <NOTARY_KEY_VAULT_2_NAME>.
If you don't have the required roles then contact someone with the Owner or Role Based
Access Control Administrator role on the subscription to grant you the required roles.
#### 9.2.2.1 Remove old keys
1. Set the following environment variables:
> Note: **ACR_NOTARY_KEY_VAULT_NAME** will be changed during the procedure. It should begin this
section set to the old key vault you're moving away from. The first time through the MOP this will begin
equal to **KEY_VAULT_NAME**. In subsequent runs it will begin equal to one of **NOTARY_KEY_VAULT_1_NAME**
or **NOTARY_KEY_VAULT_2_NAME**.
```
ACR_NAME={{ ACR_NAME | default('<ACR_NAME>') }}
ACR_NOTARY_KEY_VAULT_NAME={{ ACR_NOTARY_KEY_VAULT_NAME | default('<ACR_NOTARY_KEY_VAULT_NAME>') }}
KEY_VAULT_RG={{ KEY_VAULT_RG | default('<KEY_VAULT_RG>') }}
SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')
```
1. Set the default subscription by running the command:
```
az account set --subscription "${SUBSCRIPTION_ID}"
```
1. Log in to the Azure Container Registry containing the images that you want to revoke signatures
on:
```
az acr login -n "${ACR_NAME}"
```
1. Disable Content Trust on the ACR, this removes all keys and signatures:
```
az acr config content-trust update \
-r "${ACR_NAME}" \
--status disabled
```
> Note: Once the keys and signatures have been removed from the ACR you may start to see alerts raised by the image verification service saying that image verification has failed. These alerts are expected and can be ignored while these steps are being followed.
1. Allow your current session to access the current notary key vault by running the
following command:
```
az keyvault network-rule add \
--name ${ACR_NOTARY_KEY_VAULT_NAME} \
--resource-group ${KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. Delete the following keys and passphrases that have been stored in the Key Vault:
- root-prv
- root-pub
- root-passphrase
- repository-passphrase
- For the previous and current release (e.g. 11-3-0 and 11-3-1):
- RELEASE-pub (e.g. 11-3-0-pub and 11-3-1-pub)
- RELEASE-prv (e.g. 11-3-0-prv and 11-3-1-prv)
For each entry in the list above (e.g. root-prv, repository-passphrase or 11-3-0-pub)
set the name as the environment variable SECRET and perform the following commands:
```
az keyvault secret delete \
--vault-name "${ACR_NOTARY_KEY_VAULT_NAME}" \
--name "${SECRET}"
```
1. The keys for each repository also need to be deleted. These are all 64-character hexadecimal
values (e.g. f968628abaabb872750032cd377ebf4c5975ac279240276ecd9f6efd80305ca3). This
can be done in one step but you must check that it isn't also going to delete secrets that aren't
related to image signing by following these steps:
- Run the following command to get a list of the secrets:
```
az keyvault secret list \
--vault-name "${ACR_NOTARY_KEY_VAULT_NAME}" \
--query "[].name | [?length(@) == \`64\`]"
```
and check that the list of secrets only includes random strings of 64 hexadecimal
characters. For example:
```
[
"00f3bab12395c711286a5c63f69bde702d02f84392bfde425e073da1cab05f7c",
"096cea132f19820f047f539fb4e1d1790ed92770de4c9c145b41d9a539506b1f",
"0a6ba8c5ae1c03c5e27726d29764a727f0051178b8d46362db6d8336d7c466c3",
"1c79a2ecebfede4a2c8a80c35c83ba969d83a3da772eafbb89505255f247b488",
"2c5900cbc61281f1795fe1891f65e40d31616cf5adafa3a225557453e1be54d3",
"333435756e8c902e200794c6541eacb2a061fc86d0a746d9004e056bf5a13995",
"347e2e2a9695e71f1e82a96b93d1b84481bb548fb756c463c97d2978ae7fa9e2",
"39642f486c0807e19f190341e1d13c4dcd1fae65775052506c98f70eaafdff09"
]
```
- If the list consists only of random strings then delete all of the entries with the
following commands:
```
export SECRET_LIST=$(az keyvault secret list \
--vault-name "${ACR_NOTARY_KEY_VAULT_NAME}" \
--query "[].name | [?length(@) == \`64\`]" \
-o tsv | tr '\n' ' ')
for SECRET in $SECRET_LIST; do
az keyvault secret delete \
--vault-name "${ACR_NOTARY_KEY_VAULT_NAME}" \
--name "${SECRET}";
done
```
1. Remove the current session from the notary key vault access list by running the
following command:
```
az keyvault network-rule remove \
--name ${ACR_NOTARY_KEY_VAULT_NAME} \
--resource-group ${KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. Delete all of the Docker images in the ACR:
```
export REPOS_LIST=$(az acr repository list \
-n "${ACR_NAME}" \
-o tsv)
for REPO in $REPOS_LIST; do
az acr repository delete --name "${ACR_NAME}" \
--repository "${REPO}" -y;
done
```
1. Re-enable Content Trust on the ACR:
```
az acr config content-trust update \
-r "${ACR_NAME}" \
--status enabled
```
#### 9.2.2.2 Update Key Vault Name in the Automation Repository
This is the git repository that holds the pipelines, Terraform scripts etc.
1. Set the following environment variables:
```
BRANCH={{ CHANGE_ID | default('<CHANGE_ID>') }}_rotate_notary_keys
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_AUTOMATION_URL=https://{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}@dev.azure.com/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/_git/{{ GIT_AUTOMATION_REPOSITORY | default('<GIT_AUTOMATION_REPOSITORY>') }}
```
- If using ssh to interact with the git repository
```
GIT_AUTOMATION_URL=git@ssh.dev.azure.com:v3/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/{{ GIT_AUTOMATION_REPOSITORY | default('<GIT_AUTOMATION_REPOSITORY>') }}
```
1. Change to an appropriate working directory in Cloud shell. Your Git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir automation_repo
cd automation_repo
```
1. Clone the existing Azure DevOps Git repository with **<GIT_AUTOMATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${GIT_AUTOMATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. In all files pipelines/configuration/pipeline_vars*.yml that contain the field, replace
the value of **ACR_NOTARY_KEY_VAULT_NAME** with the name of a different Notary Key Vault.
This must be one of the Notary Key Vaults that hasn't been used for notary keys for at
least 90 days. Use one of the two Key Vaults set up in section 9.2.1 unless neither of
them fits the above criteria and you need to create another new one.
1. Add any newly created files by running the command:
```
git add -A
```
1. Commit the change to the local branch by running the command:
```
git commit -a -m "Rotate Notary Keys"
```
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf automation_repo
```
(We have finished with the local copy of the repository)
#### 9.2.2.3 Generate and Upload New Keys
1. Update the **ACR_NOTARY_KEY_VAULT_NAME** environment variable to the new value in all
active sessions, and in any separate records you may keep of regional configuration variables, if
applicable. For example, make sure to update the template variable files that are used to populate SMOPs.
1. Run the mvmartifacts pipeline. This will upload the Docker images and generate a new
set of keys for the ACR, each repository and the delegate associated with the current
release. The pipeline will also sign the images with the new delegate key. The same
release name as before will generally be used here, but the pipeline will generate a new
set of keys.
See Markdown MOP r1154leauploadartifacts.md for full details on running this pipeline.
> Note: If you are currently using multiple MVM versions in your deployment (for
example because of an in-progress upgrade), you will need to run mvmartifacts
repeatedly, once for each version currently deployed.
#### 9.2.2.4 Update the Configuration Repository with New Public Key
This is the git repository that holds the configuration for MVM.
1. Set the following environment variables:
```
BRANCH={{ CHANGE_ID | default('<CHANGE_ID>') }}_rotate_notary_keys
CONFIG_VERSION={{ CONFIG_VERSION | default('<CONFIG_VERSION>') }}
```
Export the correct form of the URL to access the git repository
- If using https to interact with the git repository
```
GIT_CONFIGURATION_URL=https://{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}@dev.azure.com/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/_git/{{ GIT_CONFIGURATION_REPOSITORY | default('<GIT_CONFIGURATION_REPOSITORY>') }}
```
- If using ssh to interact with the git repository
```
GIT_CONFIGURATION_URL=git@ssh.dev.azure.com:v3/{{ ORGANIZATION_NAME | default('<ORGANIZATION_NAME>') }}/{{ PROJECT|default('<PROJECT>') }}/{{ GIT_CONFIGURATION_REPOSITORY | default('<GIT_CONFIGURATION_REPOSITORY>') }}
```
1. Change to an appropriate working directory in Cloud shell. Your Git repository
will live in a subdirectory off of this path.
```
cd ~
mkdir configuration_repo
cd configuration_repo
```
1. Clone the existing Azure DevOps Git repository with **<GIT_CONFIGURATION_URL>**.
The repository can be cloned using either ssh or https. In both cases you will
run the following command:
```
git clone ${GIT_CONFIGURATION_URL} .
```
(note the trailing whitespace and period after the URL)
- If using HTTPS:
- When prompted, input the password, **<GIT_PASSWORD>**, that you specified when the repository was first created
- If using SSH:
- You will not be prompted for a password.
This will create a local copy of the repository in the current working directory.
1. Create a new working branch by running the command
```
git checkout -b ${BRANCH}
```
The branch currently only exists on your local server - it will be pushed to the DevOps repository in a later step
1. Create a new config set with an appropriate description by running the command:
```
mvm-config-manager prepare-new-config \
--from-config-version ${DOWNLEVEL_CONFIG_VERSION} \
--description "Rotate notary keys"
```
This will create create a new config version and commit (but not push) the code.
On success this will return a message of the form
`Created new config version <UPLEVEL_CONFIG_VERSION> ...`
Record the value of **UPLEVEL_CONFIG_VERSION**; it will be required later in this procedure.
Set it as an environment variable as well:
```
UPLEVEL_CONFIG_VERSION=<UPLEVEL_CONFIG_VERSION>
```
1. In the new config version folder, replace the value of the ACR_NOTARY_ROOT_PUBLIC_KEY
variable in your config repository (config/**UPLEVEL_CONFIG_VERSION**/region.vars) to
match the new value of the root-pub secret. The new value of the root-pub secret can
be retrieved from the mvmartifacts pipeline, or directly from the notary key vault
with the following command:
```
az keyvault secret show \
-n root-pub \
--vault-name "${ACR_NOTARY_KEY_VAULT_NAME}" \
--query value \
-o tsv
```
1. Commit the change to the local branch by running the command:
```
git commit -a -m "Rotate notary keys"
```
1. Apply the new config to the existing SIMPL version, by running:
```
mvm-config-manager upgrade-simpl --config-version ${UPLEVEL_CONFIG_VERSION}
```
This should apply the new ACR_NOTARY_ROOT_PUBLIC_KEY to the current SIMPL version folder.
1. Push the change to the DevOps repository by running the command:
```
git push --set-upstream origin ${BRANCH}
```
1. Merge the change into the main branch via the 'pull request' mechanism
1. Tidy up by running the command:
```
cd ~
rm -rf configuration_repo
```
(We have finished with the local copy of the repository)
#### 9.2.2.5 Redeploy SIMPL to pick up the changes
SIMPL and all SIs must be redeployed to ensure that any stored signature information is
cleared and the images with the new signatures are used.
> Note: This MOP requires SIMPL to be destroyed before being recreated. This means SIs
cannot be managed while the SIMPL replacement is ongoing.
##### 9.2.2.5.1 Set the default subscription
1. Set the default subscription by running the command:
```
az account set --subscription "${SUBSCRIPTION_ID}"
```
##### 9.2.2.5.2 List the Application SIs SIMPL is managing
This allows you to verify that the restarted SIMPL is managing the same set of application SIs
1. Set the following environment variables:
```
REGIONAL_PIPELINE_KEY_VAULT_NAME={{ REGIONAL_PIPELINE_KEY_VAULT_NAME | default('<REGIONAL_PIPELINE_KEY_VAULT_NAME>') }}
REGIONAL_PIPELINE_KEY_VAULT_RG={{ REGIONAL_PIPELINE_KEY_VAULT_RG | default('<REGIONAL_PIPELINE_KEY_VAULT_RG>') }}
SIMPL_SERVICE_NAME={{ SIMPL_SERVICE_NAME | default('<SIMPL_SERVICE_NAME>') }}
SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')
```
1. Allow your current session to access the key vault by running the following commands:
```
az keyvault network-rule add \
--name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. Extract the SIMPL Resource Group by running the following command:
```
SIMPL_SERVICE_RG=$(az keyvault secret show \
--name simpl-service-rg \
--vault-name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--query 'value' -o tsv)
echo $SIMPL_SERVICE_RG
```
1. Remove the current session from the key vault access list by running the following command:
```
az keyvault network-rule remove \
--name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. List the Application SIs being managed by this SIMPL:
```
az aks command invoke \
--name ${SIMPL_SERVICE_NAME}-k8s \
--resource-group ${SIMPL_SERVICE_RG} \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get si"
az aks command invoke \
--name ${SIMPL_SERVICE_NAME}-k8s \
--resource-group ${SIMPL_SERVICE_RG} \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get dc"
```
Save the results of these commands, for comparison after SIMPL is restarted.
##### 9.2.2.5.3 Delete the SIMPL cluster
These commands are run from the DevOps Portal created in section 9.1
1. Select pipelines from the Left hand menu
1. Select pipelines from the sub menu
1. Select "All" from the resultant page
1. Expand pipelines
1. Select the pipeline `simpldestroy`
1. Select Run pipeline
1. In the resultant Run pipeline menu
- Set the name of the release containing the SIMPL definitions to `{{ MVM_FILESHARE | default('<MVM_FILESHARE>') }}`
- Set the name of the config file to `{{ PIPELINE_CONFIGURATION_NAME | default('<PIPELINE_CONFIGURATION_NAME>') }}`
- Select Run
Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.
The progress of the pipeline can be followed by clicking on the jobs list.
Once the pipeline has completed, success, or failure, is reported to screen
##### 9.2.2.5.4 Recreate the SIMPL cluster
These commands are run from the DevOps Portal created in section 9.1
1. Select pipelines from the Left hand menu
1. Select pipelines from the sub menu
1. Select "All" from the resultant page
1. Expand pipelines
1. Select the pipeline `simplapply`
1. Select Run pipeline
1. In the resultant Run pipeline menu
- Set the name of the release containing the SIMPL definitions to `{{ MVM_FILESHARE | default('<MVM_FILESHARE>') }}`
- Set the name of the config file to `{{ PIPELINE_CONFIGURATION_NAME | default('<PIPELINE_CONFIGURATION_NAME>') }}`
- Leave `Enable grace mode for image verification` and `Enabled caching of image signatures` at their default values.
- Select Run
Selecting Run will cause an approval email to be sent to the reviewers list. The pipeline will stall until the reviewers have approved the request. Once approval has been given the pipeline will run.
The progress of the pipeline can be followed by clicking on the jobs list.
Once the pipeline has completed, success, or failure, is reported to screen
##### 9.2.2.5.5 Verify that the SIMPL Cluster is still managing all the configuration
1. Set the following environment variables:
```
REGIONAL_PIPELINE_KEY_VAULT_NAME={{ REGIONAL_PIPELINE_KEY_VAULT_NAME | default('<REGIONAL_PIPELINE_KEY_VAULT_NAME>') }}
REGIONAL_PIPELINE_KEY_VAULT_RG={{ REGIONAL_PIPELINE_KEY_VAULT_RG | default('<REGIONAL_PIPELINE_KEY_VAULT_RG>') }}
SUBSCRIPTION_ID={{ SUBSCRIPTION_ID | default('<SUBSCRIPTION_ID>') }}
USER_IP=$(curl -s http://ipinfo.io/json | jq -r '.ip')
```
1. Allow your current session to access the key vault by running the following commands:
```
az keyvault network-rule add \
--name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. Extract the SIMPL Resource Group by running the following command:
```
SIMPL_SERVICE_RG=$(az keyvault secret show \
--name simpl-service-rg \
--vault-name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--query 'value' -o tsv)
echo $SIMPL_SERVICE_RG
```
1. Remove the current session from the key vault access list by running the following command:
```
az keyvault network-rule remove \
--name ${REGIONAL_PIPELINE_KEY_VAULT_NAME} \
--resource-group ${REGIONAL_PIPELINE_KEY_VAULT_RG} \
--ip-address ${USER_IP}
```
1. Verify that the pods start successfully by running the command:
```
az aks command invoke \
--name ${SIMPL_SERVICE_NAME}-k8s \
--resource-group ${SIMPL_SERVICE_RG}\
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get pods -A"
```
You should see pods come up in the following namespaces:
- connaisseur
- csi-driver
- default
- gitops
- istio-system
- kube-system
- simon
- simpl
If any of the pods enter a failed state (anything other than `Init`, `PodInitializing` or `Running`), see the Troubleshooting section in the Deployment Guide for troubleshooting guidance.
>Depending on your Azure configuration, you might be prompted to re-login to Azure (including MFA) the first time the kubectl command executes. This is normal and to be expected
1. Get the application SIs now being managed by the SIMPL, by running the following commands:
```
az aks command invoke \
--name ${SIMPL_SERVICE_NAME}-k8s \
--resource-group ${SIMPL_SERVICE_RG} \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get si"
az aks command invoke \
--name ${SIMPL_SERVICE_NAME}-k8s \
--resource-group ${SIMPL_SERVICE_RG} \
--subscription "${SUBSCRIPTION_ID}" \
--command "kubectl get dc"
```
Verify that the list of SIs returned above are the same as those extracted before respinning SIMPL.
#### 9.2.2.6 Redeploy the SIs
1. To complete propagation of the new notary keys, redeploy the SIs in the following order:
- Simon
- SAS
- ESC
The SI should be replaced by following the appropriate upgrade MOP(s) as shown below.
|**SI**|**MOPs**|
|--|--|
|Simon | r1154leaupgradesimon.md |
|SAS | r1154leaupgradesas.md & r1154leadeletesas.md |
|ESC | r1154leaupgradeesc.md |
#### 9.2.2.7 Verification
1. There should be no further alerts raised about image verification failures at this point.
If there are then you should investigate them following the troubleshooting instructions
in the Deployment Guide, or contact your Support representative.
## 9.3 Test Plan
There is no explicit test plan associated with this MOP. You should refer to the
test plan associated with the appropriate SI upgrade MOP.
## 9.4 Backout Procedure
There is no back-out process associated with this MOP. Once any of the keys have
been deleted it is not possible to recreate them without completing the procedure.
If the new keys are compromised the MOP should be re-run to replace them again.
# 10. Post checks
[System healthchecks]
# 11. Risk Assessment Score
1 - TBD
# 12. Execute MOP clean up if required
# 13. End of Document MOP
# 14. Service Assurance/Monitoring
# A. Appendix and Tables
# B. Approvers
# C. Peer Reviewers
# D. References for Other Documents
# E. Additional Appendices (If required)