MSR repository size utility
You can use the Python 3 utility tool to learn the size of your MSR repository. With the tool, you can make both basic size queries and simple size queries:
Basic size queries return the total size shared with other repositories and the portion unique to the repository itself.
Simple size queries return the total size of a repository only, without information as to which portion is shared with other repositories or which portions are unique to the repository itself.
Activate Python 3 utility tool
Create a new Python 3 virtual environment:
~ python3 -m venv myenv
Activate the virtual environment:
~ source myenv/bin/activate
Create the
requirements.txtfile with the following content:certifi==2024.7.4 charset-normalizer==3.3.2 idna==3.7 requests==2.32.3 urllib3==2.2.2
Install the requirements in the virtual environment:
(myenv) ➜ ~ pip install -r requirements.txt
Run the tool from within the virtual environment:
(myenv) ➜ ~ python3 repository_size.py --help usage: repository_size.py [-h] [--host HOST] [--username USERNAME] [--password PASSWORD] [--page-size PAGE_SIZE] [--simple [SIMPLE]] [--namespaces NAMESPACES] [--repositories REPOSITORIES] [--cacert CACERT] [--insecure [INSECURE]] [--output OUTPUT] [--log-level LOG_LEVEL]
Query MSR repository size
To query the sizes of all repositories within MSR and output a summary to stdout:
(myenv) ➜ ~ python3 repository_size.py --cacert=/path/to/msr/cacert.pem --output -
Example output:
[2024-06-25 00:04:55,132] - INFO - Fetching simple=False repository sizes from MSR (127.0.0.1:8443) with user (admin)
[2024-06-25 00:04:55,132] - INFO - Received 0 repositories from user input
[2024-06-25 00:04:55,132] - INFO - No repositories or namespaces provided, so getting ALL repositories from MSR
[2024-06-25 00:04:55,737] - INFO - Fetched 11 repositories from MSR. Now fetching size for each repository. This may take a while...
[2024-06-25 00:04:55,737] - INFO - Retrieving sizes for 11 repositories (duplicates removed). This may take a while...
[2024-06-25 00:04:56,011] - INFO - Fetched size for 'msr/msr-api' repository: (unique=42680869, shared=5338472, total=48019341)
[2024-06-25 00:04:56,288] - INFO - Fetched size for 'msr/msr-content-cache' repository: (unique=15473937, shared=5338278, total=20812215)
[2024-06-25 00:04:56,568] - INFO - Fetched size for 'msr/msr-garant' repository: (unique=26427185, shared=5338472, total=31765657)
[2024-06-25 00:04:56,847] - INFO - Fetched size for 'msr/msr-installer' repository: (unique=26337376, shared=5338472, total=31675848)
[2024-06-25 00:04:57,148] - INFO - Fetched size for 'msr/msr-jobrunner' repository: (unique=1261495926, shared=0, total=1261495926)
[2024-06-25 00:04:57,459] - INFO - Fetched size for 'msr/msr-nginx' repository: (unique=40166380, shared=5338472, total=45504852)
[2024-06-25 00:04:57,747] - INFO - Fetched size for 'msr/msr-notary-server' repository: (unique=5742530, shared=5338472, total=11081002)
[2024-06-25 00:04:58,023] - INFO - Fetched size for 'msr/msr-notary-signer' repository: (unique=5293658, shared=5338472, total=10632130)
[2024-06-25 00:04:58,303] - INFO - Fetched size for 'msr/msr-registry' repository: (unique=37647972, shared=5338472, total=42986444)
[2024-06-25 00:04:58,581] - INFO - Fetched size for 'admin/harbor-core-base' repository: (unique=18599503, shared=0, total=18599503)
[2024-06-25 00:04:58,859] - INFO - Fetched size for 'admin/ubuntu' repository: (unique=27207556, shared=0, total=27207556)
{
"msr/msr-api": {
"namespace": "msr",
"name": "msr-api",
"unique": 42680869,
"shared": 5338472,
"total": 48019341
},
"msr/msr-content-cache": {
"namespace": "msr",
"name": "msr-content-cache",
"unique": 15473937,
"shared": 5338278,
"total": 20812215
},
"msr/msr-garant": {
"namespace": "msr",
"name": "msr-garant",
"unique": 26427185,
"shared": 5338472,
"total": 31765657
},
"msr/msr-installer": {
"namespace": "msr",
"name": "msr-installer",
"unique": 26337376,
"shared": 5338472,
"total": 31675848
},
"msr/msr-jobrunner": {
"namespace": "msr",
"name": "msr-jobrunner",
"unique": 1261495926,
"shared": 0,
"total": 1261495926
},
"msr/msr-nginx": {
"namespace": "msr",
"name": "msr-nginx",
"unique": 40166380,
"shared": 5338472,
"total": 45504852
},
"msr/msr-notary-server": {
"namespace": "msr",
"name": "msr-notary-server",
"unique": 5742530,
"shared": 5338472,
"total": 11081002
},
"msr/msr-notary-signer": {
"namespace": "msr",
"name": "msr-notary-signer",
"unique": 5293658,
"shared": 5338472,
"total": 10632130
},
"msr/msr-registry": {
"namespace": "msr",
"name": "msr-registry",
"unique": 37647972,
"shared": 5338472,
"total": 42986444
},
"admin/harbor-core-base": {
"namespace": "admin",
"name": "harbor-core-base",
"unique": 18599503,
"shared": 0,
"total": 18599503
},
"admin/ubuntu": {
"namespace": "admin",
"name": "ubuntu",
"unique": 27207556,
"shared": 0,
"total": 27207556
}
}
To make a simple size query for a specific list of namespaces and repositories using an insecure connection:
(myenv) ➜ ~ python3 repository_size.py --insecure --namespaces=admin --repositories=msr/msr-api,msr/msr-nginx --simple --output -
Example output:
[2024-06-25 00:06:15,567] - INFO - Fetching simple=True repository sizes from MSR (127.0.0.1:8443) with user (admin)
[2024-06-25 00:06:15,567] - INFO - Received 2 repositories from user input
[2024-06-25 00:06:15,855] - INFO - Fetched 2 repositories in 'admin' namespace
[2024-06-25 00:06:15,856] - INFO - Retrieving sizes for 4 repositories (duplicates removed). This may take a while...
[2024-06-25 00:06:16,124] - INFO - Fetched size for 'msr/msr-api' repository: (unique=0, shared=0, total=48019341)
[2024-06-25 00:06:16,426] - INFO - Fetched size for 'msr/msr-nginx' repository: (unique=0, shared=0, total=45504852)
[2024-06-25 00:06:16,699] - INFO - Fetched size for 'admin/harbor-core-base' repository: (unique=0, shared=0, total=18599503)
[2024-06-25 00:06:16,964] - INFO - Fetched size for 'admin/ubuntu' repository: (unique=0, shared=0, total=27207556)
{
"msr/msr-api": {
"namespace": "msr",
"name": "msr-api",
"total": 48019341
},
"msr/msr-nginx": {
"namespace": "msr",
"name": "msr-nginx",
"total": 45504852
},
"admin/harbor-core-base": {
"namespace": "admin",
"name": "harbor-core-base",
"total": 18599503
},
"admin/ubuntu": {
"namespace": "admin",
"name": "ubuntu",
"total": 27207556
}
}