Requirements

To start the migration process, your Private Packagist Self-Hosted Replicated Native instance needs to run version 1.12.8 or newer. To reduce the disruption to a minimum, we recommend that you set up your Private Packagist Self-Hosted Kubernetes instance on a different machine before you start the migration process.

Please note that it is possible to install Private Packagist Self-Hosted Kubernetes on the same machine as your current Private Packagist Self-Hosted Installation. You can do so after you follow the instructions below to back up your existing data and before you start restoring the application data. However, the Private Packagist Self-Hosted installation process will uninstall certain components used by your current Private Packagist Self-Hosted installation.

Backing up the data

Private Packagist Self-Hosted stores data in three different ways. A PostgreSQL database, a Redis database used as storage for installation and usage statistics, cache, and job queue, and a file/blob storage for uploaded artifacts and dist files used during composer install.

To avoid any data inconsistency and random errors during application usage all three of them need to be backed up and restored at the same time.

Please note that you won't be able to restore Private Packagist Self-Hosted Kubernetes from snapshots created by Private Packagist Self-Hosted Replicated Native.

Stop the Private Packagist application

docker stop packagist-worker packagist-web packagist-repo

Backup the dist and artifact files

At the end of this step, you should have a packagist_storage.tar.gz file in your working directory.

Please note that depending on the number of packages and versions used in your projects and the size of your dependencies it can take several minutes for this command to finish.

docker exec -it packagist-ui /srv/manager/bin/console packagist:self-hosted:migrate-storage export
docker cp packagist-ui:/tmp/storage.tar.gz packagist_storage.tar.gz
docker exec -it packagist-ui rm /tmp/storage.tar.gz
docker stop packagist-ui

In case you run into timeout problems while running the migrate-storage command, we offer an alternative approach using only the tar command.

# The following commands may require root permissions
docker stop packagist-ui
cd /data/packagist

# Ensure all needed folders are present
mkdir -p artifacts composer

# Replace YOUR_BACKUP_PATH with the path you wish to save the backup to
tar -cvzf YOUR_BACKUP_PATH/packagist_storage.tar.gz --transform s/^composer/dist/ --transform s/^artifacts/artifact/ composer artifacts

Backup the PostgreSQL database

At the end of this step, you should have a packagist_db.sql file in your working directory.

docker exec -it packagist-postgres pg_dump --clean --if-exists -U postgres -d packagist_db > packagist_db.sql
docker stop packagist-postgres

Backup the Redis database

At the end of this step, you should have a packagist_redis.rdb file in your working directory.

docker exec -it packagist-redis /usr/local/bin/redis-backup.sh
docker cp packagist-redis:/data/dump.rdb packagist_redis.rdb
docker stop packagist-redis

Start Private Packagist again (optional)

Now that you have exported all the data, you may restart the Private Packagist application. This will allow your developers to continue with their work while you import the data in the Private Packagist Self-Hosted Kubernetes setup.

docker start packagist-postgres packagist-redis packagist-ui packagist-repo packagist-web packagist-worker    

Please note that any changes made in Private Packagist will now be lost and will have to be reapplied once everyone switches to Private Packagist Self-Hosted Kubernetes.

Installing Private Packagist Self-Hosted Kubernetes

If you haven't done so already, now is the time to follow the installation guide to install Private Packagist Self-Hosted Kubernetes.

Import the application data in Private Packagist Self-Hosted Kubernetes

All commands assume that you are logged in to the server where Private Packagist Self-Hosted Kubernetes is installed and your working directory contains the three files generated by the previous steps.

If you installed Private Packagist Kubernetes in an existing cluster and are not using the default namespace, make sure to run the commands in the correct namespace or switch the namespace before running any of the command.

kubectl config set-context --current --namespace=<namespace-name>

To avoid any data inconsistency, we highly recommend going through the steps below in order and only start the application once all the data has been restored.

Stop the Private Packagist application

kubectl scale deployment ui repo worker --replicas 0

To verify that all ui, repo, and worker pods have been run execute the command below and verify that the pods don't appear in the list anymore.

kubectl get pods

Import the PostgreSQL database

Follow the instructions below to restore the PostgreSQL database in the cluster. If you are using your own PostgreSQL instance, use the packagist_db.sql file directly to import the data into your instance.

Please note that the backup will overwrite any existing data in the database and any changes you made during the setup will be lost.

After each scale command, use kubectl get pods again to verify that the postgres pod is in the correct running state.

kubectl scale statefulset postgres --replicas 0
kubectl delete pvc postgres-data-postgres-0
kubectl scale statefulset postgres --replicas 1
cat packagist_db.sql | kubectl exec -it statefulsets/postgres -- psql -U postgres -d packagist_db

Import the Redis database

Follow the instructions below to restore the Redis database for Private Packagist Self-Hosted without an existing Kubernetes cluster. If you are using your own Redis instance, use the packagist_redis.rdb file directly to restore Redis from an RDB file or follow the instructions of your hosted service provider. If you are installing Private Packagist Self-Hosted in an existing cluster and aren't planning on using your own Redis then please contact us.

After each scale command, use kubectl get pods again to verify that the redis pod is in the correct running state.

export PV_NAME=$(kubectl get pvc redis-data-redis-0 -ojsonpath='{.spec.volumeName}')
kubectl scale statefulset -n default redis --replicas 0
# Depending on your setup, you might need to run the following three commands as `root`
rm -f /var/openebs/local/$PV_NAME/appendonly.aof
rm -rf /var/openebs/local/$PV_NAME/appendonlydir
cp packagist_redis.rdb /var/openebs/local/$PV_NAME/dump.rdb
kubectl scale statefulset redis --replicas 1

Import the dist and artifact files

Importing the dist and artifact files requires the UI pods to be back online. Start them with the command below. This will take a few seconds. You can use the second command to verify once 2/2 pods are running.

kubectl scale deployment ui --replicas 1
kubectl get pods -w

Please note that depending on the size of the packagist_storage.tar.gz file and the type of blob storage that you have configured it can take several minutes for this command to finish.

export UI_POD=$(kubectl get pods --field-selector=status.phase=Running --no-headers -o custom-columns=":metadata.name"|grep ui-)
kubectl cp packagist_storage.tar.gz $UI_POD:/tmp/packagist_storage.tar.gz -c ui
kubectl exec $UI_POD -c ui -- /bin/sh -c "/srv/manager/bin/console packagist:self-hosted:migrate-storage import /tmp/packagist_storage.tar.gz && rm /tmp/packagist_storage.tar" 

Start the Private Packagist application

Once your data has been restored. Start the application:

kubectl scale deployment ui repo worker --replicas 1

This can take a few minutes. You can run the command below to see when all pods are back up and running.

kubectl get pods -w

Once all the pods are back up and running, log in to Private Packagist and verify that your organization is shown in the UI. Ideally, run a composer update command in one of your projects to assert that the Composer repository responds as expected.

Adjust domain names

In case you initially set up Private Packagist Self-Hosted Kubernetes with a different domain name, you can now edit the domain names in the admin panel. If necessary, adjust your DNS entries, and shut down the old Private Packagist Self-Hosted instance.

Clear the cache

The final step is to clear the Composer endpoint cache. If you are not using the built-in Redis database, check your internal documentation how to connect to the Redis instance.

Important: Please note that there are multiple databases created in your Redis instance, and that Redis is not solely used for caching. Be careful not to flush any of the other migrated databases.

kubectl exec -it redis-0 -- redis-cli -p 9869 
select 5
flushdb

Update Composer projects

Update the Composer projects that use your self-hosted Private Packagist instance to make sure all repository references are up-to-date:

composer update mirrors