With the Gateway gunicorn process and daemons stopped, run the following command to apply migrations to the Gateway’s store:
$ rok-gw-manage migrate
Note that this migration will move all Gateway documents from the v2 to v3 etcd
store, and is expected to be slow in large Gateway installations. For more
information on the available options, you can use the
--help as well as the
rok-gw-manage showmigrations commands.
This migration will cause buckets published by this Rok Gateway to stop syncing, since moving to a new etcd store may cause the store’s revision to go back in time. These buckets should be manually re-published after the migration is performed.
From v0.14 onwards, libmap migrated to etcd v3 API. This means that the libmap epoch cache will operate on the v3 store and libmap will not have access to the previous values of the v2 store. Instead of migrating the existent epochs to v3, we opt to recreate the epoch cache, since we do not care about the relative epoch differences between maps/chocks.
To successfully migrate a Rok cluster where composers are running in nodelocal mode:
Ensure that no GC instance runs and will run in any node, until the upgrade process is complete.
- For each cluster node:
Upgrade the node and restart the necessary Rok daemons.
Set a new epoch for all maps/chocks found in the underlying nodelocal (mutable) storage:
$ rok-composer-tool epoch-update maps-uuid chocks-idx
Update the epoch for all maps/chocks found in the underlying local (immutable) storage. In the master node run:
$ rok-composer-tool epoch-update maps-ca chocks-ca
Cleanup the stale etcd2 epoch cache:
$ ETCDCTL_API=2 etcdctl --endpoints $ROK_COMPOSER_ETCD_ENDPOINT rm -r /epochs
The epoch update steps may take a while, depending on the number of existent maps/chocks. Afterwards, all maps/chocks in the cluster will have a valid epoch in the v3 store, which will both protect new maps/chocks while a GC instance is running, and will work as a map/chock CA cache. From this point on, one is allowed to perform garbage collection again.
For Rok deployments where composer runs in local mode, the epoch update step can be compacted into:
$ rok-composer-tool epoch-update --all
Still, garbage collection should not be performed before both node upgrade and epoch update have been completed.
While the epoch update is not vital to avoid data corruption, since the GC will not collect any maps/chocks without an epoch, it is advisable, so that it takes just one GC run to collect the maps/chocks eligible for deletion.
To speedup the epoch update process, one can run
starting the upgrade steps, to collect any useless maps/chocks that would
otherwise, unecessarily, obtain an epoch.