Setup disconnected Docker registry for Rhosp 13
Pre-requisites:
1. Server for registry that has 2 nics to connect to internet and disconnected servers
2. Director configured with latest minor release
3. Current Redhat subscription
4. Hundreds of GB free, preferably a TB. It is wise to have /var as a separate mounted volume.
On registry run:$ yum update -y
Install the Offline registry and Tag the images
$ yum install -y docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen
For the registry itself – to use after images are pushed to the offline registry$ systemctl enable docker
$ systemctl start docker
$ systemctl enable docker-distribution
$ systemctl start docker-distribution
1. ONLY for registry server: edit /etc/docker-distribution/registry/config.yml
Edit last line that has http address add the server domain name before the :5000
2. Each server that uses the offline registry needs to have the following, using your domain name before :5000 this includes the registry and director$ echo 'INSECURE_REGISTRY="--insecure-registry ol-rhel-disconnected-registry:5000"' >> /etc/sysconfig/docker
3. Each server that uses the offline registry needs to have this.
For the offline registry server, which itself is completely online, Edit
/etc/containers/registries.conf to the following did allow docker pull to work.
Add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP.
# vi /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: ‘registries.search’, ‘registries.insecure’,
# and ‘registries.block’.
[registries.search]
registries = [‘ol-rhel-disconnected-registry:5000’, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]
# If you need to access insecure registries, add the registry’s fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = [‘ol-rhel-disconnected-registry:5000’]
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []
The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file looks like:
$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.
[registries.search]
registries = ['ol-rhel-disconnected-registry:5000']
# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['ol-rhel-disconnected-registry']
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
Restart Docker
# systemctl restart docker
# systemctl restart docker-distribution
On registry download, tag and push the images. This is simpler than Red Hat Openshift since all containers are standard tags
From Red Hat registry.
For testing purposes I ran this from the registry server instead of the director, I used a slightly different command to create the overcloud-images.yaml file
I ran the pull-tag-push script on the registry, then from the director I ran the openstack command to create overcloud-images.yaml
Here are the scripts that I ran:
From registry:
#!/bin/bash
if test -n "$STY"
then
printf "This is a screen session named '$STY'.\n"
ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=13.0
echo "Pulling ${IMAGE_NAME}:${TAG} ..."
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf "This is NOT a screen session.\n"
exit 1
fi
to create the overcloud-images.yaml file
$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry/rhosp13 --prefix=openstack- --tag 13.0 --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
On a real director the script and command should be:
#!/bin/bash
if test -n "$STY"
then
printf "This is a screen session named '$STY'.\n"
ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=$(openstack overcloud container image tag discover --image ${IMAGE_NAME} --tag-from-label {version}-{release})
echo "Pulling ${IMAGE_NAME}:${TAG} ..."
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf "This is NOT a screen session.\n"
exit 1
fi
The openstack command to build the overcloud-images should be
$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry:5000/rhosp13 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml
DockerInsecureRegistryAddress: [‘myregistry.local:8787’]
Troubleshooting
When building the overcloud-images.yaml file. I got an error
Not found image: docker://ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”
to solve, I manually downloaded and tagged the image.
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0
ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0
[root@rhel-disconnected-registry cloud-user]# docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0
When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :
Image ol-rhel-disconnected-registry:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.
Available tags: 13.0, latest
For monitoring you will need to add to the overcloud-image.yaml creation:
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \
–output-env-file /home/stack/templates/overcloud_images.yaml
Octopus Computer Solutions is a Red Hat business partner. We deploy OpenStack and OpenShift throughout Israel and the world.
