Visits: 10219
Setup disconnected Docker registry for Rhosp
Setup disconnected Docker registry for Rhosp 13
Pre-requisites:
1. Server for registry that has 2 nics to connect to internet and disconnected servers
2. Director configured with latest minor release
3. Current Redhat subscription
4. Hundreds of GB free, preferably a TB. It is wise to have /var as a separate mounted volume.
Read the original article that I wrote where I work at Octopus Computer Solutions
Sources for this article are below
On registry run:$ yum update -y
Install the Offline registry and Tag the images
$ yum install -y docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen
For the registry itself – to use after images are pushed to the offline registry$ systemctl enable docker
$ systemctl start docker
$ systemctl enable docker-distribution
$ systemctl start docker-distribution
1. ONLY for registry server: edit /etc/docker-distribution/registry/config.yml
Edit last line that has http address add the server domain name before the :5000
2. Each server that uses the offline registry needs to have the following, using your domain name before :5000 this includes the registry and director$ echo 'INSECURE_REGISTRY="--insecure-registry ol-rhel-disconnected-registry:5000"' >> /etc/sysconfig/docker
3. Each server that uses the offline registry needs to have this.
For the offline registry server, which itself is completely online, Edit
/etc/containers/registries.conf to the following did allow docker pull to work.
Add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP.
# vi /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.
[registries.search]
registries = ['ol-rhel-disconnected-registry:5000', 'registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['ol-rhel-disconnected-registry:5000']
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []
The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file looks like:
$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.
[registries.search]
registries = ['ol-rhel-disconnected-registry:5000']
# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['ol-rhel-disconnected-registry']
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
Restart Docker
# systemctl restart docker
# systemctl restart docker-distribution
On registry download, tag and push the images. This is simpler than Red Hat Openshift since all containers are standard tags
From Red Hat registry.
For testing purposes I ran this from the registry server instead of the director, I used a slightly different command to create the overcloud-images.yaml file
I ran the pull-tag-push script on the registry, then from the director I ran the openstack command to create overcloud-images.yaml
Here are the scripts that I ran:
From registry:
#!/bin/bash
if test -n "$STY"
then
printf "This is a screen session named '$STY'.\n"
ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=13.0
echo "Pulling ${IMAGE_NAME}:${TAG} ..."
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf "This is NOT a screen session.\n"
exit 1
fi
to create the overcloud-images.yaml file
$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry/rhosp13 --prefix=openstack- --tag 13.0 --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
On a real director the script and command should be:
#!/bin/bash
if test -n "$STY"
then
printf "This is a screen session named '$STY'.\n"
ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=$(openstack overcloud container image tag discover --image ${IMAGE_NAME} --tag-from-label {version}-{release})
echo "Pulling ${IMAGE_NAME}:${TAG} ..."
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf "This is NOT a screen session.\n"
exit 1
fi
The openstack command to build the overcloud-images should be
$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry:5000/rhosp13 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml
DockerInsecureRegistryAddress: [‘myregistry.local:8787’]
Troubleshooting
When building the overcloud-images.yaml file. I got an error
Not found image: docker://ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”
to solve, I manually downloaded and tagged the image.
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0
ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0
[root@rhel-disconnected-registry cloud-user]# docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0
When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :
Image ol-rhel-disconnected-registry:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.
Available tags: 13.0, latest
For monitoring you will need to add to the overcloud-image.yaml creation:
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \
–output-env-file /home/stack/templates/overcloud_images.yaml
Pre-requisites:
- Server for registry that has 2 nics to connect to internet and disconnected servers
- Director configured with latest minor release
- Current Redhat subscription
- Hundreds of GB free, preferably a TB. It is wise to have /var as a separate mounted volume.
On registry run:
$ yum update -y
3.2 Install the Offline registry and Tag the images
yum install -y docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen
For the registry itself – to use after images are pushed to the offline registry
$ systemctl enable docker
$ systemctl start docker
$ systemctl enable docker-distribution
$ systemctl start docker-distribution
- ONLY for registry server edit /etc/docker-distribution/registry/config.yml
Edit last line that has http address add the server domain name before the :5000
- Each server that uses the offline registry needs to have, using your domain name before :5000 this includes the registry and director
$ echo ‘INSECURE_REGISTRY=”–insecure-registry ol-rhel-disconnected-registry:5000″‘ >> /etc/sysconfig/docker
- Each server that uses the offline registry needs to have, this includes the registry and director
in /etc/containers/registries.conf add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP.
# vi /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: ‘registries.search’, ‘registries.insecure’,
# and ‘registries.block’.
[registries.search]
registries = [‘ol-rhel-disconnected-registry:5000′, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]
# If you need to access insecure registries, add the registry’s fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = [‘ol-rhel-disconnected-registry:5000’]
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []
The same file /etc/containers/registries.conf on the servers looks like:
$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: ‘registries.search’, ‘registries.insecure’,
# and ‘registries.block’.
[registries.search]
registries = [‘ol-rhel-disconnected-registry:5000’]
# If you need to access insecure registries, add the registry’s fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = [‘ol-rhel-disconnected-registry’]
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]
COPY it to the other servers
ansible others -m copy -a “src=/etc/containers/registries.conf dest=/etc/containers/registries.conf”
Restart Docker
# systemctl restart docker
# systemctl restart docker-distribution
On registry download, tag and push the images. This is simpler than openshift since all containers are standard tags
From
As in https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/architecture.html
For the offline server, which itself is completely online, Edit
/etc/containers/registries.conf to the following did allow docker pull to work
# cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: ‘registries.search’, ‘registries.insecure’,
# and ‘registries.block’.
[registries.search]
registries = [‘ol-rhel-disconnected-registry:5000’, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]
# If you need to access insecure registries, add the registry’s fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = [‘ol-rhel-disconnected-registry:5000’]
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []
The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file looks like:
$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: ‘registries.search’, ‘registries.insecure’,
# and ‘registries.block’.
[registries.search]
registries = [‘ol-rhel-disconnected-registry:5000’]
# If you need to access insecure registries, add the registry’s fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = [‘ol-rhel-disconnected-registry’]
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]
COPY it to the other servers
ansible others -m copy -a “src=/etc/containers/registries.conf dest=/etc/containers/registries.conf”
Restart Docker
# systemctl restart docker
# systemctl restart docker-distribution
Since we did NOT do a complete install of same minor release with the director, I used a slightly different command to create the overcloud-images.yaml file
I ran the pull-tag-push script on the registry, then from the director I ran the openstack command to create overcloud-images.yaml
Here are the scripts that I ran:
From registry:
#!/bin/bash
#from:https://access.redhat.com/articles/3348761
if test -n “$STY”
then
printf “This is a screen session named ‘$STY’.\n”
ASSET_SERVER=172.28.117.56
docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=13.0
echo “Pulling ${IMAGE_NAME}:${TAG} …”
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf “This is NOT a screen session.\n”
exit 1
fi
to create the overcloud-images.yaml file
openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag 13.0 –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
On a real director the script and command should be:
#!/bin/bash
#from:https://access.redhat.com/articles/3348761
if test -n “$STY”
then
printf “This is a screen session named ‘$STY’.\n”
ASSET_SERVER=172.28.117.56
docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
TAG=$(openstack overcloud container image tag discover –image ${IMAGE_NAME} –tag-from-label {version}-{release})
echo “Pulling ${IMAGE_NAME}:${TAG} …”
docker pull ${IMAGE_NAME}:${TAG}
IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
else printf “This is NOT a screen session.\n”
exit 1
fi
The openstack command to build the overcloud-images should be
openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag-from-label {version}-{release} –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml
I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml
DockerInsecureRegistryAddress: [‘myregistry.local:8787’]
Troubleshooting
When building the overcloud-images.yaml file. I got an error
Not found image: docker://172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest
time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”
to solve, I manually downloaded and tagged the image.
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker tag registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0
[root@rhel-disconnected-registry cloud-user]# docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]# docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0
When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :
Image 172.28.117.56:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.
Available tags: 13.0, latest
For monitoring you will need to add to the overcloud-image.yaml creation:
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \
–output-env-file /home/stack/templates/overcloud_images.yaml