Disconnected Registry for Red Hat Openstack and Openshift

Hits: 278

Setup disconnected Docker registry for Rhosp

Setup disconnected Docker registry for Rhosp 13

Pre-requisites:
1.    Server for registry that has 2 nics to connect to internet and disconnected servers
2.    Director configured with latest minor release
3.    Current Redhat subscription 
4.    Hundreds of GB free, preferably a TB. It is wise to have /var as a separate mounted volume.

Read the original article that I wrote where I work at Octopus Computer Solutions

Sources for this article are below

On registry run:
$ yum update -y

Install the Offline registry and Tag the images

$ yum install -y  docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen
For the registry itself – to use after images are pushed to the offline registry
$ systemctl enable docker
$ systemctl start docker 
$ systemctl enable docker-distribution
$ systemctl start  docker-distribution


1.    ONLY for registry server: edit  /etc/docker-distribution/registry/config.yml
Edit last line that has http address add the server domain name before the  :5000
2.    Each server that uses the offline registry needs to have the following, using your domain name before :5000 this includes the registry and director
$ echo 'INSECURE_REGISTRY="--insecure-registry ol-rhel-disconnected-registry:5000"' >> /etc/sysconfig/docker

3.    Each server that uses the offline registry needs to have this.

For the offline registry server, which itself is completely online, Edit 
/etc/containers/registries.conf to the following did allow docker pull to work. 

Add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP. 

# vi /etc/containers/registries.conf

# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
 
# The default location for this configuration file is /etc/containers/registries.conf.
 
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.
 
[registries.search]
registries = ['ol-rhel-disconnected-registry:5000', 'registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
 
# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['ol-rhel-disconnected-registry:5000']
 
 
# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []

 

The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file  looks like:

$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.

[registries.search]
registries = ['ol-rhel-disconnected-registry:5000']

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['ol-rhel-disconnected-registry']

# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']

Restart Docker

# systemctl restart docker
# systemctl restart docker-distribution

On registry download, tag and push the images. This is simpler than Red Hat Openshift since all containers are standard tags
From Red Hat registry.

For testing purposes I ran this from the registry server instead of the director, I used a slightly different command to create the overcloud-images.yaml file

I ran the pull-tag-push script on the registry, then from the director I ran  the openstack command to create overcloud-images.yaml

Here are the scripts that I ran:
From registry:

#!/bin/bash
   if  test -n "$STY"
    then
    printf "This is a screen session named '$STY'.\n"

ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
  TAG=13.0
  echo "Pulling ${IMAGE_NAME}:${TAG} ..."
  docker pull ${IMAGE_NAME}:${TAG}
  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
     else printf "This is NOT a screen session.\n"
    exit 1
    fi

to create the overcloud-images.yaml file 

$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry/rhosp13 --prefix=openstack- --tag 13.0   --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml


On a real director the script and command should be:

#!/bin/bash
   if  test -n "$STY"
    then
    printf "This is a screen session named '$STY'.\n"

ASSET_SERVER=ol-rhel-disconnected-registry
docker search registry.access.redhat.com/rhosp13| awk '{print $2}' | grep ^registry.access.redhat.com | while read IMAGE_NAME
do
  TAG=$(openstack overcloud container image tag discover --image ${IMAGE_NAME} --tag-from-label {version}-{release})
  echo "Pulling ${IMAGE_NAME}:${TAG} ..."
  docker pull ${IMAGE_NAME}:${TAG}
  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)
  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}
  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest
  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}
done
     else printf "This is NOT a screen session.\n"
    exit 1
    fi

The openstack command to build the overcloud-images should be

$ openstack overcloud container image prepare --namespace=ol-rhel-disconnected-registry:5000/rhosp13 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/test-disconnect-overcloud_images.yaml


I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml 

DockerInsecureRegistryAddress: [‘myregistry.local:8787’]

Troubleshooting

When building the overcloud-images.yaml file. I got an error

Not found image: docker://ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”

to solve, I manually downloaded and tagged the image. 


[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0

ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0
[root@rhel-disconnected-registry cloud-user]#  docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:latest
[root@rhel-disconnected-registry cloud-user]#  docker push ol-rhel-disconnected-registry:5000/rhosp13/openstack-gnocchi-statsd:13.0

When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :

Image ol-rhel-disconnected-registry:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.
Available tags: 13.0, latest


For monitoring you will need to add to the overcloud-image.yaml creation:


 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \
 –output-env-file /home/stack/templates/overcloud_images.yaml

Pre-requisites:

  1. Server for registry that has 2 nics to connect to internet and disconnected servers
  2. Director configured with latest minor release
  3. Current Redhat subscription
  4. Hundreds of GB free, preferably a TB. It is wise to have /var as a separate mounted volume.

On registry run:

$ yum update -y

3.2 Install the Offline registry and Tag the images

yum install -y  docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen

For the registry itself – to use after images are pushed to the offline registry

$ systemctl enable docker

$ systemctl start docker

$ systemctl enable docker-distribution

$ systemctl start  docker-distribution

  1. ONLY for registry server edit  /etc/docker-distribution/registry/config.yml

Edit last line that has http address add the server domain name before the  :5000

  1. Each server that uses the offline registry needs to have, using your domain name before :5000 this includes the registry and director

$ echo ‘INSECURE_REGISTRY=”–insecure-registry ol-rhel-disconnected-registry:5000″‘ >> /etc/sysconfig/docker

  1. Each server that uses the offline registry needs to have, this includes the registry and director

in /etc/containers/registries.conf add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP.

# vi /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000′, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = []

The same file /etc/containers/registries.conf on the servers looks like:

$ cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

COPY it to the other servers

ansible others -m  copy -a  “src=/etc/containers/registries.conf  dest=/etc/containers/registries.conf”

Restart Docker

# systemctl restart docker

# systemctl restart docker-distribution

On registry download, tag and push the images. This is simpler than openshift since all containers are standard tags

From

https://access.redhat.com/articles/3348761

As in https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/architecture.html

For the offline server, which itself is completely online, Edit

/etc/containers/registries.conf to the following did allow docker pull to work

# cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = []

The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file  looks like:

$ cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

COPY it to the other servers

ansible others -m  copy -a  “src=/etc/containers/registries.conf  dest=/etc/containers/registries.conf”

Restart Docker

# systemctl restart docker

# systemctl restart docker-distribution

Since we did NOT do a complete install of same minor release with the director, I used a slightly different command to create the overcloud-images.yaml file

I ran the pull-tag-push script on the registry, then from the director I ran  the openstack command to create overcloud-images.yaml

Here are the scripts that I ran:

From registry:

#!/bin/bash

#from:https://access.redhat.com/articles/3348761

   if  test -n “$STY”

    then

    printf “This is a screen session named ‘$STY’.\n”

ASSET_SERVER=172.28.117.56

docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME

do

  TAG=13.0

  echo “Pulling ${IMAGE_NAME}:${TAG} …”

  docker pull ${IMAGE_NAME}:${TAG}

  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)

  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}

  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest

  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}

done

     else printf “This is NOT a screen session.\n”

    exit 1

    fi

to create the overcloud-images.yaml file

openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag 13.0   –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml

On a real director the script and command should be:

#!/bin/bash

#from:https://access.redhat.com/articles/3348761

   if  test -n “$STY”

    then

    printf “This is a screen session named ‘$STY’.\n”

ASSET_SERVER=172.28.117.56

docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME

do

  TAG=$(openstack overcloud container image tag discover –image ${IMAGE_NAME} –tag-from-label {version}-{release})

  echo “Pulling ${IMAGE_NAME}:${TAG} …”

  docker pull ${IMAGE_NAME}:${TAG}

  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)

  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}

  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest

  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}

done

     else printf “This is NOT a screen session.\n”

    exit 1

    fi

The openstack command to build the overcloud-images should be

openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag-from-label {version}-{release} –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml

I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml

https://bugzilla.redhat.com/show_bug.cgi?id=1500507

DockerInsecureRegistryAddress: [‘myregistry.local:8787’]

Troubleshooting

When building the overcloud-images.yaml file. I got an error

Not found image: docker://172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”

to solve, I manually downloaded and tagged the image.

[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0

[root@rhel-disconnected-registry cloud-user]#  docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

[root@rhel-disconnected-registry cloud-user]#  docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0

When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :

Image 172.28.117.56:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.

Available tags: 13.0, latest

For monitoring you will need to add to the overcloud-image.yaml creation:

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \

 –output-env-file /home/stack/templates/overcloud_images.yaml