Notes from setting up Disconnected Docker registry

Hits: 1129

Clients concerned with security like financial institutions need to have their containers pulled from an internal Docker registry

Setup disconnected Docker registry for Rhosp

Pre-requisites:

On registry run:

$ yum update -y

3.2 Install the Offline registry and Tag the images

yum install -y  docker-distribution docker yum-utils docker git wget git net-tools bind- bridge-utils bash-completion kexec-tools sos screen

For the registry itself – to use after images are pushed to the offline registry

$ systemctl enable docker

$ systemctl start docker

$ systemctl enable docker-distribution

$ systemctl start  docker-distribution

Edit last line that has http address add the server domain name before the  :5000

$ echo ‘INSECURE_REGISTRY=”–insecure-registry ol-rhel-disconnected-registry:5000″‘ >> /etc/sysconfig/docker

in /etc/containers/registries.conf add IP and port of insecure registries as well as registries. Remember to put apostrophes around inside the square brackets, as below. Replace ‘ol-rhel-disconnected-registry’ with YOURDOMAIN or IP.

# vi /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000′, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = []

The same file /etc/containers/registries.conf on the servers looks like:

$ cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

COPY it to the other servers

ansible others -m  copy -a  “src=/etc/containers/registries.conf  dest=/etc/containers/registries.conf”

Restart Docker

# systemctl restart docker

# systemctl restart docker-distribution

On registry download, tag and push the images. This is simpler than openshift since all containers are standard tags

From

https://access.redhat.com/articles/3348761

As in https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/architecture.html

For the offline server, which itself is completely online, Edit

/etc/containers/registries.conf to the following did allow docker pull to work

# cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’, ‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = []

The same file /etc/containers/registries.conf on the servers/director should have the public servers removed from search and even moved to the block setting, so they wont be tried by default. The file  looks like:

$ cat /etc/containers/registries.conf

# This is a system-wide configuration file used to

# keep track of registries for various container backends.

# It adheres to TOML format and does not support recursive

# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: ‘registries.search’, ‘registries.insecure’,

# and ‘registries.block’.

[registries.search]

registries = [‘ol-rhel-disconnected-registry:5000’]

# If you need to access insecure registries, add the registry’s fully-qualified name.

# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.

[registries.insecure]

registries = [‘ol-rhel-disconnected-registry’]

# If you need to block pull access from a registry, uncomment the section below

# and add the registries fully-qualified name.

#

# Docker only

[registries.block]

registries = [‘registry.access.redhat.com’, ‘docker.io’, ‘registry.fedoraproject.org’, ‘quay.io’, ‘registry.centos.org’]

COPY it to the other servers

ansible others -m  copy -a  “src=/etc/containers/registries.conf  dest=/etc/containers/registries.conf”

Restart Docker

# systemctl restart docker

# systemctl restart docker-distribution

Since we did NOT do a complete install of same minor release with the director, I used a slightly different command to create the overcloud-images.yaml file

I ran the pull-tag-push script on the registry, then from the director I ran  the openstack command to create overcloud-images.yaml

Here are the scripts that I ran:

From registry:

#!/bin/bash

#from:https://access.redhat.com/articles/3348761

   if  test -n “$STY”

    then

    printf “This is a screen session named ‘$STY’.\n”

ASSET_SERVER=172.28.117.56

docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME

do

  TAG=13.0

  echo “Pulling ${IMAGE_NAME}:${TAG} …”

  docker pull ${IMAGE_NAME}:${TAG}

  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)

  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}

  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest

  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}

done

     else printf “This is NOT a screen session.\n”

    exit 1

    fi

to create the overcloud-images.yaml file

openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag 13.0   –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml

On a real director the script and command should be:

#!/bin/bash

#from:https://access.redhat.com/articles/3348761

   if  test -n “$STY”

    then

    printf “This is a screen session named ‘$STY’.\n”

ASSET_SERVER=172.28.117.56

docker search registry.access.redhat.com/rhosp13| awk ‘{print $2}’ | grep ^registry.access.redhat.com | while read IMAGE_NAME

do

  TAG=$(openstack overcloud container image tag discover –image ${IMAGE_NAME} –tag-from-label {version}-{release})

  echo “Pulling ${IMAGE_NAME}:${TAG} …”

  docker pull ${IMAGE_NAME}:${TAG}

  IMAGE_PATH=$(echo ${IMAGE_NAME} | cut -f2- -d/)

  docker tag ${IMAGE_NAME}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG}

  docker tag ${ASSET_SERVER}:5000/${IMAGE_PATH}:${TAG} ${ASSET_SERVER}:5000/${IMAGE_PATH}:latest

  docker push ${ASSET_SERVER}:5000/${IMAGE_PATH}

done

     else printf “This is NOT a screen session.\n”

    exit 1

    fi

The openstack command to build the overcloud-images should be

openstack overcloud container image prepare –namespace=172.28.117.56:5000/rhosp13 –prefix=openstack- –tag-from-label {version}-{release} –output-env-file=/home/stack/test-disconnect-overcloud_images.yaml

I am unsure whether DockerInsecureRegistryAddress needs to be manually added to overcloud-images.yaml

clinet

https://bugzilla.redhat.com/show_bug.cgi?id=1500507

DockerInsecureRegistryAddress: [‘myregistry.local:8787’]

Troubleshooting

When building the overcloud-images.yaml file. I got an error

Not found image: docker://172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

time=”2019-03-12T09:40:48+02:00″ level=fatal msg=”Error reading manifest latest in 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd: manifest unknown: manifest unknown”

to solve, I manually downloaded and tagged the image.

[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

[root@rhel-disconnected-registry cloud-user]#  docker tag  registry.redhat.io/rhosp13/openstack-gnocchi-statsd:13.0 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0

[root@rhel-disconnected-registry cloud-user]#  docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:latest

[root@rhel-disconnected-registry cloud-user]#  docker push 172.28.117.56:5000/rhosp13/openstack-gnocchi-statsd:13.0

When building the overcloud-images.yaml file, If you get the following error, you will need to either label each image as told by error or change the tag variable to 13.0 as in the first script :

Image 172.28.117.56:5000/rhosp13/openstack-aodh-notifier has no tag 13.0-66.

Available tags: 13.0, latest

For monitoring you will need to add to the overcloud-image.yaml creation:

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/fluentd-client.yaml \

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sensu-client.yaml \

 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/collectd.yaml \

 –output-env-file /home/stack/templates/overcloud_images.yaml