Configuring DGDocker2 // The Grinnell College Digital Library Application Developer's Blog

My mission today is to successfully migrate the images/containers/services chronicled in post 030, “Dockerized Omeka-S: Starting Over” to Docker-ready node dgdocker2 without compromising any of the services that already run there.

Pushing WMI Omeka-S to Production on dgdocker2

Grinnell’s dgdocker2 server, specifically dgdocker2.grinnell.edu with an IP address of 132.161.132.143, is a Docker-ready CentOS 7 node that’s currently supporting the following containers and configuration:

╭─root@dgdocker2 ~
╰─# docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
ef20d71ffea8        mcfatem/ohscribe               "./boot.sh"              6 days ago          Up 6 days           5000/tcp                                                           ohscribe
b525f4670cd2        mariadb:latest                 "docker-entrypoint.s…"   2 weeks ago         Up 2 weeks          3306/tcp                                                           omekasdocker_mariadb_1
7f107a24c204        traefik:latest                 "/traefik --docker -…"   2 weeks ago         Up 2 weeks          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8080->8080/tcp   traefik_proxy
9282ab53ecc4        portainer/portainer:latest     "/portainer --admin-…"   5 weeks ago         Up 5 weeks          0.0.0.0:9000->9000/tcp                                             portainer
60ce06301101        dodeeric/omeka-s:latest        "docker-php-entrypoi…"   7 weeks ago         Up 7 weeks          80/tcp                                                             omekasdocker_omeka_1
54bd82694f3c        phpmyadmin/phpmyadmin:latest   "/docker-entrypoint.…"   2 months ago        Up 2 months         80/tcp                                                             omekasdocker_pma_1
0cd019c5456e        emilevauge/whoami              "/whoamI"                2 months ago        Up 2 months         80/tcp                                                             omekasdocker_whoami_1
7b3d4961ec21        v2tec/watchtower               "/watchtower"            2 months ago        Up 2 months                                                                            watchtower

Grinnell’s DNS is configured with the following addresses pointed to dgdocker2:

https://textline.grinnell.edu - 404 page not found
https://portainer2.grinnell.edu - 404 page not found
https://omeka-s.grinnell.edu - Multi-Cultural Reunion 2019 home page
https://traefik2.grinnell.edu - 404 page not found
https://rootstalkx.grinnell.edu - 404 page not found
https://ohscribe.grinnell.edu - OHScribe home page
https://dgdocker2.grinnell.edu - 404 page not found

The information following each address is the status or page returned when I tried opening each on 3-September-2019.

The https://omeka-s.grinnell.edu on dgdocker2 is experimental (at least it was in August 2019) and soon-to-be-replaced with our new Omeka-S. Consequently, the only properly configured service on this node is OHScribe, and the Traefik container is properly configured to serve it as well as the experimental Omeka-S instance. All of the other containers/services should be removed, and the new Omeka-S with WMI configured to work with the existing Traefik.

Since nearly all of the containers/services running on dgdocker2 are broken or obsolete, I’m going to remove them all and clean up the node using this sequence as a copy/paste one-liner…

docker stop $(docker ps -q); docker rm -v $(docker ps -qa); docker image rm -f $(docker image ls -q); docker system prune --force;

Deploying a Stand-Alone Traefik Reverse-Proxy

There are at least a dozen ways to do this, and I really don’t want to reinvent the wheel here, so I searched the web for some of the latest info and settled on this post from DigitalOcean. It’s current, I like DigitalOcean’s approach in general, and it appears to be well-documented.

Perhaps the best of Traefik’s qualities is its ability to support additional services/containers using labels. Let’s roll with that. The plan here is to turn dgdocker2 into the home for many Omeka-S instances with the server answering to https://omeka-s.grinnell.edu.

One-Time/Preliminary Stuff

I’m starting now with a “clean”, Docker-ready node in dgdocker2. From a terminal/shell opened as root on dgdocker2 we need some preliminary stuff:

# Create a home on dgdocker2 for the project
mkdir -p /opt/traefik
cd /opt/traefik
nano traefik.toml

The traefik.toml file should look like this:

defaultEntryPoints = ["http", "https"]

# CA server to use
# Uncomment the line to run on the staging Let's Encrypt server
# Leave comment to go to prod
#
# Optional
#
caServer = "https://acme-staging.api.letsencrypt.org/directory"

[entryPoints]
  [entryPoints.dashboard]
    address = ":8080"
    [entryPoints.dashboard.auth]
      [entryPoints.dashboard.auth.basic]
        users = ["admin:$2y$05$pJEzHJBzfoYYS7/hGAedcOP8XdsqNXE7j.LHFBVjueASOqOvvjGOy"]
  [entryPoints.http]
    address = ":80"
      [entryPoints.http.redirect]
        entryPoint = "https"
  [entryPoints.https]
    address = ":443"
      [entryPoints.https.tls]
        minVersion = "VersionTLS12"
        cipherSuites = [
          "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
          "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
          "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
         ]

[api]
entrypoint="dashboard"

[acme]
### email = "your_email@your_domain"
email = "digital@grinnell.edu"
storage = "acme.json"
entryPoint = "https"
onHostRule = true
  [acme.httpChallenge]
  entryPoint = "http"

[docker]
### domain = "your_domain"
domain = "omeka-s.grinnell.edu"
watch = true
network = "web"

Note: The 11 lines, including “minVerson” and “cipherSuites” definitions, which appear in the “[entryPoints.https.tls]” section above were lifted from “Removing Traefik’s Weak Cipher Suites”.

The “preliminary” steps above, and the creation of the traefik.toml file should NOT be repeated, they are good-to-go!

Launching Traefik

# Clean up first!
docker stop $(docker ps -q); docker rm -v $(docker ps -qa); # docker image rm -f $(docker image ls -q); docker system prune --force;
# rm -f /opt/traefik/acme.json    # probably not necessary?
# Create the "web" network
docker network create web
# Create a home on dgdocker2 for the project... if one does not already exist
mkdir -p /opt/traefik
cd /opt/traefik
# Setup for Let's Encrypt certs
touch acme.json
chmod 600 acme.json
# Launch Traefik
docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $PWD/traefik.toml:/traefik.toml \
  -v $PWD/acme.json:/acme.json \
  -p 80:80 \
  -p 443:443 \
  -l traefik.frontend.rule=Host:traefik2.grinnell.edu \
  -l traefik.port=8080 \
  --network web \
  --name traefik \
  traefik:1.7.14-alpine

Ok, let’s see what we’ve got…

Eureka! https://traefik2.grinnell.edu returns an admin login prompt for my new Traefik instance at https://traefik2.grinnell.edu/dashboard/, as promised, and it’s complete with a green lock icon indicating that we have a valid TLS cert for it. Presumably this Traefik will have NO weak ciphers or vulnerabilities. Note to self: Test this assumption!

Let’s Add Portainer

In addition to the Treafik dashboard, I like having Portainer available to help with stack management too. So, let’s add that using docker-compose and an appropriately modified version of the guidance provided in Step 3 - Registering Containers with Traefik.

The aforementioned guidance wants us to create a new project directory (optional: we could use the /opt/traefik directory we already have) and populate it with a docker-compose.yml file with contents like this:

version: "3"

networks:            # This "networks" section is key.  "web" refers to our already-running Docker network
  web:
    external: true
  internal:
    external: false

services:
  blog:
    image: wordpress:4.9.8-apache
    environment:
      WORDPRESS_DB_PASSWORD:
    labels:
      - traefik.backend=blog
      - traefik.frontend.rule=Host:blog.your_domain
      - traefik.docker.network=web
      - traefik.port=80
    networks:
      - internal
      - web
    depends_on:
      - mysql
  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD:
    networks:
      - internal
    labels:
      - traefik.enable=false
  adminer:
    image: adminer:4.6.3-standalone
    labels:
      - traefik.backend=adminer
      - traefik.frontend.rule=Host:db-admin.your_domain
      - traefik.docker.network=web
      - traefik.port=8080
    networks:
      - internal
      - web
    depends_on:
      - mysql

The Portainer configuration that I like to use was derived from the docker-compose.demo.yml file in the ISLE project, and it typically looks something like this:

version: "3"

#### docker-compose up -d

networks:
  web:
    external: true    ## Connect to the existing "web" network!
  internal:
    external: false

services:
  portainer2:     ## Renamed to avoid conflicts on systems/servers with portainer already running.
    image: portainer/portainer
    container_name: portainer2
    command: -H unix:///var/run/docker.sock --no-auth   ## Swap this out with an auth challenge for security!
    networks:
      - web
      - internal
    ports:
      - "9010:9000"     ## Remapped to avoid conflicts on systems/servers with portainer already running.
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer-data:/data
    labels:
      - traefik.port=9000
      - traefik.docker.network=web     ## Another critical reference to the "web" network
      - traefik.enable=true
      - "traefik.frontend.rule=Host:portainer2.grinnell.edu;"

volumes:
  portainer-data:

I built this content into a new /opt/portainer/docker-compose.yml file and subsequently launched Portainer like so:

cd /opt/portainer
docker-compose up -d

Visiting https://portainer2.grinnell.edu in my browser shows that it works and has a valid TLS cert too!

Securing Portainer Auth

The previous outcome is great, but there are at least 3 issues that need to be dealt with. The first issue is Portainer authentication. My initial spin of Portainer, above, is an “unprotected” instance. Anyone can currently visit https://portainer2.grinnell.edu and see what the stack looks like there. Not good. The culprit is the last line shown in this snippet from our docker-compose.yml file:

services:
  portainer2:
    ...
    command: -H unix:///var/run/docker.sock --no-auth

The remedy is to preserve the indentation, that’s critical in a .yml file, but change that line to read:

    command: --admin-password "$$2y$$05$$pJEzHJBzfoYYS7/hGAedcOP8XdsqNXE7j.LHFBVjueASOqOvvjGOy" -H unix:///var/run/docker.sock

The hash following “–admin-password” is one I generated for my own use with a htpasswd -nb admin... command as documented in Step 1 — Configuring and Running Traefik. Important: Note that in this instance every single dollar sign ($) is DOUBLED and the hash appears in double quotes!

Visiting https://portainer2.grinnell.edu again and this time the Portainer interface is behind an authentication login pop-up. Nice!

Switching to Subdirectory Addressing

Now we have https://traefik2.grinnell.edu and https://portainer2.grinnell.edu both working properly on dgdocker2 in what I call a “sub-second-top” domain name structure. I so named this structure because it follows the convention documented in The Parts of a URL: A Short & Sweet Guide. That blog post identifies the parts of a URL as:

scheme://subdomain.second-level.top-level/subdirectory

In case you didn’t pick up on it, the “2” at the end of each subdomain reflects the fact that the server, or host, is named dgdocker2.

While these addresses are fine, they require considerable coordination with the folks who manage our DNS names; I enlisted their help months ago to “create” the two addresses we now have. To avoid having to coordinate every change I’d like to change things up and identify this server, and the services that run on it, to the world in a form like:

https://omeka-s.grinnell.edu/something

This implies that Traefik should respond at https://omeka-s.grinnell.edu/traefik, and Portainer at https://omeka-s.grinnell.edu/portainer. Likewise, our first Omeka-S site, World Music Instruments, or WMI, should respond at https://omeka-s.grinnell.edu/wmi.

I’ve already asked our DNS managers to make https://omeka-s.grinnell.edu resolve to our dgdocker2 host, so all that’s necessary now is a change in some of our Traefik labels to specify a different URL structure. Specifically, we need to change the expression in our docker-compose.yml “traefik.frontend.rule” label from this:

- "traefik.frontend.rule=Host:portainer2.grinnell.edu;"

…to this set of configuration labels:

- "traefik.frontend.rule=PathPrefixStrip:/portainer"
- "traefik.frontend.redirect.regex=^(.*)/portainer$$"
- "traefik.frontend.redirect.replacement=$$1/portainer/"
- "traefik.frontend.rule=PathPrefix:/portainer;ReplacePathRegex: ^/portainer/(.*) /$$1"

This nice example was taken verbatim from Using labels in docker-compose.yml. After editing these changes into /opt/portainer/docker-compose.yml I did a new cd /opt/portainer; docker-compose up -d and…

Now, visiting https://omeka-s.grinnell.edu/portainer brings my authentication-protected Portainer interface up as planned. However, my TLS cert for this domain is not valid yet. Hmmm, wonder why that is? In any case… this is progress!

Moving Traefik to a Subdirectory

I’m going to make the same kind of changes for Traefik now, but this time the modifications are to the docker run... command that I use to launch it. Some trial, and lots of errors, lead me to this new docker run... command syntax:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $PWD/traefik.toml:/traefik.toml \
  -v $PWD/acme.json:/acme.json \
  -p 80:80 \
  -p 443:443 \
  -l traefik.frontend.rule=PathPrefixStrip:/traefik \
  -l traefik.frontend.redirect.regex='^(.*)/traefik$' \
  -l traefik.frontend.redirect.replacement=$1/traefik/ \
  -l traefik.port=8080 \
  --network web \
  --name traefik \
  traefik:1.7.14-alpine

Unfortunately, just like Portainer, my new Traefik address failed to get a valid TLS cert. 😦

Who Am I

Everything on the dgdocker2 server responds to a “subdirectory” address now and there’s nothing registered at https://omeka-s.grinnell.edu. To help eliminate the possibility that this is a problem I’m going to try adding a WhoAmI service, at the aforementioned address, using the configuration documented in this simple and straightforward repo.

# Create a home on dgdocker2 for the project... if one does not already exist
cd /opt
git clone https://github.com/lukasnellen/dc-whoami.git whoami
cd /opt/whoami
# Edit the docker-compose.yml file as needed
nano docker-compose.yml    # see completed edits below

Continuing after edits to /opt/whoami/docker-compose.yml…

docker-compose --log-level DEBUG up -d

Now, if I visit https://omeka-s.grinnell.edu I can see that the WhoAmI is working, but again, it does not have a valid cert. 😦

Investigating Invalid Certs

In an attempt to determine why my certs are not valid, I found Debugging Let’s Encrypt Errors, Sometimes It’s Not Your Fault. From my workstation I tried some of the suggestions in the post and got these results:

╭─mark@Marks-Mac-Mini ~/Projects/blogs-McFateM ‹master*›
╰─$ host omeka-s.grinnell.edu
omeka-s.grinnell.edu has address 132.161.132.143
╭─mark@Marks-Mac-Mini ~/Projects/blogs-McFateM ‹master*›
╰─$ host omeka-s.grinnell.edu 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

omeka-s.grinnell.edu has address 132.161.132.143
╭─mark@Marks-Mac-Mini ~/Projects/blogs-McFateM ‹master*›
╰─$ curl -k https://omeka-s.grinnell.edu
Hostname: 67c6f570dc5b
IP: 127.0.0.1
IP: 192.168.80.3
IP: 192.168.96.2
GET / HTTP/1.1
Host: omeka-s.grinnell.edu
User-Agent: curl/7.54.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 173.18.136.80
X-Forwarded-Host: omeka-s.grinnell.edu
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 34e5bc377410
X-Real-Ip: 173.18.136.80

These results make me believe that our DNS entries are NOT the problem. That leaves me believing that I’ve probably hit a Let’s Encrypt rate limit. 😦 So, moving on, I’m going to accept the invalid certs and just try to get Omeka-S up and running.

Who Am I

That didn’t help with the invalid certs issue. So, now I’m thinking the problem here is that nothing is registered at https://omeka-s.grinnell.edu; everything lives in subdirectory paths “below” that subdomain. Based on that hunch, I’m going to try adding a WhoAmI service, at the aforementioned address, using the configuration documented in this simple and straightforward repo.

# Create a home on dgdocker2 for the project... if one does not already exist
cd /opt
git clone https://github.com/lukasnellen/dc-whoami.git whoami
cd /opt/whoami
# Edit the docker-compose.yml file as needed
nano docker-compose.yml    # see completed edits below

Continuing after edits to /opt/whoami/docker-compose.yml…

docker-compose --log-level DEBUG up -d

Now, if I visit https://omeka-s.grinnell.edu I can see that the WhoAmI is working, but again, it does not have a valid cert.

Capture As a Project

I like the direction this server setup has taken, apart from the invalid certs issue 😦, so I’m taking steps to formally “capture” this setup. I will chronicle that process in My dockerized-server Config.

Back to Omeka-S Configuration

Having wrapped up My dockerized-server Config, I’m back to finally get Omeka-S configured on dgdocker2. Unfortunately, while configuring this final spin of Omeka-S I ran short on time and failed to document every step. However, the outcome is working nicely at dgdocker2:/opt and is captured in a new GitHub repo at McFateM/omeka-s-dgdocker2.

This repo includes:

A dockerized-server component where Traefik, Portainer and Who Am I are configured;
A solr component Solr is configured;
An omeka-s-docker component where Omeka-S, MariaDB, and PHPMyAdmin (PMA) are configured;
A docker-reset.sh script that can be used to reset the host’s Docker; and
A launch-stack.sh script that can be used to reset Docker and then re-start the entire stack.

Persistence

As currently configured, the stack maintains persistent Omeka site data in a Docker volume (NOT a “bind mount”, but a named volume managed by Docker). There is a comment line in the docker-reset.sh that can be enabled to wipe the aforementioned volume clean; use it with extreme caution! There’s also a comment line in omeka-s-docker/docker-compose.yml that can be enabled to re-initialize the omeka database with a backup of the original World Music Instruments site on server omeka1.

Launch and Addressing

I did a git clone https://github.com/DigitalGrinnell/omeka-s-dgdocker2 /opt and then source /opt/launch-stack.sh from a root terminal/shell on dgdocker2. The result is this working set of services and addresses:

Service	Address	Note
Traefik dashboard	https://traefik2.grinnell.edu	Requires authentication
Portainer dashboard	https://omeka-s.grinnell.edu/portainer/	Requires authentication
Who Am I	https://omeka-s.grinnell.edu/who-am-i
Solr administration	https://portainer2.grinnell.edu	Temporary. Requires authentication
MariaDB administration	None	See ./pma/ below
PHPMyAdmin	https://omeka-s.grinnell.edu/pma/	Trailing slash is REQUIRED
Omeka-S	https://omeka-s.grinnell.edu

Addressing Update

Today, 11-Sep-2019, I got word that my DNS requests for subdomain names solr2.grinnell.edu and pma2.grinnell.edu were completed. So this morning I made necessary changes to dgdocker2:/opt/omeka-s-docker/docker-compose.yml and did a new docker-compose up -d in that directory. It worked, and I didn’t even have to take the stack down and restart it!

So, we now have this updated, and final, addressing scheme:

Service	Address	Note
Traefik dashboard	https://traefik2.grinnell.edu	Requires authentication
Portainer dashboard	https://omeka-s.grinnell.edu/portainer/	Requires authentication
Who Am I	https://omeka-s.grinnell.edu/who-am-i
Solr administration	https://solr2.grinnell.edu	No longer temporary
MariaDB administration	None	See pma2.grinnell.edu below
PHPMyAdmin	https://pma2.grinnell.edu	No longer temporary. Behaving properly
Omeka-S	https://omeka-s.grinnell.edu

Now with Valid Certs!

Earlier in this process we learned that Solr won’t work properly without a valid TLS certificate, not a temporary one. So I was forced to move our certificate authority server spec from Let’s Encrypt “stagging” back to “production” (see the ./dockerized-server/data/traefik.toml file for details). Fortunately, when I ran our addressing update (see section above) the production certs obtained from Let’s Encrypt were valid this time. Woot! I guess that means that our recent rate-limit induced ban had expired? It also means I can now close the books on this project. Double woot!

And that’s a wrap… until next time.