Teaser

Late last night (don’t ask how late it was) I discovered a really slick trick, aka “feature”, of docker-compose. Full disclosure: I love docker-compose “overrides”, a feature I found a couple of months ago. However, implementing overrides in a granular fashion, as I’d like, and within a docker-compose hierarchy of environments like the ISLE stack, can be difficult and counter-productive. I may have found a workable compromise last night. If you’re interested, please, read on…

History

About a year ago I created a “portable” FEDORA object repository “on a stick”, a USB stick. I call that “stick” DG-FEDORA, because it contains a number of sample objects gleaned from Digital.Grinnell.edu. Shortly after I created the “stick” I started using a docker-compose.override.yml file (see Adding and overriding configuration for details) to implement it in some of my ISLE work. But there were lots of other “customizations” that I also wanted to implement, and over time my docker-compose.override.yml grew too large to be easily maintained…there was just too much stuff to squeeze into a file that was supposed to make life easier.

ISLE Details

The ISLE stack now uses a very simple and elegant .env file – four lines of environment variables, nothing more – to ultimately control what the docker-compose command does. The last environment variable in the file is COMPOSE_FILE, and I found that this variable can be used to specify “multiple” .yml files to build a stack; and docker-compose does so using a well-documented and sensible set of rules. Nirvana!

The Solution

So what I’ve done is simply provide a modified .env file and some instruction for using it properly. My .env file (there is a copy of it on the DG-FEDORA USB stick) reads like this:

#### Activated ISLE environment
# To use an environment other than the default Demo, please change values below
# from the default Demo to one of the following: Local, Test, Staging or Production
# For more information, consult https://islandora-collaboration-group.github.io/ISLE/install/install-environments/

COMPOSE_PROJECT_NAME=isle_demo
BASE_DOMAIN=isle.localdomain
CONTAINER_SHORT_ID=ld
# COMPOSE_FILE=docker-compose.demo.yml
COMPOSE_FILE=docker-compose.demo.yml:docker-compose.DG-FEDORA.yml

In the file above, note that I’ve commented out the original definition of COMPOSE_FILE and added my own definition which appends a second .yml file reference to the original. That 2nd file, docker-compose.DG-FEDORA.yml is also provided on the DG-FEDORA USB stick and it reads like this:

version: '3.7'

#### docker-compose up -d;
## Local Environment - Used for Drupal site development, more extensive metadata, Solr and Fedora development work on a local laptop or workstation
## Updated 2019-09 - Release 1.3.0-dev (@ 1.3.0-dev)

services:

  # Bind-mount the datastreamStore and objectStore directories in /Volumes/DG-FEDORA (the USB stick)
  #  into the FEDORA container to become our FEDORA datastream and object store directories.  FEDORA digital
  #  live in these directories, so this effectively puts your FEDORA repository on the DG-FEDORA stick.

  fedora:
    volumes:
      - /Volumes/DG-FEDORA/datastreamStore:/usr/local/fedora/data/datastreamStore
      - /Volumes/DG-FEDORA/objectStore:/usr/local/fedora/data/objectStore

  # Bind-mount the /Volumes/DG-FEDORA/Storage directory tree in the Apache container. This bind-mount to /mnt/storage
  #  is usefule with IMI, the Islandora Multi-Importer when a local* file fetch hook is defined, as it is in the DG7
  #  custom module.

  apache:
    volumes:
      - /Volumes/DG-FEDORA/Storage:/mnt/storage

Simple. It’s just a repeat of the original docker-compose.demo.yml file with only the necessary “overrides” included.

The DG-FEDORA USB stick also includes a README.md file and since you don’t have a copy of the “stick”, I’ll share that with you here.

A Master Copy of DG-FEDORA

My Digital.Grinnell staging server, DGDockerX is a networked CentOS 7 virtual machine with lots of available storage, but no accessible USB ports, of course…it’s a VM after all. Consequently, it’s not practical to maintain all of my “portable” FEDORA data on a USB stick alone. So, on DGDockerX I’ve created a copy of the DG-FEDORA USB stick at /mnt/data/DG-FEDORA, and I’ve declared it to be copy “zero”, so on it there’s a DG-FEDORA-0.md file proclaiming it to be the “master” copy of DG-FEDORA.

It’s worth noting, perhaps, that at the time of this writing, DG-FEDORA-0 contains 126 FEDORA objects. Anything less than that number should be considered an incomplete set.

The README.md file in the DG-FEDORA-0 MASTER volume suggests using a command like this to copy the MASTER repository to a mounted DG-FEDORA USB stick:

sudo rsync -aruvi --exclude DG-FEDORA-0.md --exclude from-DGDocker1 --exclude site-public islandora@132.161.132.101:/mnt/data/DG-FEDORA/. /Volumes/DG-FEDORA/. --progress

README.md from DG-FEDORA

DG-FEDORA is the name of a USB stick/volume (I actually maintain 2 USB memory sticks named DG-FEDORA, numbers 1 and 2) which holds a small sample of FEDORA digital objects gleaned from Digital.Grinnell. Using DG-FEDORA you can easily add a pre-populated repository of objects to your “demo” or “local” environment ISLE project in as little as 5 minutes.

Prerequisites

To successfully use DG-FEDORA your system will need to meet the following prerequisite requirements.

  • Your workstation must meet all the hardware requirements of ISLE. See ISLE’s ./docs/install/host-hardware-requirements.md for details.
  • Your workstation environment must meet all the minimum software requirements of ISLE. See ISLE’s ./docs/install/host-software-dependencies.md for details.
  • This workflow assumes your workstation is running OSX. Other workstation types supported by ISLE may be acceptable, but the DG-FEDORA USB volume will have to be mounted differently.
  • You must have a working “demo” or “local” ISLE environment already running.

Mounting DG-FEDORA

In MacOSX you simply have to plug the DG-FEDORA USB stick into an available USB port on your Mac workstation. After a few seconds the USB stick will be automatically mounted as /Volumes/DG-FEDORA, and the /Volumes directory is automatically “shared” with Docker.

Make DG-FEDORA Writable

In MacOSX, open a terminal and use the following command to make DG-FEDORA writable, so that you can save newly ingested objects and updates to existing objects:

  • sudo mount -u -w /Volumes/DG-FEDORA

Not Using MacOSX?

If your workstation is not a Mac, you will need to insert the USB stick and take appropriate steps to:

  • Mount the stick with read/write permissions as /Volumes/DG-FEDORA.
  • Share the /Volumes or /Volumes/DG-FEDORA directory in your Docker settings/preferences.

Modifying the ISLE Environment

Navigate to your ISLE project directory and execute the following operations:

  • Shut down ISLE if it is running. Example: cd ~/pathto/ISLE; docker-compose down. These commands will do no harm if ISLE is not currently running.
  • Copy two files from DG-FEDORA to your local project. Example: cd /Volumes/DG-FEDORA; cp -f .env docker-compose.DG-FEDORA.yml ~/pathto/ISLE/.
  • Edit the new ISLE project .env file according to directions within the file. Example: nano ~/pathto/ISLE/.env. The objective is to select the appropriate “demo” or “local” environment as needed.
  • Navigate to your project directory and restart the stack. Example: cd ~/pathto/ISLE; docker-compose up -d.
  • Wait until the stack has started, open your browser and visit your site at https://isle.localdomain (demo) or https://yourprojectnamehere.localdomain (local).

Rebuild FEDORA’s resourceIndex

Rebuild your FEDORA resourceIndex using the steps documented in Step 17: On Remote Production - Re-Index Fedora & Solr.

  • Open a terminal in the isle-fedora-ld container, docker exec -it isle-fedora-ld bash, and then run cd utility_scripts/; ./rebuildFedora.sh.

Rebuild the Solr Index

Once the previous rebuild process is complete, you should rebuild your Solr search index using the remaining steps documented in Step 17: On Remote Production - Re-Index Fedora & Solr.

  • Open a terminal in the isle-fedora-ld container, docker exec -it isle-fedora-ld bash (or using the terminal opened in the previous step), and then run cd utility_scripts/; ./updateSolrIndex.sh.

This rebuilding process may take a few minutes. Proceed to the check your work after some minutes have passed.

Check Your Work

Feedback is Welcome

This is very much a work-in-progress. If you have an opinion, or would like to suggest improvements, please share!

That’s a wrap… until next time. 😄