Staging Digital.Grinnell (DG) on DigitalOcean (DO)

ISLE v1.3.0 has been running on my staging server, DGDockerX, for months now and it seems to be performing as-expected with one exception… when I try to import a batch of objects using IMI, the Islandora Multi-Importer, I get the following error:

The website encountered an unexpected error. Please try again later.

Examinations of Recent log messages seem to point to DNS issues that I’m unable to overcome because I have no control over our DNS records, campus networking, or firewalls. So this post is intended to chronicle steps I’m taking to stand-up an instance of dg-isle and dg-islandora on a “clean” DigitalOcean droplet, namely summitt-services-droplet-01, that I have been leasing for the past couple of years.

Directories

I’ll begin by opening a terminal on my workstation/host and subsequently a terminal into the aforementioned droplet where I’ll login as user centos with a UID of 1000. Once I’m in there I’ll also clone the two aformentioned GitHub repositories to the DO host, like so:

Next, I’ll attempt to rsync the contents of my DG-FEDORA directory from DGDockerX (IP address: 132.161.132.101), my original staging server, to the aforementioned DigitalOcean droplet/host (public IP address: 165.227.83.186).

Note that the above rsync command DID NOT WORK, so I tried reversing the process by opening a VPN connection to Grinnell College, opening a terminal on DGDockerX, and “pushing” the files to summitt-services-droplet-01, but this also FAILED, presumably due to network/communications issues. So, ultimately I used a series of rsync commands to “pull” the files to my local workstation, then “push” them out to summitt-services-droplet-01.

So, in the end, the contents of dgdockerx.grinnell.edu:/mnt/data/DG-FEDORA were copied to summitt-services-droplet-01:/mnt/data/DG-FEDORA where the directory looks like this:

drwxrwxr-x.  10 centos centos  275 Feb 11 13:37 .
drwxrwxrwx.   3 root   root     23 Feb 10 10:19 ..
drwxrwxrwx. 236 centos centos 4.0K Dec 17 14:53 datastreamStore
-rwxrwxrwx.   1 centos centos  211 Dec 23 11:15 DG-FEDORA-0.md
-rwxrwxrwx.   1 centos centos 1.1K Oct  5 09:36 docker-compose.DG-FEDORA.yml
drwxrwxrwx.   2 centos centos  172 Dec 12 18:21 Extras
drwxr-xr-x.   3 centos centos   85 Nov 11 14:10 from-DGDocker1
drwx------.   2 centos centos   28 Dec  1 14:44 .fseventsd
-rwxrwxrwx.   1 centos centos  11K Oct  4 12:28 local.env
drwxrwxrwx. 184 centos centos 4.0K Dec 13 12:51 objectStore
-rwxrwxrwx.   1 centos centos 5.1K Dec 23 11:09 README.md
drwxrwx---.   2 centos centos    6 Dec 17 14:47 site-public
drwxrwxrwx.   7 centos centos  131 Oct  9 23:17 Storage
drwxrwxrwx.   2 centos centos   53 Mar 12  2019 System Volume Information

Note that the above directory and all its contents are owned by centos:centos.

Modifying the Environment (.env)

Next, I opened a terminal on summitt-services-droplet-01, did cd ~/opt/dg-isle; git checkout staging and used sudo nano to edit files as necessary. The edits have all been saved and pushed back to the staging branch of https://GitHub/Digital-Grinnell/dg-isle.

The new .env file includes a second “staging” configuration block, the first is commented out, and second block refers to docker-compose.staging2.yml, a new file duplicated from docker-compose.staging.yml, but with modifications made explicitly for the summitt-services-droplet-01 server.

Launch the Staging.SummittServices.com Stack

Having edited all necessary files, I will launch the staging stack using:

The startup will take a couple of minutes, and it does not “signal” when it’s done, so that’s the reason for the last command above. The -f option will keep the output spooling to your terminal so that you don’t have to keep repeating the command over and over again. You will know the startup is complete when you see something like the following at the bottom of the log output:

...
Done setting proper permissions on files and directories
XDEBUG OFF
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.16.7. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.16.7. Set the 'ServerName' directive globally to suppress this message
[Tue Feb 11 18:32:35.664921 2020] [mpm_prefork:notice] [pid 12683] AH00163: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations
[Tue Feb 11 18:32:35.665016 2020] [core:notice] [pid 12683] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'

Press ctrl-c to interrupt the docker logs... command and get your terminal back.

A visit to https://staging.summittservices.com yields an SQL error so it looks like we haven’t quite created a viable staging instance of Digital.Grinnell, yet. The next logical step is to backup the database from https://isle-stage.grinnell.edu, and import it here.

Backup the Site Database

Let’s begin by visiting the original staging site’s home page (VPN connection may be required) and using the Quick Backup block at the bottom of the right-hand sidebar. Normally I would select a Backup Destination: Manual Backups Directory option to save the database backup on the server, but in this case it will be advantageous to have the backup in-hand, locally. So, I choose Backup Destination: Download instead, and the result is in my local ~/Downloads directory, specifically:

/Users/mark/Downloads/DigitalGrinnell-2020-02-11T12-46-36.mysql.gz

Upload and Import the Database

I uploaded the database to summitt-services-droplet-01 like so:

Then, in a terminal on summitt-services-droplet-01 as user centos…

Import the Database Backup Locally

Since the site is not working I cannot use drush nor Backup and Migrate to do this, so I opened a terminal in the MySQL container, namely isle-mysql-dg and did this:

White Screen of Death

Sorry, I had to jump to another project for the past couple of weeks, and now that I’m back (March 2, 2020) the situation is not good. 😦 When I visit https://staging.summittservices.com now I get a dreaded WSOD. So I peek at the Apache container logs using docker logs isle-apache-ld and find this:

[Mon Mar 02 20:35:50.641509 2020] [php7:error] [pid 12676] [client 192.168.176.4:44300] PHP Fatal error:  require_once(): Failed opening required '/var/www/html/sites/all/modules/islandora/islandora_multi_importer/vendor/autoload.php' (include_path='.:/usr/share/php') in /var/www/html/sites/all/modules/islandora/islandora_multi_importer/islandora_multi_importer.module on line 19

Bottom line, there are critical issues with IMI, the Islandora Multi-Importer, just as there were in staging. Hmmm, what to do now?

Fixing IMI

I open a terminal into the Apache container and attempt to repair/re-install IMI like so:

The commands and output from all of this are reflected in this gist.

Now, a visit to https://staging.summittservices.com yields a very welcome “Site under maintenance” page. That’s progress! Let’s take the site out of “maintenance mode” using that same Apache container terminal session, like so:

That worked, but also returned a handful of warnings summarized by these two:

Warning: file_put_contents(private:///.htaccess): failed to open stream: "DrupalPrivateStreamWrapper::stream_open" call failed in file_create_htaccess() (line 496 of /var/www/html/includes/file.inc).
User warning: The following module is missing from the file system: islandora_binary_object. For information about how to fix this, see the documentation page. in _drupal_trigger_error_with_delayed_logging() (line 1156 of /var/www/html/includes/bootstrap.inc).

Looks like we have a missing module and at least one file/directory permissions issue.

Addressing “Private” Directory Permissions

To address the permissions issue I logged in to the site as “System Admin” and visited https://staging.summittservices.com/admin/config/media/file-system where I see that our “private” file system path is: /var/www/private.

Back to my Apache container terminal to have a look at that… Aha, /var/www/private does indeed exist, but it has ownership and protections inside the container of: drwxr-xr-x. 2 root root 6 Feb 10 16:18 private/. Let’s set the owner and group here to match other directories, specifically: islandora:www-data.

Woot! The permissions warnings are gone.

Addressing the Missing islandora_binary_object Warning

So, remember back in the Installing the Missing Islandora and Custom Modules section of post 058, we commented out the installation of islandora_binary_object like so:

cd /var/www/html/sites/all/modules/islandora
git submodule add https://github.com/DigitalGrinnell/dg7.git
git submodule add https://github.com/DigitalGrinnell/idu.git
# git submodule add git://github.com/discoverygarden/islandora_binary_object.git
git submodule add https://github.com/discoverygarden/islandora_collection_search
git submodule add https://github.com/DigitalGrinnell/islandora_mods_display.git
git submodule add https://github.com/Islandora-Labs/islandora_solution_pack_oralhistories.git
# git submodule add git://github.com/nhart/islandora_pdfjs_reader.git
git submodule add https://github.com/Islandora-Labs/islandora_solr_collection_view.git
chown -R islandora:www-data *
cd /var/www/html/sites/default
drush cc all

Well, it’s evidently time to put that back! So, I’m taking a new snapshot of the server then I’ll try to solve this mystery. A little research tells me that the islandora_binary_object command above is pointing to the wrong project for Islandora v7 (it points to an Islandora v8 module), so we require some additional configuration on the host, like so:

That’s not a wrap. More about this soon…