Migrating Digital.Grinnell (DG) to ISLE 1.2.0 (ld) for Local Development
This post is a follow-up to previous post 034 where I successfully completed a “local” build of ISLE v1.2.0, but did no “customization” of that local instance. So, this post’s intent is to complete the goal stated in post 034, specifically to:
The goal of this project is to spin up a pristine, local Islandora stack using an updated fork of the ISLE project at https://github.com/DigitalGrinnell/dg-isle, then introduce elements like the Digital Grinnell theme and custom modules like DG7. Once these pieces are in-place and working, I’ll begin adding other critical components as well as a robust set of data gleaned from https://digital.grinnell.edu.
As before, this effort will involve an ld
, or local development
, instance of Digital.Grinnell on one of my Mac workstations. Unlike my previous work, this instance will follow the guidance of a different document, specifically install-local-migrate.md
.
Fork Synchronization
Before beginning this process I need to get my Github environment updated by synchronizing my ISLE fork with the canonical copy. I followed this workflow to do so, like this:
Workstation Commands |
---|
cd ~/Projects git clone https://github.com/McFateM/ISLE.git cd ISLE git remote add upstream https://github.com/Islandora-Collaboration-Group/ISLE.git git fetch upstream git pull upstream master git push atom . |
Nice! In case you haven’t seen it before, the last command in that sequence, atom .
, simply opens my Atom
editor to the new local instance of the ISLE
project.
Installing per ISLE’s install-local-migrate.md
This is first-and-foremost a local development
copy of ISLE, but with considerable Digital Grinnell customization, so I’m following the process outlined in the project’s ./docs/install/install-local-migrate.md
. References to Step X
that follow refer to corresponding sections of https://github.com/Islandora-Collaboration-Group/ISLE/blob/master/docs/install/install-local-migrate.md. Each section or “Step” listed below is also a link back to the corresponding section of the canonical document.
Step 0: Copy Production Data to Your Local
Drupal Site Files & Code
Per the aforementioned guidance, I did this…
Opened a zsh
shell (terminal) on Digital.Grinnell production node DGDocker1
as user islandora
using iTerm2
at ssh://islandora@dgdocker1.grinnell.edu
. Then, in that shell on the DGDocker1
host I did:
DGDocker1 Commands |
---|
cd ~ mkdir -p migration-copy/var/www/html/sites/default/files cd migration-copydocker cp isle-apache-dg:/var/www/html/sites/default/files/. var/www/html/sites/default/files/ |
Note that the -p
option on mkdir
is critical, it will create all, or port of, the entire directory tree as-specified, if it does not already exist. The final command, docker cp...
, subsequently takes advantage of this new directory tree and copies all of the Drupal .../default/files
from the production container to a similar directory on the host, DGDocker1
in this case.
For the 2nd bullet item in this step, I’ve identified that my Drupal/Islandora customizations should eventually reside in the private repo that is https://github.com/McFateM/dg-islandora.
Drupal Site Database
For this process I already have a workflow in place, and it’s a little different than what’s documented. So here’s what I did:
- Visit my production site at https://digital.grinnell.edu.
- Login as the
System Admin
, that’sUser 1
or the super-user in Drupal terms. - The home page at https://digital.grinnell.edu provides, in the right-hand menu bar, a
Management
menu with a first option toClear cache
. Click it. - The home page also provides a
Quick Backup
block where the default options do a great job of backing up only what’s needed. Accept all defaults and click theBackup Now
button. This feature takes the site offline, makes and downloads a backup of the database (in my case it createddigital.grinnell.edu-2019-08-13T15-51-20.mysql
), and brings the site back online…“automagically”.
Fedora Hash Size (Conditional)
I found this section a little confusing, which I assume means that it does not apply to Digital.Grinnell. Moving on.
Solr Schema & Islandora Transforms
This section definitely applies to Digital.Grinnell! 😊 Since I recently completed spin-up of a “Demo” ISLE using install-local-new.md
, and since this is “not my first rodeo”, I choose to take the “advanced” path here and will “diff & merge current production customization edits into ISLE configs”. Wish me luck. 🍀
To begin this portion of the process I created a new ~/diff-and-merge-customizations
directory structure and with https://isle.localdomain running the Demo
, I copied files from it like so:
Workstation Commands |
---|
mkdir -p ~/diff-and-merge-customizations/ld mkdir -p ~/diff-and-merge-customizations/prod cd ~/diff-and-merge-customizations/ld docker cp isle-solr-ld:/usr/local/solr/collection1/conf/schema.xml schema.xml docker cp isle-fedora-ld:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt foxmlToSolr.xslt docker cp isle-fedora-ld:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms islandora_transforms |
I subsequently pulled the three diff targets from my production instance of ISLE using these commands on DGDocker1
, followed by another set on my workstation:
DGDocker1 Commands |
---|
mkdir -p ~/migration-copy/solr mkdir -p migration-copy/fedora cd migration-copy/solr docker cp isle-solr-dg:/usr/local/solr/collection1/conf/schema.xml . cd ../fedora docker cp isle-fedora-dg:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt . docker cp isle-fedora-dg:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms islandora_transforms |
Workstation Commands |
---|
cd ~/diff-and-merge-customizations/prod rsync -aruvi islandora@dgdocker1.grinnell.edu:migration-copy/solr/. . rsync -aruvi islandora@dgdocker1.grinnell.edu:migration-copy/fedora/. . |
This left me with a local ~/diff-and-merge-customizations
directory with sub-dirs ld
and prod
. The diff-and-merge operation subsequently compared the contents of these two sub-dirs, merging the differences from prod
into the corresponding files found in ld
. I did both the diff
and merge operations using Atom
.
The command and output from the first diff
of the islandora_transforms
was:
╭─markmcfate@ma8660 ~
╰─$ cd ~/diff-and-merge-customization/ld
╭─markmcfate@ma8660 ~/diff-and-merge-customization/ld
╰─$ diff -r islandora_transforms ../prod/islandora_transforms 1 ↵
Only in islandora_transforms: .git
diff -r islandora_transforms/WORKFLOW_to_solr.xslt ../prod/islandora_transforms/WORKFLOW_to_solr.xslt
95c95
</xsl:stylesheet>
\ No newline at end of file
---
</xsl:stylesheet>
So, no significant differences here. Yay!
Next, I compared the two files, one-at-a-time, using Atom
and its Split Diff
package. There were very few differences in foxmToSolr.xslt
so it was easy to merge. I took the significant differences in the prod
copy and merged them into the ld
copy. I did the same with schema.xml
, but it was an entirely different beast. There were 34 sections of differences, and in many cases it was not clear if our existing customizations should be merged. As a result, the changes I made in schema.xml
, and some I did NOT make, will need to be carefully re-evaluated as this local migrate “test” proceeds.
At this point I’ve saved my “merged” customizations in ~/diff-and-merge-customizations/ld
until Step 2a
, below.
Phew, that was a lot of Step 0!
Step 1: Edit /etc/hosts
File
Easy peasy, relatively speaking. 😄
Step 2: Setup Git for the ISLE Project
OK, I already have two private Github repositories…
- https://gitubh.com/McFateM/dg-isle for my customized copy of the ISLE project, and
- https://github.com/McFateM/dg-islandora for my customized copy of the Drupal and Islandora code.
However, in my case neither of these repositories is “complete” because they are products of the install-local-new.md
process, so my focus in this step will be to get these two repos in line with the Git workflow that’s being established.
Regarding the dg-isle
repo…
- I cloned it to my local workstation and set things up as directed with:
Workstation Commands |
---|
cd ~/Projects git clone https://github.com/McFateM/dg-isle.git cd dg-isle git remote add icg-upstream https://github.com/Islandora-Collaboration-Group/ISLE.git git fetch icg-upstream git pull icg-upstream master git push -u origin master |
Step 2a: Add Customizations from Step 0
Before executing the documented commands I elected to create a new branch in my local dg-isle
repo, so my workflow was this:
Workstation Commands |
---|
cd ~/Projects/dg-isle git checkoput -b solr-and-gsearch-customizations mkdir -p ./config/solr mkdir -p ./config/fedora/gsearch cp -f ~/diff-and-merge-customizations/ld/schema.xml ./config/solr/ cp -f ~/diff-and-merge-customizations/ld/foxmlToSolr.xslt ./config/fedora/gsearch/ cp -fr ~/diff-and-merge-customizations/ld/islandora_transforms ./config/fedora/gsearch/ |
Next, I edited docker-compose.local.yml
as prescribed, and then saved it all like so:
Workstation Commands |
---|
cd ~/Projects/dg-isle git checkout solr-and-gsearch-customizations git add -A git commit -m “Customizations from Step 2a” git push –set-upstream origin solr-and-gsearch-customizations |
Step 3. git clone
the Production Drupal Site Code
OK, this section will deal with my Islandora/Drupal code repository with all my customization in it, in my case that’s https://github.com/McFateM/dg-islandora, but my copy isn’t complete yet, so…
Post 038 - Building My dg-islandora
Code Repository
I’m creating a new blog post that will become 038-Building-My-dg-islandora
-Code-Repository. It will combine elements of post 021 with post 034 to create a “customized” local ISLE v1.2.0 instance with features of Digital.Grinnell.
Once that process is complete, I’ll return here to continue Step 3
. Until then…