Migrating Digital.Grinnell (DG) to ISLE 1.3.0 (ld) for Local Development
This is a follow-up to previous posts 034, Building ISLE 1.2.0 (ld) and 037, Migrating Digital.Grinnell (DG) to ISLE 1.2.0 (ld) for Local Development where I successfully completed a “local” build of ISLE v1.2.0 and subsequently started “customization” of that local instance. So, this post’s intent is to complete the goal stated in Migrating Digital.Grinnell (DG) to ISLE 1.2.0 (ld) for Local Development, but for ISLE 1.3.0, specifically to:
The goal of this project is to spin up a pristine, local Islandora stack using an updated fork of the ISLE project at https://github.com/Digital-Grinnell/dg-isle, then introduce elements like the Digital Grinnell theme and custom modules like DG7. Once these pieces are in-place and working, I’ll begin adding other critical components as well as a robust set of data gleaned from https://digital.grinnell.edu.
As before, this effort will involve an ld
, or local development
, instance of Digital.Grinnell on one of my Mac workstations. Unlike my previous work, this instance will follow the guidance of a different document, specifically install-local-migrate.md
.
Host: MA6879
For portability, and longevity, this work is being conducted on MA6879, the “student” Mac Mini that normally resides in my office on campus.
Fork Synchronization
Before beginning this process I need to get my Github environment updated by synchronizing my ISLE fork with the canonical copy. I followed this workflow to do so, like this:
Workstation Commands |
---|
cd ~/Projects git clone https://github.com/McFateM/ISLE.git cd ISLE git remote add upstream https://github.com/Islandora-Collaboration-Group/ISLE.git git fetch upstream git pull upstream master git push origin atom . |
Nice! In case you haven’t seen it before, the last command in that sequence, atom .
, simply opens my Atom
editor to the new local instance of the ISLE
project.
Installing per ISLE’s install-local-migrate.md
This is first-and-foremost a local development
copy of ISLE, but with considerable Digital.Grinnell customization, so I’m following the process outlined in the project’s ./docs/install/install-local-migrate.md
. References to Step X
that follow refer to corresponding sections of https://github.com/Islandora-Collaboration-Group/ISLE/blob/master/docs/install/install-local-migrate.md. Each section or “Step” listed below is also a link back to the corresponding section of the canonical document.
Step 0: Copy Production Data to Your Personal Computer
Drupal Site Files and Code
Per the aforementioned guidance, I did this…
Opened a zsh
shell (terminal) on Digital.Grinnell production node DGDocker1
as user islandora
using iTerm2
at ssh://islandora@dgdocker1.grinnell.edu
. Then, in that shell on the DGDocker1
host I did:
DGDocker1 Commands |
---|
cd ~ mkdir -p migration-copy/var/www/html/sites/default/files cd migration-copydocker cp isle-apache-dg:/var/www/html/sites/default/files/. var/www/html/sites/default/files/ |
Note that the -p
option on mkdir
is critical, it will create all, or part of, the entire directory tree as-specified, if it does not already exist. The final command, docker cp...
, subsequently takes advantage of this new directory tree and copies all of the Drupal .../default/files
from the production container to a similar directory on the host, DGDocker1
in this case.
For the 2nd bullet item in this step, I’ve identified that my Drupal/Islandora customizations should eventually reside in the private repo that is https://github.com/Digital-Grinnell/dg-islandora.
Drupal Site Database
For this process I already have a workflow in place, and it’s a little different than what’s documented. So here’s what I did:
- Visit my production site at https://digital.grinnell.edu.
- Login as the
System Admin
, that’sUser 1
or the super-user in Drupal terms. - The home page at https://digital.grinnell.edu provides, in the right-hand menu bar, a
Management
menu with a first option toClear cache
. Click it. - The home page also provides a
Quick Backup
block where the default options do a great job of backing up only what’s needed. Accept all defaults and click theBackup Now
button. This feature takes the site offline, makes and downloads a backup of the database (in my case it createddigital.grinnell.edu-2019-10-31T14-07-37.mysql
), and brings the site back online…“automagically”.
Fedora Hash Size (Conditional)
I found this section a little confusing, which I assume means that it does not apply to Digital.Grinnell. Moving on.
Solr Schema & Islandora Transforms
This section definitely applies to Digital.Grinnell! 😊 Since I recently completed spin-up of a “Demo” ISLE using install-local-new.md
, and since this is “not my first rodeo”, I choose to take the “advanced” path here and will “diff & merge current production customization edits into ISLE configs”. Wish me luck. 🍀
To begin this portion of the process I created a new ~/diff-and-merge-customizations
directory structure and with https://isle.localdomain running the Demo
, I copied files from it like so:
Workstation Commands |
---|
mkdir -p ~/diff-and-merge-customizations/ld mkdir -p ~/diff-and-merge-customizations/prod cd ~/diff-and-merge-customizations/ld docker cp isle-solr-ld:/usr/local/solr/collection1/conf/schema.xml schema.xml docker cp isle-fedora-ld:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt foxmlToSolr.xslt docker cp isle-fedora-ld:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms islandora_transforms |
I subsequently pulled the three diff targets from my production instance of ISLE using these commands on DGDocker1
, followed by another set on my workstation:
DGDocker1 Commands |
---|
mkdir -p ~/migration-copy/solr mkdir -p migration-copy/fedora cd migration-copy/solr docker cp isle-solr-dg:/usr/local/solr/collection1/conf/schema.xml . cd ../fedora docker cp isle-fedora-dg:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt . docker cp isle-fedora-dg:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms islandora_transforms |
Workstation Commands |
---|
cd ~/diff-and-merge-customizations/prod rsync -aruvi islandora@dgdocker1.grinnell.edu:migration-copy/solr/. . rsync -aruvi islandora@dgdocker1.grinnell.edu:migration-copy/fedora/. . |
This left me with a local ~/diff-and-merge-customizations
directory with sub-dirs ld
and prod
. The diff-and-merge operation subsequently compared the contents of these two sub-dirs, merging the differences from prod
into the corresponding files found in ld
. I did both the diff
and merge operations using Atom
.
The command and output from the first diff
of the islandora_transforms
was:
╭─digital@MA6879 ~
╰─$ cd ~/diff-and-merge-customizations/ld
╭─digital@MA6879 ~/diff-and-merge-customizations/ld
╰─$ diff -r islandora_transforms ../prod/islandora_transforms 1 ↵
Only in islandora_transforms: .git
diff -r islandora_transforms/WORKFLOW_to_solr.xslt ../prod/islandora_transforms/WORKFLOW_to_solr.xslt
95c95
</xsl:stylesheet>
\ No newline at end of file
---
</xsl:stylesheet>
So, no significant differences here. Yay!
Next, I compared the two files, one-at-a-time, using Atom
and its Split Diff
package. There were very few differences in foxmToSolr.xslt
so it was easy to merge. I took the significant differences in the prod
copy and merged them into the ld
copy. I did the same with schema.xml
, but it was an entirely different beast. There were 34 sections of differences, and in many cases it was not clear if our existing customizations should be merged. As a result, the changes I made in schema.xml
, and some I did NOT make, will need to be carefully re-evaluated as this local migrate “test” proceeds.
At this point I’ve saved my “merged” customizations in ~/diff-and-merge-customizations/ld
until Step 2a
, below.
Phew, that was a lot of Step 0!
Step 1: Edit /etc/hosts
File
Easy peasy, relatively speaking. 😄
Step 2: Setup Git for the ISLE Project
OK, I already have two private Github repositories…
- https://gitubh.com/Digital-Grinnell/dg-isle for my customized copy of the ISLE project, and
- https://github.com/Digital-Grinnell/dg-islandora for my customized copy of the Drupal and Islandora code.
However, in my case neither of these repositories is “complete” because they are products of the install-local-new.md
process, so my focus in this step will be to get these two repos in line with the Git workflow that’s being established.
Regarding the dg-isle
repo…
- I cloned it to my local workstation and set things up as directed with:
Workstation Commands |
---|
cd ~/Projects git clone https://github.com/Digital-Grinnell/dg-isle.git cd dg-isle git remote add icg-upstream https://github.com/Islandora-Collaboration-Group/ISLE.git git fetch icg-upstream git pull icg-upstream master git push -u origin master |
Step 2a: Add Customizations from Step 0
Before executing the documented commands I elected to create a new branch in my local dg-isle
repo, so my workflow was this:
Workstation Commands |
---|
cd ~/Projects/dg-isle git checkout -b solr-and-gsearch-customizations mkdir -p ./config/solr mkdir -p ./config/fedora/gsearch cp -f ~/diff-and-merge-customizations/ld/schema.xml ./config/solr/ cp -f ~/diff-and-merge-customizations/ld/foxmlToSolr.xslt ./config/fedora/gsearch/ cp -fr ~/diff-and-merge-customizations/ld/islandora_transforms ./config/fedora/gsearch/ |
Next, I edited docker-compose.local.yml
as prescribed, and then saved it all like so:
Workstation Commands |
---|
cd ~/Projects/dg-isle git checkout solr-and-gsearch-customizations git add -A git commit -m “Customizations from Step 2a” git push –set-upstream origin solr-and-gsearch-customizations |
Step 3: Git Clone the Production Islandora Drupal Site Code
OK, this section will deal with my Islandora/Drupal code repository with all my customization in it, in my case that’s https://github.com/Digital-Grinnell/dg-islandora, but my copy isn’t complete yet, so…
Post 038 - Building My dg-islandora
Code Repository
Earlier I created blog post 038-Building-My-dg-islandora
-Code-Repository. It combined elements of post 021 with post 034 to create a “customized” local ISLE v1.2.0 instance with features of Digital.Grinnell.
The product of all that work is the Digital-Grinnell/dg-islandora private GitHub repo.
I followed the procedure outline in Step 3: Git Clone the Production Islandora Drupal Site Code with my Digital-Grinnell/dg-islandora private repo like so:
Workstation Commands |
---|
cd ~/Projects git clone https://github.com/Digital-Grinnell/dg-islandora.git cd dg-islandora atom . |
Again, the final atom .
command opened my newly cloned project in Atom to simplify editing. I followed the guidance of Step 3 to update my docker-compose.local.yml
as directed. This concludes Step 3; moving on.
Step 4: Edit the “.env” File to Change to the Local Environment
I simply confirmed the suggested edits to my .env
file using the Atom editing session opened above.
Step 5: Create New Users and Passwords by Editing “local.env” File
Since this is “local” I’ll keep no secrets here. I set all the variable passwords, and hashes to “password”, and “thisisalengthyhashstring”, respectively, as permitted. All other variables, like usernames and more, I set to “local”. Note that I left the original “# Replace…” strings intact, but made sure to push each one to a new line.
There’s a copy of my complete customized local.env
file in this gist.
Step 6: Create New Self-Signed Certs for Your Project
Simply followed the instructions to-the-letter.
Step 7: Download the ISLE Images
Simply followed the instructions to-the-letter.
Step 8: Launch Process
Simply followed the instructions to-the-letter.
Step 9: Import the Production MySQL Drupal Database
In this step, employing Method B: Use the Command Line, I returned to the DGDocker1
production database, digital.grinnell.edu-2019-10-31T14-07-37.mysql.gz
, saved in the Drupal Site Database section of this document.
First, I unzipped the aforementioned database archive using my MacBook’s Archive Utility
to produce digital.grinnell.edu-2019-10-31T14-07-37.mysql
on my workstation desktop. The full path to this file was /Users/digital/Desktop/digital.grinnell.edu-2019-10-31T14-07-37.mysql
. Then, on my workstation and in the MySQL container:
Workstation Commands |
---|
docker cp /Users/digital/Desktop/digital.grinnell.edu-2019-10-31T14-07-37.mysql isle-mysql-ld:/database.mysql docker exec -it isle-mysql-ld bash |
MySQL Container Commands |
---|
mysql -u local -p local < database.mysql exit |
Step 10: Run Islandora Drupal Site Scripts
Following the provided guidance yielded a set of workstation commands like this:
Workstation Commands |
---|
docker cp scripts/apache/migration_site_vsets.sh isle-apache-ld:/var/www/html/migration_site_vsets.sh docker exec -it isle-apache-ld bash -c “chmod +x /var/www/html/migration_site_vsets.sh” docker exec -it isle-apache-ld bash -c “cd /var/www/html && ./migration_site_vsets.sh” docker cp scripts/apache/install_solution_packs.sh isle-apache-ld:/var/www/html/install_solution_packs.sh docker exec -it isle-apache-ld bash -c “chmod +x /var/www/html/install_solution_packs.sh” docker exec -it isle-apache-ld bash -c “cd /var/www/html && ./install_solution_packs.sh” |
Output from the migration_site_vsets.sh
command:
The following module is missing from the file system: <em class="placeholder">masquerade</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">announcements</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">backup_migrate</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following theme is missing from the file system: <em class="placeholder">bootstrap</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">dg7</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following theme is missing from the file system: <em class="placeholder">digital_grinnell_bootstrap</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">email</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">git_deploy</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">idu</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">ihc</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_binary_object</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_collection_search</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_google_scholar</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_mods_display</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_multi_importer</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_oralhistories</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_pdfjs_reader</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">islandora_solr_collection_view</em>. For information about how to fix this, see [warning]
<a href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">ldap_servers</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">maillog</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">phpexcel</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">r4032login</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">smtp</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">transcripts_ui</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">views_bootstrap</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
The following module is missing from the file system: <em class="placeholder">field_group</em>. For information about how to fix this, see <a [warning]
href="https://www.drupal.org/node/2487215">the documentation page</a>. bootstrap.inc:1156
WD php: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'local.users' doesn't exist: SELECT base.uid AS uid, base.name AS name, [error]
base.pass AS pass, base.mail AS mail, base.theme AS theme, base.signature AS signature, base.signature_format AS signature_format, base.created AS created,
base.access AS access, base.login AS login, base.status AS status, base.timezone AS timezone, base.language AS language, base.picture AS picture, base.init AS
init, base.data AS data
FROM
{users} base
WHERE (base.uid IN (:db_condition_placeholder_0)) ; Array
(
[:db_condition_placeholder_0] => 1
)
in DrupalDefaultEntityController->load() (line 198 of /var/www/html/includes/entity.inc).