Symbioquine's 2.x Migration Testing Log

[Symbioquine]

Today I started playing around with the 2.x migration and I wanted to share my process and any issues/learnings I encounter along the way.

Note: This is not intended for direct use as a tutorial so don’t complain if these steps don’t work as-is for you - though I’m certainly happy to answer questions and take feedback that will be helpful to others.

The complexity of my farmOS installation configuration has certainly grown in the last year so I’ll try to keep some of the irrelevant details out of the way for clarity’s sake.

In short I’m running the standard farmOS 7.x-1.7 image behind a reverse proxy similar to the documentation here.

The relevant part of my production docker-compose.yml file is this;

version: '3.7'
services:
  db:
    image: mariadb:latest
    volumes:
      - './db:/var/lib/mysql'
      - './db-init:/docker-entrypoint-initdb.d'
    expose:
      - '3306'
    environment:
      MYSQL_ROOT_PASSWORD: farm
      MYSQL_DATABASE: farm
      MYSQL_USER: farm
      MYSQL_PASSWORD: farm

  www:
    depends_on:
      - db
    image: farmos/farmos:7.x-1.7
    volumes:
      - './sites:/var/www/html/sites'
      - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
    expose:
      - '80'

---

I also have a number of modifications and custom modules which I hope build migration logic for incrementally. My first milestone will be getting the vanilla asset/log types migrated using the provided migration implementation.

Since 2.x is a work-in-progress and I expect to need to repeat the migration process many times as both 2.x and my customizations evolve, I will be automating everything…

[Symbioquine]

My current production environment consists of a single host with the DB and site run with the docker-compose file I described above. Under other circumstances I might use a managed DB service (RDS), a cloud storage service (S3/GCS) bucket, and a number of farmOS instances behind a load balancer, but with my current traffic/user volume the risk of data loss is pretty low and I don’t need high availability/redudancy. I use the Backup and Migrate to make snapshots of the farmOS db/files to S3 on a regular basis.

This provides an easy route for development testing since I can take one of those snapshots and restore it locally;

# Stop and remove my old containers and their data (Note: this is destructive and I'm only doing it in my local development environment)
docker-compose down
sudo rm -rf db sites

# Extract the database dump and site directory
sudo tar --extract --file=$(ls *.sitearchive.tar.gz | tail -n 1) --strip-components=1 --directory=./db-init --wildcards '*/database.sql'
tar --extract --file=$(ls *.sitearchive.tar.gz | tail -n 1) --strip-components=2 --wildcards '*/docroot/sites'

# Scary one-liners to put some basic settings in place
sudo cp sites/default/default.settings.php sites/default/settings.php
sudo sh -c "printf \"\n\n\\\$conf['reverse_proxy'] = TRUE;\n\\\$conf['reverse_proxy_addresses'] = [@\\\$_SERVER['REMOTE_ADDR']];\n\\\$base_url = \\\$_SERVER['HTTP_X_FORWARDED_PROTO'] . '://' . \\\$_SERVER['SERVER_NAME'];\n\" >> sites/default/settings.php"
sudo sed -i "s/$databases = array();/$databases = array (\n  'default' =>\n  array (\n    'default' =>\n    array (\n      'database' => 'farm',\n      'username' => 'farm',\n      'password' => 'farm',\n      'host' => 'db',\n      'port' => '3306',\n      'driver' => 'mysql',\n      'prefix' => '',\n    ),\n  ),\n);/g" sites/default/settings.php

# Fix the permissions
sudo chown -R www-data:www-data sites

# Start the DB and wait for it to be ready (including loading the dump from ./db-init/database.sql)
docker-compose up -d db
docker-compose exec db bash -c 'while { ! exec 3<>/dev/tcp/db/3306; } > /dev/null 2>&1; do sleep 0.1; done'

# Disable backups/statsd for development (Note: I could do this later with Drush, but prefer to do it before starting the farmOS container to ensure they never run in development)
docker-compose exec db /bin/sh -c "echo \"UPDATE system SET status='0' WHERE name IN ('backup_migrate', 'statsd'); DELETE FROM cache_bootstrap WHERE cid='system_list';\" | mysql -u farm -pfarm -D farm"

# Start the farmOS container
docker-compose up -d www

# Install Drush (Yes, I know this isn't permanent and I could build my own image or use the development one, but I'm opting for a setup that's closer to my production one than either of those might produce)
docker-compose exec www bash -c "curl -fsSL -o /usr/local/bin/drush.phar 'https://github.com/drush-ops/drush/releases/download/8.3.5/drush.phar' && chmod +x /usr/local/bin/drush.phar && ln -s /usr/local/bin/drush.phar /usr/local/bin/drush && drush core-status"

# Set the 'root' user's password to 'test' for development (Yes, I know doing "apt-get install" at runtime in a docker container is pretty unconventional/discouraged)
docker-compose exec www bash -c "apt-get install -y mariadb-client && drush upwd root --password='test'"

That’s pretty much all the background. In my next post I’ll start experimenting with the actual migration of the resulting development testing site.

[Symbioquine]

Now for the fun part. Based on https://2x.farmos.org/hosting/migration/ (version permalink) I’ve updated my docker-compose.yml file as follows;

version: '3.7'
services:
  db:
    image: mariadb:latest
    volumes:
      - './db:/var/lib/mysql'
      - './db.cnf:/etc/mysql/conf.d/my.cnf'
      - './dblogs:/var/log/mysql'
      - './db-init:/docker-entrypoint-initdb.d'
    ports:
      - '3306:3306'
    environment:
      MYSQL_ROOT_PASSWORD: farm
      MYSQL_DATABASE: farm
      MYSQL_USER: farm
      MYSQL_PASSWORD: farm

  www:
    depends_on:
      - db
    image: farmos/farmos:7.x-1.7
    volumes:
      - './sites:/var/www/html/sites'
      - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
    expose:
      - '80'
+
+  db2:
+    image: postgres:12
+    volumes:
+      - './db2:/var/lib/postgresql/data'
+    expose:
+      - '5432'
+    environment:
+      POSTGRES_USER: farm
+      POSTGRES_PASSWORD: farm
+      POSTGRES_DB: farm
+
+  www2:
+    depends_on:
+      - db2
+    image: farmos/farmos:2.x-dev
+    entrypoint: /bin/bash
+    command:
+      - -c
+      - |
+        set -ex
+
+        wait_db_ready() {
+            while { ! exec 3<>/dev/tcp/db2/5432; } > /dev/null 2>&1; do sleep 0.1; done
+        }
+
+        if [ -d /opt/drupal ] && ! [ "$$(ls -A /opt/drupal/composer.json)" ]; then
+          echo "farmOS codebase not detected. Copying from pre-built files in the Docker image."
+          cp -rp /var/farmOS/. /opt/drupal
+          wait_db_ready
+          su www-data -s /bin/bash -c 'drush site-install farm --locale=en --db-url=pgsql://farm:farm@db2/farm --site-name=Test0 --account-name=root --account-pass=test'
+
+          echo "
+        \$$databases['migrate']['default'] = [
+          'database' => 'farm',
+          'username' => 'farm',
+          'password' => 'farm',
+          'prefix' => '',
+          'host' => 'db',
+          'port' => '3306',
+          'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
+          'driver' => 'mysql',
+        ];" >> /opt/drupal/web/sites/default/settings.php
+
+          cp -r /opt/farmOS_1x_files /opt/drupal/web/sites/default/files/migrate
+          chown -R www-data:www-data /opt/drupal/web/sites/default/files/migrate
+
+        fi
+
+        wait_db_ready
+
+        su www-data -s /bin/bash <<'EOF'
+          set -ex
+
+          drush --root=/opt/drupal pm-enable --yes farm_migrate
+          drush --root=/opt/drupal migrate:import --group=farm_migrate
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_area
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_log
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_reference
+
+        EOF
+
+        exec docker-entrypoint.sh apache2-foreground
+    volumes:
+      - './www2:/opt/drupal'
+      - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
+      - './sites/default/files:/opt/farmOS_1x_files:ro'
+    expose:
+      - '80'
+    environment:
+      XDEBUG_MODE: debug

After running docker-compose up -d we can watch the www2 container attempting the migration;

$ docker-compose logs --follow www2 
Attaching to devel_www2_1
www2_1           | + '[' -d /opt/drupal ']'
www2_1           | ++ ls -A /opt/drupal/composer.json
www2_1           | ls: cannot access '/opt/drupal/composer.json': No such file or directory
www2_1           | + '[' '' ']'
www2_1           | + echo 'farmOS codebase not detected. Copying from pre-built files in the Docker image.'
www2_1           | + cp -rp /var/farmOS/. /opt/drupal
www2_1           | farmOS codebase not detected. Copying from pre-built files in the Docker image.
www2_1           | + wait_db_ready
www2_1           | + su www-data -s /bin/bash -c 'drush site-install farm --locale=en --db-url=pgsql://farm:farm@db2/farm --site-name=Test0 --account-name=root --account-pass=test'
www2_1           | 
www2_1           |  You are about to:
www2_1           |  * Create a sites/default/settings.php file
www2_1           |  * DROP all tables in your 'farm' database.
www2_1           | 
www2_1           |  Do you want to continue? (yes/no) [yes]:
www2_1           |  [notice] Starting Drupal installation. This takes a while.
...
www2_1           |  1/2 [==============>-------------]  50%
www2_1           |  2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_structure'
www2_1           |  [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_water'
www2_1           | + drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
www2_1           | 
www2_1           |    1/559 [>---------------------------]   0%
www2_1           |   56/559 [==>-------------------------]  10%
www2_1           |  102/559 [=====>----------------------]  18%
www2_1           |  112/559 [=====>----------------------]  20%
www2_1           |  168/559 [========>-------------------]  30%
www2_1           |  211/559 [==========>-----------------]  37%
www2_1           |  224/559 [===========>----------------]  40%
www2_1           |  280/559 [==============>-------------]  50%
www2_1           |  336/559 [================>-----------]  60%
www2_1           |  386/559 [===================>--------]  69%
www2_1           |  392/559 [===================>--------]  70% [notice] Processed 427 items (427 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_quantity'
www2_1           | + drush --root=/opt/drupal migrate:import --group=farm_migrate_log
www2_1           |  [error]  Migration farm_migrate_log_activity did not meet the requirements. Missing migrations farm_migrate_quantity. requirements: farm_migrate_quantity. 
www2_1           | 
www2_1           | In MigrateToolsCommands.php line 866:
www2_1           |                                                
www2_1           |   farm_migrate_log_activity migration failed.  
www2_1           |                                                
www2_1           | 
www2_1           | migrate:import [--all] [--group GROUP] [--tag TAG] [--limit LIMIT] [--feedback FEEDBACK] [--idlist IDLIST] [--idlist-delimiter [IDLIST-DELIMITER]] [--update] [--force] [--continue-on-failure] [--execute-dependencies] [--skip-progress-bar] [--sync] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--druplicon] [--xh-link XH-LINK] [--notify [NOTIFY]] [--] <command> [<migration_names>]
www2_1           | 
devel_www2_1 exited with code 1

The full log is available here.

Next I’ll be trying to figure out why I’m getting that somewhat inscrutable error

“Migration farm_migrate_log_activity did not meet the requirements. Missing migrations farm_migrate_quantity. requirements: farm_migrate_quantity.”

which is occurring immediately after the successful drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity command…

[Symbioquine]

First let’s check that I’m using the latest version of farmos/farmos:2.x-dev;

$ docker pull farmos/farmos:2.x-dev
2.x-dev: Pulling from farmos/farmos
Digest: sha256:8ec061a681a6a9d25936069070e03a5e316e5f654d36a96862d8f56915de756e
Status: Image is up to date for farmos/farmos:2.x-dev
docker.io/farmos/farmos:2.x-dev

Now we’ll blow away the attempted migration and try again;

docker-compose stop db2 www2 && docker-compose rm -f db2 www2 && sudo rm -rf db2 www2
docker-compose up -d

And same error;

[error] Migration farm_migrate_log_activity did not meet the requirements. Missing migrations farm_migrate_quantity. requirements: farm_migrate_quantity.

Awesome, it’s reproducible…

[Symbioquine]

Next I decided to try the status command the documentation refers to so I’ve updated my docker-compose.yml as follows;

           drush --root=/opt/drupal pm-enable --yes farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
           drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
           drush --root=/opt/drupal migrate:import --group=farm_migrate_area
           drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
+          drush --root=/opt/drupal migrate:status --tag="farmOS 1.x"
           drush --root=/opt/drupal migrate:import --group=farm_migrate_log
           drush --root=/opt/drupal migrate:import --group=farm_migrate_reference

Which yields the following;

www2_1           |  392/559 [===================>--------]  70% [notice] Processed 427 items (427 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_quantity'
www2_1           | + drush --root=/opt/drupal migrate:status '--tag=farmOS 1.x'
www2_1           |  ---------------- -------------- -------- ------- ---------- ------------- --------------- 
www2_1           |   Group            Migration ID   Status   Total   Imported   Unprocessed   Last Imported  
www2_1           |  ---------------- -------------- -------- ------- ---------- ------------- --------------- 
www2_1           |   farmOS 1.x       farm_migr      Idle     352     0          352                          
www2_1           |   References       ate_area_                                                               
www2_1           |   Migration        field_par                                                               
www2_1           |   (farm_migrate_   ent                                                                     
www2_1           |   reference)                                                                               
www2_1           |   farmOS 1.x       farm_migr      Idle     2945    0          2945                         
www2_1           |   References       ate_asset                                                               
www2_1           |   Migration        _field_pa                                                               
www2_1           |   (farm_migrate_   rent                                                                    
www2_1           |   reference)                                                                               
www2_1           |                                                                                            
www2_1           |   farmOS 1.x       farm_migr      Idle     1       1          0             2021-          
www2_1           |   Migration        ate_syste                                                02-14          
www2_1           |   (farm_migrate)   m_date                                                   02:38          
www2_1           |                                                                             :34            
www2_1           |   farmOS 1.x       farm_migr      Idle     7       7          0             2021-          
www2_1           |   Migration        ate_user                                                 02-14          
www2_1           |   (farm_migrate)                                                            02:38          
www2_1           |                                                                             :34            
www2_1           |   farmOS 1.x       farm_migr      Idle     728     728        0             2021-          
www2_1           |   Migration        ate_file                                                 02-14          
www2_1           |   (farm_migrate)                                                            02:38          
www2_1           |                                                                             :44            
www2_1           |   farmOS 1.x       farm_migr      Idle     0       0          0             2021-          
www2_1           |   Migration        ate_file_                                                02-14          
www2_1           |   (farm_migrate)   private                                                  02:38          
www2_1           |                                                                             :44            
www2_1           |                                                                                            
www2_1           |   farmOS 1.x       farm_migr      Idle     179     179        0             2021-          
www2_1           |   Area Migration   ate_area_                                                02-14          
www2_1           |   (farm_migrate_   land                                                     02:39          
www2_1           |   area)                                                                     :12            
www2_1           |   farmOS 1.x       farm_migr      Idle     171     171        0             2021-          
www2_1           |   Area Migration   ate_area_                                                02-14          
www2_1           |   (farm_migrate_   none                                                     02:39          
www2_1           |   area)                                                                     :16            
www2_1           |   farmOS 1.x       farm_migr      Idle     2       2          0             2021-          
www2_1           |   Area Migration   ate_area_                                                02-14          
www2_1           |   (farm_migrate_   structure                                                02:39          
www2_1           |   area)                                                                     :16            
www2_1           |   farmOS 1.x       farm_migr      Idle     0       0          0             2021-          
www2_1           |   Area Migration   ate_area_                                                02-14          
www2_1           |   (farm_migrate_   water                                                    02:39          
www2_1           |   area)                                                                     :16            
www2_1           |                                                                                            
www2_1           |   farmOS 1.x       farm_migr      Idle     8       8          0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_anima                                                02:38          
www2_1           |   (farm_migrate_   l_type                                                   :45            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     0       0          0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_crop_                                                02:38          
www2_1           |   (farm_migrate_   family                                                   :45            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     372     372        0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_plant                                                02:38          
www2_1           |   (farm_migrate_   _type                                                    :53            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     57      57         0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_seaso                                                02:38          
www2_1           |   (farm_migrate_   n                                                        :54            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     8       8          0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_log_c                                                02:38          
www2_1           |   (farm_migrate_   ategory                                                  :54            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     5       5          0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_unit                                                 02:38          
www2_1           |   (farm_migrate_                                                            :54            
www2_1           |   taxonomy)                                                                                
www2_1           |   farmOS 1.x       farm_migr      Idle     1       1          0             2021-          
www2_1           |   Taxonomy         ate_taxon                                                02-14          
www2_1           |   Migration        omy_mater                                                02:38          
www2_1           |   (farm_migrate_   ial                                                      :54            
www2_1           |   taxonomy)                                                                                
www2_1           |                                                                                            
www2_1           |   farmOS 1.x       farm_migr      Idle     67      67         0             2021-          
www2_1           |   Asset            ate_asset                                                02-14          
www2_1           |   Migration        _animal                                                  02:38          
www2_1           |   (farm_migrate_                                                            :57            
www2_1           |   asset)                                                                                   
www2_1           |   farmOS 1.x       farm_migr      Idle     2       2          0             2021-          
www2_1           |   Asset            ate_asset                                                02-14          
www2_1           |   Migration        _equipmen                                                02:38          
www2_1           |   (farm_migrate_   t                                                        :57            
www2_1           |   asset)                                                                                   
www2_1           |   farmOS 1.x       farm_migr      Idle     372     372        0             2021-          
www2_1           |   Asset            ate_asset                                                02-14          
www2_1           |   Migration        _plant                                                   02:39          
www2_1           |   (farm_migrate_                                                            :08            
www2_1           |   asset)                                                                                   
www2_1           |                                                                                            
www2_1           |   farmOS 1.x       farm_migr      Idle     559     427        132           2021-          
www2_1           |   Quantity         ate_quant                                                02-14          
www2_1           |   Migration        ity                                                      02:39          
www2_1           |   (farm_migrate_                                                            :25            
www2_1           |   quantity)                                                                                
www2_1           |                                                                                            
www2_1           |   farmOS 1.x Log   farm_migr      Idle     414     0          414                          
www2_1           |   Migration        ate_log_a                                                               
www2_1           |   (farm_migrate_   ctivity                                                                 
www2_1           |   log)                                                                                     
www2_1           |   farmOS 1.x Log   farm_migr      Idle     72      0          72                           
www2_1           |   Migration        ate_log_h                                                               
www2_1           |   (farm_migrate_   arvest                                                                  
www2_1           |   log)                                                                                     
www2_1           |   farmOS 1.x Log   farm_migr      Idle     2       0          2                            
www2_1           |   Migration        ate_log_i                                                               
www2_1           |   (farm_migrate_   nput                                                                    
www2_1           |   log)                                                                                     
www2_1           |   farmOS 1.x Log   farm_migr      Idle     0       0          0                            
www2_1           |   Migration        ate_log_m                                                               
www2_1           |   (farm_migrate_   aintenanc                                                               
www2_1           |   log)             e                                                                       
www2_1           |   farmOS 1.x Log   farm_migr      Idle     7015    0          7015                         
www2_1           |   Migration        ate_log_o                                                               
www2_1           |   (farm_migrate_   bservatio                                                               
www2_1           |   log)             n                                                                       
www2_1           |   farmOS 1.x Log   farm_migr      Idle     349     0          349                          
www2_1           |   Migration        ate_log_s                                                               
www2_1           |   (farm_migrate_   eeding                                                                  
www2_1           |   log)                                                                                     
www2_1           |  ---------------- -------------- -------- ------- ---------- ------------- --------------- 
www2_1           | + drush --root=/opt/drupal migrate:import --group=farm_migrate_log
www2_1           |  [error]  Migration farm_migrate_log_activity did not meet the requirements. Missing migrations farm_migrate_quantity. requirements: farm_migrate_quantity.

I’m now suspecting it may have something to do with the 132 “Unprocessed” items from the farm_migrate_quantity migration;

+ drush --root=/opt/drupal migrate:status '--tag=farmOS 1.x'
Group                                                    Migration ID               Status   Total   Imported   Unprocessed   Last Imported  
farmOS 1.x Quantity Migration (farm_migrate_quantity)    farm_migrate_quantity      Idle     559     427        132           2021-02-14 02:39:25

It’s interesting that the farm_migrate_quantity migration completed successfully with the following output;

[notice] Processed 427 items (427 created, 0 updated, 0 failed, 0 ignored) - done with ‘farm_migrate_quantity’

[Symbioquine]

Logs from that console session are here.

So now I guess I’m a little stuck. Short of hooking up my debugger and tracing through the migration code or writing a script to figure out which 132 items can’t be processed and whether there’s a pattern there, I don’t see an obvious next step.

It’s surprising that the quantity migration stops at 427 items with a successful return code despite clearly thinking there’s 559 total things to process.

I would also expect the Drupal Migrate Tools to provide some sort of useful stack trace or a list of which quantities couldn’t be migrated. It’s actually not even very clear whether that’s the reason the farm_migrate_log_activity migration can’t proceed - I can imaging lots of ways a dependency might not resolve.

Anybody have an idea what’s going on here or what steps a typical site admin would try next?

[Symbioquine]

The previous issues are still unresolved, but I decided to see how far I could get with the rest of the migration by adding --force to the drush migrate:import --group=farm_migrate_log command…

           drush --root=/opt/drupal pm-enable --yes farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
           drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
           drush --root=/opt/drupal migrate:import --group=farm_migrate_area
           drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
-           drush --root=/opt/drupal migrate:import --group=farm_migrate_log
+           drush --root=/opt/drupal migrate:import --force --group=farm_migrate_log
           drush --root=/opt/drupal migrate:import --group=farm_migrate_reference

Which got me far enough to encounter the problem the docs suggested one might have with multiple geometries;

166/414 [===========>----------------] 40% [error] Migration failed with source plugin exception: Log 7688 has both a geometry and a movement geometry. in /opt/drupal/web/profiles/farm/modules/core/migrate/src/Plugin/migrate/source/d7/FarmLog.php line 119

But there were just two of those and they were easily resolved by abusing modifying my input data;

# Delete the offending log geometry fields;
docker-compose exec db /bin/sh -c "echo \"DELETE FROM field_data_field_farm_geofield WHERE entity_type = 'log' AND bundle = 'farm_activity' AND entity_id IN (7688, 7689);\" | mysql -u farm -pfarm -D farm"

Which allows the farm_migrate_log and farm_migrate_reference migration groups to complete;

$ docker-compose exec www2 drush migrate:import  --update --force --group=farm_migrate_log
 [notice] Processed 414 items (0 created, 414 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_activity'
 [notice] Processed 72 items (0 created, 72 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_harvest'
 [notice] Processed 2 items (0 created, 2 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_input'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_maintenance'
 [notice] Processed 7015 items (0 created, 7015 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_observation'
 [notice] Processed 349 items (0 created, 349 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_seeding'
$ docker-compose exec www2 drush --root=/opt/drupal migrate:import --group=farm_migrate_reference
 [notice] Processed 352 items (352 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_field_parent'
 [notice] Processed 2945 items (441 created, 0 updated, 0 failed, 2504 ignored) - done with 'farm_migrate_asset_field_parent'

So to summarize my current status is as follows;

• Almost everything migrated
• Need to look into how to find/troubleshoot the 132 quantities which aren’t migrating
• Need to look into scripting some sort of parity mechanism to validate what data is missing
• Need to port custom stuff to farmOS 2.x and write migration logic for it

[mstenta]

Awesome record keeping @Symbioquine ! Thanks for sharing all this!

Regarding the quantity migration issue: we just added this recently, so I’m curious if your data might have something we overlooked in our testing. We probably also need to add some logging to it so that drush migrate:messages shows some clues!

Here is the code for the quantity migrate source plugin (for gathering quantity data from the old database): farmOS/Quantity.php at 2.x · farmOS/farmOS · GitHub

And here is the migration configuration: farmOS/migrate_plus.migration.farm_migrate_quantity.yml at 2.x · farmOS/farmOS · GitHub

There’s not much to them… so perhaps putting a breakpoint in the source class’s query() method would lead to some clues…

[mstenta]

Oh! I just had a hunch: do 132 of your quantities have a blank “Value” field, perhaps?

Currently the value field is required on quantities. I plan to fix this soon… though we don’t have an open issue for it yet.

[Symbioquine]

Thanks for the thoughts @mstenta!

Looking into the quantity stuff deeper…

I left it out of earlier posts for simplicity, but I’m actually logging all SQL queries to the 1.x DB;

docker-compose.yml

 version: '3.7'
 services:
   db:
     image: mariadb:latest
     volumes:
       - './db:/var/lib/mysql'
+      - './db.cnf:/etc/mysql/conf.d/my.cnf'
+      - './dblogs:/var/log/mysql'
       - './db-init:/docker-entrypoint-initdb.d'
     ports:
       - '3306:3306'

db.cnf

[mysqld]
general_log = 1
general_log_file = /var/log/mysql/query.log

slow_query_log = 1
long_query_time = 1 # seconds
slow_query_log_file = /var/log/mysql/slow.log
log_queries_not_using_indexes = 0

From which, I can find the query which is making the farm_migrate_quantity migration think it has 559 items to migrate, but only complete 427 of them; (formatted and added the sql_mode/LIMIT parts to make it more readable/runnable)

SET
    sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY';
SELECT DISTINCT
    "fci"."item_id" AS "id",
    "fdffqm"."field_farm_quantity_measure_value" AS "measure",
    "fdffqv"."field_farm_quantity_value_numerator" AS "value_numerator",
    "fdffqv"."field_farm_quantity_value_denominator" AS "value_denominator",
    "fdffqu"."field_farm_quantity_units_tid" AS "units",
    "fdffql"."field_farm_quantity_label_value" AS "label",
    "l"."uid" AS "uid",
    1 AS "expression"
FROM
    "field_collection_item" "fci"
LEFT OUTER JOIN "field_data_field_farm_quantity_measure" "fdffqm" ON
    fci.item_id = fdffqm.entity_id
LEFT OUTER JOIN "field_data_field_farm_quantity_value" "fdffqv" ON
    fci.item_id = fdffqv.entity_id
LEFT OUTER JOIN "field_data_field_farm_quantity_units" "fdffqu" ON
    fci.item_id = fdffqu.entity_id
LEFT OUTER JOIN "field_data_field_farm_quantity_label" "fdffql" ON
    fci.item_id = fdffql.entity_id
LEFT JOIN "field_data_field_farm_quantity" "fdffq" ON
    fci.item_id = fdffq.field_farm_quantity_value
LEFT JOIN "log" "l" ON
    fdffq.entity_type = 'log' AND fdffq.entity_id = l.id
WHERE
    ("fci"."archived" = '0') AND("fdffq"."deleted" = '0')
LIMIT 1000

Which yields;

From there, I notice the queries don’t follow the normal “Drupal” pattern of also filtering by “entity_type”, “bundle”, and “deleted”. That would be a problem if there were overlapping records from other entity types or bundles…

Modifying the query now to be a bit more pedantic;

 SET
     sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY';
 SELECT DISTINCT
     "fci"."item_id" AS "id",
     "fdffqm"."field_farm_quantity_measure_value" AS "measure",
     "fdffqv"."field_farm_quantity_value_numerator" AS "value_numerator",
     "fdffqv"."field_farm_quantity_value_denominator" AS "value_denominator",
     "fdffqu"."field_farm_quantity_units_tid" AS "units",
     "fdffql"."field_farm_quantity_label_value" AS "label",
     "l"."uid" AS "uid",
     1 AS "expression"
 FROM
     "field_collection_item" "fci"
 LEFT OUTER JOIN "field_data_field_farm_quantity_measure" "fdffqm" ON
-    fci.item_id = fdffqm.entity_id
+    fci.item_id = fdffqm.entity_id AND fdffqm.entity_type = 'field_collection_item' AND fdffqm.bundle = 'field_farm_quantity' AND fdffqm.deleted = '0'
 LEFT OUTER JOIN "field_data_field_farm_quantity_value" "fdffqv" ON
-    fci.item_id = fdffqv.entity_id
+    fci.item_id = fdffqv.entity_id AND fdffqv.entity_type = 'field_collection_item' AND fdffqv.bundle = 'field_farm_quantity' AND fdffqv.deleted = '0'
 LEFT OUTER JOIN "field_data_field_farm_quantity_units" "fdffqu" ON
-    fci.item_id = fdffqu.entity_id
+    fci.item_id = fdffqu.entity_id AND fdffqu.entity_type = 'field_collection_item' AND fdffqu.bundle = 'field_farm_quantity' AND fdffqu.deleted = '0'
 LEFT OUTER JOIN "field_data_field_farm_quantity_label" "fdffql" ON
-    fci.item_id = fdffql.entity_id
+    fci.item_id = fdffql.entity_id AND fdffql.entity_type = 'field_collection_item' AND fdffql.bundle = 'field_farm_quantity' AND fdffql.deleted = '0'
 LEFT JOIN "field_data_field_farm_quantity" "fdffq" ON
     fci.item_id = fdffq.field_farm_quantity_value
 LEFT JOIN "log" "l" ON
     fdffq.entity_type = 'log' AND fdffq.entity_id = l.id
 WHERE
     ("fci"."archived" = '0') AND("fdffq"."deleted" = '0')
 LIMIT 1000

Which in turn yields;

Great, now to patch the actually migration code…

[Symbioquine]

Now with this patch we can try again;

   www2:
     depends_on:
       - db2
     image: farmos/farmos:2.x-dev
     entrypoint: /bin/bash
     command:
       - -c
       - |
         set -ex
 
         wait_db_ready() {
             while { ! exec 3<>/dev/tcp/db2/5432; } > /dev/null 2>&1; do sleep 0.1; done
         }
 
         if [ -d /opt/drupal ] && ! [ "$$(ls -A /opt/drupal/composer.json)" ]; then
           echo "farmOS codebase not detected. Copying from pre-built files in the Docker image."
           cp -rp /var/farmOS/. /opt/drupal
           wait_db_ready
           su www-data -s /bin/bash -c 'drush site-install farm --locale=en --db-url=pgsql://farm:farm@db2/farm --site-name=Test0 --account-name=root --account-pass=test'
 
           echo "
         \$$databases['migrate']['default'] = [
           'database' => 'farm',
           'username' => 'farm',
           'password' => 'farm',
           'prefix' => '',
           'host' => 'db',
           'port' => '3306',
           'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
           'driver' => 'mysql',
         ];
         " >> /opt/drupal/web/sites/default/settings.php
 
           cp -r /opt/farmOS_1x_files /opt/drupal/web/sites/default/files/migrate
           chown -R www-data:www-data /opt/drupal/web/sites/default/files/migrate
 
+          cp /opt/testing_patched/Quantity.php /opt/drupal/web/profiles/farm/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php
+          chown www-data:www-data /opt/drupal/web/profiles/farm/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php
         fi
 
         wait_db_ready
 
         (
         su www-data -s /bin/bash <<'EOF'
           set -ex
 
           drush --root=/opt/drupal pm-enable --yes farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
           drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
           drush --root=/opt/drupal migrate:import --group=farm_migrate_area
           drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
-          drush --root=/opt/drupal migrate:import --force --group=farm_migrate_log
+          drush --root=/opt/drupal migrate:import --group=farm_migrate_log
           drush --root=/opt/drupal migrate:import --group=farm_migrate_reference
 
         EOF
         ) || echo "failed"
 
         exec docker-entrypoint.sh apache2-foreground
     volumes:
       - './www2:/opt/drupal'
       - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
       - './sites/default/files:/opt/farmOS_1x_files:ro'
+      - '../../farmOS/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php:/opt/testing_patched/Quantity.php:ro'
     expose:
       - '80'
     environment:
       XDEBUG_MODE: debug

SUCCESS!

[notice] Processed 427 items (427 created, 0 updated, 0 failed, 0 ignored) - done with ‘farm_migrate_quantity’

… and the farm_migrate_log migration group succeeded without --force too.

[mstenta]

Great sleuthing @Symbioquine ! I’ve merged your patch upstream. Thanks again for sharing your process here and helping to identify that bug!

[Symbioquine]

Yeah, thanks @mstenta! Now with that fix merged in we can pull the latest docker image and re-run without the manual patching;

$ docker pull farmos/farmos:2.x-dev
2.x-dev: Pulling from farmos/farmos
45b42c59be33: Already exists 
...
c4cd65145065: Pull complete 
Digest: sha256:bc4a80515cec6f368b2cf602ec3fdcad99e605620446fae2925594fc1d957505
Status: Downloaded newer image for farmos/farmos:2.x-dev
docker.io/farmos/farmos:2.x-dev

   www2:
     depends_on:
       - db2
     image: farmos/farmos:2.x-dev
     entrypoint: /bin/bash
     command:
       - -c
       - |
         set -ex
 
         wait_db_ready() {
 ...
 
-          cp /opt/testing_patched/Quantity.php /opt/drupal/web/profiles/farm/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php
-          chown www-data:www-data /opt/drupal/web/profiles/farm/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php
         fi
 
         wait_db_ready

 ...

     volumes:
       - './www2:/opt/drupal'
       - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
       - './sites/default/files:/opt/farmOS_1x_files:ro'
-      - '../../farmOS/modules/core/migrate/src/Plugin/migrate/source/d7/Quantity.php:/opt/testing_patched/Quantity.php:ro'
     expose:
       - '80'
     environment:
       XDEBUG_MODE: debug

Clean re-run;

$ docker-compose stop db2 www2 && docker-compose rm -f db2 www2 && sudo rm -rf db2 www2
$ docker-compose up -d
$ docker-compose logs --follow --no-log-prefix www2
Attaching to devel_www2_1
+ '[' -d /opt/drupal ']'
++ ls -A /opt/drupal/composer.json
ls: cannot access '/opt/drupal/composer.json': No such file or directory
farmOS codebase not detected. Copying from pre-built files in the Docker image.
+ '[' '' ']'
+ echo 'farmOS codebase not detected. Copying from pre-built files in the Docker image.'
+ cp -rp /var/farmOS/. /opt/drupal
+ wait_db_ready
+ su www-data -s /bin/bash -c 'drush site-install farm --locale=en --db-url=pgsql://farm:farm@db2/farm --site-name=Test0 --account-name=root --account-pass=test'

 You are about to:
 * Create a sites/default/settings.php file
 * DROP all tables in your 'farm' database.

 Do you want to continue? (yes/no) [yes]:
 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [notice] Performed install task: install_profile_modules
 [notice] Performed install task: install_profile_themes
 [notice] Performed install task: install_install_profile
 [notice] Performed install task: install_configure_form
 [notice] Performed install task: \Drupal\farm\Form\FarmModulesForm
 [warning] is_dir(): Unable to find the wrapper "private" - did you forget to enable it when you configured PHP? FileSystem.php:523
 [warning] is_dir(): Unable to find the wrapper "private" - did you forget to enable it when you configured PHP? FileSystem.php:523
 [notice] entity_reference_integrity_enforce.settings rewritten by farm_entity
 [notice] simple_oauth.settings rewritten by farm_api
 [notice] jsonapi.settings rewritten by farm_api
 [notice] jsonapi_extras.settings rewritten by farm_api
 [notice] gin.settings rewritten by farm_ui_theme
 [notice] Performed install task: farm_install_modules
 [notice] Cron run completed.
 [notice] Performed install task: install_finished
 [success] Installation complete.
+ echo '
$databases['\''migrate'\'']['\''default'\''] = [
  '\''database'\'' => '\''farm'\'',
  '\''username'\'' => '\''farm'\'',
  '\''password'\'' => '\''farm'\'',
  '\''prefix'\'' => '\'''\'',
  '\''host'\'' => '\''db'\'',
  '\''port'\'' => '\''3306'\'',
  '\''namespace'\'' => '\''Drupal\Core\Database\Driver\mysql'\'',
  '\''driver'\'' => '\''mysql'\'',
];
'
+ cp -r /opt/farmOS_1x_files /opt/drupal/web/sites/default/files/migrate
+ chown -R www-data:www-data /opt/drupal/web/sites/default/files/migrate
+ wait_db_ready
+ su www-data -s /bin/bash
+ drush --root=/opt/drupal pm-enable --yes farm_migrate
 > The following module(s) will be enabled: farm_migrate, migrate, migrate_drupal, migrate_plus, migrate_tools

 // Do you want to continue?: yes.                                              

 [success] Successfully enabled: farm_migrate, migrate, migrate_drupal, migrate_plus, migrate_tools
+ drush --root=/opt/drupal migrate:import --group=farm_migrate

 1/1 [============================] 100% [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_system_date'

 7/7 [============================] 100% [notice] Processed 7 items (7 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_user'

 728/728 [============================] 100% [notice] Processed 728 items (728 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_file'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_file_private'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy

 8/8 [============================] 100% [notice] Processed 8 items (8 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_animal_type'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_crop_family'

 372/372 [============================] 100% [notice] Processed 372 items (372 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_plant_type'

 57/57 [============================] 100% [notice] Processed 57 items (57 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_season'

 8/8 [============================] 100% [notice] Processed 8 items (8 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_log_category'

 5/5 [============================] 100% [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_unit'

 1/1 [============================] 100% [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_taxonomy_material'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_asset

 67/67 [============================] 100% [notice] Processed 67 items (67 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_asset_animal'

 2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_asset_equipment'

 372/372 [============================] 100% [notice] Processed 372 items (372 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_asset_plant'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_area

 179/179 [============================] 100% [notice] Processed 179 items (179 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_land'

 171/171 [============================] 100% [notice] Processed 171 items (171 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_none'

 2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_structure'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_water'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity

 427/427 [============================] 100% [notice] Processed 427 items (427 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_quantity'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_log

 414/414 [============================] 100% [notice] Processed 414 items (414 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_activity'

 72/72 [============================] 100% [notice] Processed 72 items (72 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_harvest'

 2/2 [============================] 100% [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_input'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_maintenance'

 7015/7015 [============================] 100% [notice] Processed 7015 items (7015 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_observation'

 349/349 [============================] 100% [notice] Processed 349 items (349 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_seeding'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_reference

 352/352 [============================] 100% [notice] Processed 352 items (352 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_area_field_parent'

  435/2945 [====>-----------------------]  14% [notice] Processed 2945 items (441 created, 0 updated, 0 failed, 2504 ignored) - done with 'farm_migrate_asset_field_parent'
+ exec docker-entrypoint.sh apache2-foreground
Attempting: apache2-foreground
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.14. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.14. Set the 'ServerName' directive globally to suppress this message
[Sun Feb 14 20:44:02.377157 2021] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.4.15 configured -- resuming normal operations
[Sun Feb 14 20:44:02.377182 2021] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

Of interest here is the 2504 ignored items from the farm_migrate_asset_field_parent migration;

[notice] Processed 2945 items (441 created, 0 updated, 0 failed, 2504 ignored) - done with ‘farm_migrate_asset_field_parent’

That’s right in the same ballpark as the number of seed assets I have - which aren’t expected to be migrating yet since I haven’t written a 2.x seed module…

So my current status is now as follows;

• Almost everything migrated
• Need to look into scripting some sort of parity mechanism to validate what data is missing
• Need to port custom stuff to farmOS 2.x and write migration logic for that

[mstenta]

Great!!

Need to port custom stuff to farmOS 2.x and write migration logic for that

Worth noting: I have been putting some thoughts about “contrib” migrations in this issue…

[META] farmOS 2.x Contrib Migrations

I haven’t updated that in a bit, though, so hopefully I haven’t forgotten any major changes.

I’m tying to design the migrations with contrib in mind, but we may still find some tricky cases - so eager to hear how it goes for you!

[Symbioquine]

@mstenta

Worth noting: I have been putting some thoughts about “contrib” migrations in this issue

Awesome, that looks helpful…!

@mstenta

I’m tying to design the migrations with contrib in mind, but we may still find some tricky cases - so eager to hear how it goes for you!

Yeah, I’m going to try and keep this thread going as a record of each step I end up taking. Hopefully it will serve as an example process and hopefully to highlight any tricky cases I encounter.

[Symbioquine]

Starting to work on a ‘seed asset’ module now. I’ve created the minimal code to have an installable farmOS module - which can be viewed in this farmOS_seed_asset commit.

Then I’ve updated my docker-compose.yml file to install that module;

          cp -r /opt/farmOS_1x_files /opt/drupal/web/sites/default/files/migrate
          chown -R www-data:www-data /opt/drupal/web/sites/default/files/migrate
        fi

        wait_db_ready

+       su www-data -s /bin/bash <<'EOF'
+           composer config repositories.farmos_dev_modules '{"type": "path", "url": "/farmos_dev_modules/farmos_seed_asset"}'
+
+           composer require symbioquine/farmos_seed_asset @dev
+
+           drush --root=/opt/drupal pm-enable farmos_seed_asset
+       EOF

        (
        su www-data -s /bin/bash <<'EOF'
          set -ex

          drush --root=/opt/drupal pm-enable --yes farm_migrate
 ...
    volumes:
      - './www2:/opt/drupal'
      - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
+     - '../../farmOS_seed_asset/farmos_seed_asset:/farmos_dev_modules/farmos_seed_asset'
+     - '../../farmOS_seed_asset/composer.json:/farmos_dev_modules/farmos_seed_asset/composer.json'
      - './sites/default/files:/opt/farmOS_1x_files:ro'

Which yields a farmOS 2.x instance with that module installed and the same effective migration state - since the new module doesn’t actually do anything yet…

[Symbioquine]

Next I’ve created the minimal custom asset type by adding the farmos_seed_asset/src/Plugin/Asset/AssetType/Seed.php and farmos_seed_asset/config/install/asset.type.seed.yml files to the module code - see this farmOS_seed_asset commit.

Which yields the standard farmOS 2.x asset fields;

…but still no migrated data;

[Symbioquine]

Next I’ve added a minimal farm_migrate_asset_seed migration by adding a farmos_seed_asset/config/optional/migrate_plus.migration.farm_migrate_asset_seed.yml file - see this farmOS_seed_asset commit.

2460/2460 [============================] 100% [notice] Processed 2460 items (2460 created, 0 updated, 0 failed, 0 ignored) - done with ‘farm_migrate_asset_seed’

Which yields all my seed assets “migrated”;

…but without much of their data yet;

One outstanding question is why the migration said 2460 seed assets, but both farmOS 1.x and 2.x seem to think there are 2459 seed assets…

==

[Symbioquine]

Seems like some of the fields like location should “just work”. Let’s dig into what’s going on there…

It turns out that there’s a sort of fallback provided for migrating 1.x areas without a type which turns them all into archived land assets of land_type ‘other’.

Since all my seed inventory “areas” are missing types, they fall into that logic and I assume that prevents them from being used as locations for my seed assets.

The easy fix would be to set the area type in 1.x to something like ‘other’, but it might be worth thinking about how I would ideally want this modeled under 2.x… With that mindset, it seems pretty clear that the “equipment” asset type would be a better choice than “land” for my seed boxes/tubs/bins/etc.

How could I do that…?

If I were going to run farmOS 1.x long term in parallel with 2.x, I might start maintaining a 1.x module to create an area type for those areas. However, I’m planning to keep testing this migration process locally until everything is good, then do it once and stop using 1.x. For that, it’ll be good enough just to temporarily hard-code a placeholder value in the 1.x database and write some migration logic on the 2.x side to handle that value…

First, let’s write a query to find all my seed areas in 1.x. Since they’re all children of an area called ‘OISB Inventory’, I can write a recursive query based on this SO answer…

SET
    sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY';
WITH RECURSIVE inventory_locations_cte (tid, parent) AS (
  SELECT     tid,
             parent
  FROM       taxonomy_term_hierarchy
  WHERE      parent = (SELECT tid FROM taxonomy_term_data WHERE name = 'OISB Inventory')
  UNION ALL
  SELECT     p.tid,
             p.parent
  FROM       taxonomy_term_hierarchy p
  INNER JOIN inventory_locations_cte
          ON p.parent = inventory_locations_cte.tid
)
SELECT * from inventory_locations_cte;

Next we can use that to construct a query which inserts ‘area_type’ values for each of those areas;

SET
    sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY';

INSERT INTO field_data_field_farm_area_type (entity_id, entity_type, bundle, deleted, language, delta, field_farm_area_type_value)
    WITH RECURSIVE inventory_locations_cte (tid, parent) AS (
          SELECT     tid,
                     parent
          FROM       taxonomy_term_hierarchy
          WHERE      parent = (SELECT tid FROM taxonomy_term_data WHERE name = 'OISB Inventory')
          UNION ALL
          SELECT     p.tid,
                     p.parent
          FROM       taxonomy_term_hierarchy p
          INNER JOIN inventory_locations_cte
                  ON p.parent = inventory_locations_cte.tid
        )
    SELECT
        inventory_locations_cte.tid AS entity_id,
        'taxonomy_term' AS entity_type,
        'farm_areas' AS bundle,
        0 AS deleted,
        'und' AS language,
        0 AS delta,
        'equipment_placeholder' AS field_farm_area_type_value
    FROM inventory_locations_cte
    ON DUPLICATE KEY UPDATE deleted=0;

Note: The ON DUPLICATE KEY UPDATE part is just to make it idempotent for convenience in repeated testing.

…and here’s my horrible one-liner to automate running that against my 1.x DB in docker;

docker-compose exec db /bin/sh -c "echo \"SET sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY'; INSERT INTO field_data_field_farm_area_type (entity_id, entity_type, bundle, deleted, language, delta, field_farm_area_type_value) WITH RECURSIVE inventory_locations_cte (tid, parent) AS ( SELECT tid, parent FROM taxonomy_term_hierarchy WHERE parent = (SELECT tid FROM taxonomy_term_data WHERE name = 'OISB Inventory') UNION ALL SELECT p.tid, p.parent FROM taxonomy_term_hierarchy p INNER JOIN inventory_locations_cte ON p.parent = inventory_locations_cte.tid ) SELECT inventory_locations_cte.tid AS entity_id, 'taxonomy_term' AS entity_type, 'farm_areas' AS bundle, 0 AS deleted, 'und' AS language, 0 AS delta, 'equipment_placeholder' AS field_farm_area_type_value FROM inventory_locations_cte ON DUPLICATE KEY UPDATE deleted=0;\" | mysql -u farm -pfarm -D farm"

Then I can copy the farm_migrate_area_none migration and introduce my own migration behavior for the areas that now have a 1.x area_type of ‘equipment_placeholder’;

migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml

langcode: en
status: true
dependencies:
  enforced:
    module:
      - farm_migrate
      - farm_land
id: farm_migrate_area_equipment_placeholder
label: 'Areas of equipment_placeholder type'
migration_group: farm_migrate_area
migration_tags:
  - 'Drupal 7'
  - 'farmOS 1.x'
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
source:
  plugin: d7_farm_area
  area_type:
    - equipment_placeholder
destination:
  plugin: 'entity:asset'
process:
  # Hard-code the bundle.
  type:
    plugin: default_value
    default_value: equipment
  is_location:
    plugin: default_value
    default_value: true
migration_dependencies:
  required: {  }
  optional: {  }

docker-compose.yml

   www2:
     depends_on:
       - db2
     image: farmos/farmos:2.x-dev
     entrypoint: /bin/bash
     command:
       - -c
       - |
         set -ex
 
         wait_db_ready() {
 ...
 
+          cp /opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml /opt/drupal/web/profiles/farm/modules/core/migrate/config/optional/
+          chown www-data:www-data /opt/drupal/web/profiles/farm/modules/core/migrate/config/optional/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml
         fi
 
         wait_db_ready

 ...

     volumes:
       - './www2:/opt/drupal'
       - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
       - './sites/default/files:/opt/farmOS_1x_files:ro'
+      - './migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:/opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:ro'
     expose:
       - '80'
     environment:
       XDEBUG_MODE: debug

Which yields my 168 equipment locations; (there were also 2 other equipment assets, hence the 170 number)

…but sadly, that’s not enough to make the seed assets retain their locations through the migration;

I guess I’ll have to look into that next.

[Symbioquine]

Looking at the logs, we can see that the 1.x log sets the location, but the migrated 2.x log does not;

My first suspicion was that perhaps the log source properties is_movement and field_farm_area aren’t getting set correctly in the FarmLog migrate plugin.

I haven’t added debugging logs or attached a debugger yet, but based on the MariaDB query log, I can reproduce the SQL queries which feed into that logic;

SET
    sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,ONLY_FULL_GROUP_BY';

SELECT "t".*
FROM
"field_data_field_farm_movement" "t"
WHERE ("entity_type" = 'log') AND ("entity_id" = '8201') AND ("deleted" = '0');

SELECT "fdffmt"."field_farm_move_to_tid" AS "tid"
FROM
"field_collection_item" "fci"
LEFT OUTER JOIN "field_data_field_farm_move_to" "fdffmt" ON fdffmt.entity_id = fci.item_id AND fdffmt.deleted = 0
WHERE ("fci"."item_id" = '8839') AND ("fci"."field_name" = 'field_farm_movement');

Returns the expected tid of ‘858’;

So under the assumption of the area tid getting set correctly and my vague understanding of how these lookup/mappings work, I would expect the log group location field logic to work as well for my custom seed location migration as it did for my plantings - which do have locations set;

So, not much progress yet. I’m still trying to figure out why some locations are working and others aren’t…