Today I resume testing 2.x migration.
This post is a continuation of Symbioquine's 2.x Migration Testing Log
Note: As with part 1, this post is not intended for direct use as a tutorial so don’t complain if these steps don’t work as-is for you - though I’m certainly happy to answer questions and take feedback that will be helpful to others.
In the past month there’s been a bunch of progress on farmOS 2.x and I want to make sure I’ve worked out any new migration hiccups…
I’ll start by looking at the diff between the version of the migration documentation which I referenced before and the current version;
Based on that, I need to make some updates to my docker-compose.yml file;
docker-compose.yml
(
su www-data -s /bin/bash <<'EOF'
set -ex
drush --root=/opt/drupal pm-enable --yes farm_group
drush --root=/opt/drupal pm-enable --yes farm_migrate
- drush --root=/opt/drupal migrate:import --group=farm_migrate
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_config
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_role
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_user
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_file
drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
drush --root=/opt/drupal migrate:import --group=farm_migrate_area
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_sensor_data
drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
drush --root=/opt/drupal migrate:import --group=farm_migrate_log
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_plan
drush --root=/opt/drupal migrate:import --group=farm_migrate_reference
EOF
) || echo "failed"
Next (as before) I blew away my 2.x DB and farmOS containers then re-ran the whole migration. Everything seemed to go swimmingly until the log migration step.
The log migration step produced the following error;
1/468 [>---------------------------] 0%
31/468 [=>--------------------------] 6% [error] Error: Unsupported operand types in Drupal\migrate\Plugin\migrate\process\SubProcess->transform() (line 206 of /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php) #0 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(401): Drupal\migrate\Plugin\migrate\process\SubProcess->transform(Array, Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'quantity')
#1 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(207): Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#2 /opt/drupal/vendor/drush/drush/includes/drush.inc(206): Drupal\migrate\MigrateExecutable->import()
#3 /opt/drupal/vendor/drush/drush/includes/drush.inc(197): drush_call_user_func_array(Array, Array)
#4 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(846): drush_op(Array)
#5 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'farm_migrate_lo...', Array)
#6 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(319): array_walk(Array, Array, Array)
#7 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('', Array)
#8 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#9 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#10 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /opt/drupal/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(311): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /opt/drupal/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /opt/drupal/vendor/symfony/console/Application.php(1027): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /opt/drupal/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /opt/drupal/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(118): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(49): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /opt/drupal/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#19 /opt/drupal/vendor/drush/drush/drush(4): require('/opt/drupal/ven...')
#20 {main}.
Error: Unsupported operand types in /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php on line 206 #0 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(401): Drupal\migrate\Plugin\migrate\process\SubProcess->transform(Array, Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'quantity')
#1 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(207): Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#2 /opt/drupal/vendor/drush/drush/includes/drush.inc(206): Drupal\migrate\MigrateExecutable->import()
#3 /opt/drupal/vendor/drush/drush/includes/drush.inc(197): drush_call_user_func_array(Array, Array)
#4 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(846): drush_op(Array)
#5 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'farm_migrate_lo...', Array)
#6 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(319): array_walk(Array, Array, Array)
#7 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('', Array)
#8 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#9 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#10 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /opt/drupal/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(311): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /opt/drupal/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /opt/drupal/vendor/symfony/console/Application.php(1027): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /opt/drupal/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /opt/drupal/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(118): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(49): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /opt/drupal/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#19 /opt/drupal/vendor/drush/drush/drush(4): require('/opt/drupal/ven...')
#20 {main}
Error: Unsupported operand types in Drupal\migrate\Plugin\migrate\process\SubProcess->transform() (line 206 of /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php).
[warning] Drush command terminated abnormally.
It looks like that’s something to do with how log’s quantities are being migrated. I’ll have to dig deeper to see what’s going wrong though…
Hmm this looks similar to something that happened to me recently… although it seemed to resolve itself somehow… it made me suspicious, but I thought it might have just been an oddity of the way I was doing things (frequent code changes, rollbacks, migrations, not starting completely “fresh”). Now hearing you report the same thing makes me want to try again… I’ll re-run my local migrations to see if I hit the same thing… let me know if you discover any more clues!
I updated SubProcess.php to log each key/value/row as it is processed;
foreach ($value as $key => $new_value) {
+ $migrate_executable->saveMessage("Transforming row value: " . print_r([$key, $new_value, $row], TRUE));
$new_row = new Row($new_value + $source);
And monkey-patched it into the farmOS 2.x container in my docker-compose.yml;
ln -sf /opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml /opt/drupal/web/profiles/farm/modules/core/migrate/config/optional/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml
+ ln -sf /opt/testing_src_tree/SubProcess.php /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
wait_db_ready
volumes:
- './www2:/opt/drupal'
- './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
- '../../farmOS_seed_asset/farmos_seed_asset:/farmos_dev_modules/farmos_seed_asset'
- '../../farmOS_seed_asset/composer.json:/farmos_dev_modules/farmos_seed_asset/composer.json'
- './sites/default/files:/opt/farmOS_1x_files:ro'
#- '../../farmOS/modules:/opt/testing_src_tree'
+ - './SubProcess.php:/opt/testing_src_tree/SubProcess.php'
- './migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:/opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:ro'
After blowing away my containers and re-running the migration, I discovered the log that’s failing has an inventory adjustment for a custom asset type which I haven’t ported to 2.x yet.
This manifests as a target_id in the source field_farm_asset field of the migration Row object;
[field_farm_asset] => Array(
[0] => Array(
[target_id] => 27
)
)
but an empty array in the destination asset field of the row;
[asset] => Array(
[0] => Array()
)
So the conclusion for this failure case is “expected behavior” - though I shouldn’t need to modify the migrate module source code to determine which row is failing and why…
Ah I was just going to ask if you were using inventory… that’s the only thing that changed with the quantity field migration on logs.
Can you make a simple stub migration for your custom asset type? Even if it just maps to an existing asset type for now, and just pulls in the name or something. That would at least allow for the mapping to happen, and serve as a placeholder for when you do port it.
This is pretty easy if you just copy one of the existing migration YML files like this one: farmOS/migrate_plus.migration.farm_migrate_asset_compost.yml at 2.x · farmOS/farmOS · GitHub
That compost one is the simplest, it doesn’t have any additional fields on it, and easy to search/replace “compost”. 
Easier to just port the asset type.
With that and a bit of head-scratching about why the inventory migration wasn’t happening, I got it working;
docker-compose.yml
su www-data -s /bin/bash <<'EOF'
- composer config repositories.farmos_dev_modules '{"type": "path", "url": "/farmos_dev_modules/farmos_seed_asset"}'
+ composer config repositories.farmos_dev_modules '{"type": "path", "url": "/farmos_dev_modules/*"}'
composer require symbioquine/farmos_seed_asset @dev
drush --root=/opt/drupal pm-enable farmos_seed_asset
+ composer require symbioquine/farmos_money_asset @dev
+ drush --root=/opt/drupal pm-enable farmos_money_asset
EOF
drush --root=/opt/drupal pm-enable --yes farm_group
+ drush --root=/opt/drupal pm-enable --yes farm_inventory
drush --root=/opt/drupal pm-enable --yes farm_migrate
drush --root=/opt/drupal migrate:import --group=farm_migrate_config
- './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
- '../../farmOS_seed_asset/farmos_seed_asset:/farmos_dev_modules/farmos_seed_asset'
- '../../farmOS_seed_asset/composer.json:/farmos_dev_modules/farmos_seed_asset/composer.json'
+
+ - '../../farmOS_money_asset/farmos_money_asset:/farmos_dev_modules/farmos_money_asset'
+ - '../../farmOS_money_asset/composer.json:/farmos_dev_modules/farmos_money_asset/composer.json'
+
- './sites/default/files:/opt/farmOS_1x_files:ro'
The farm_migrate_inventory and farm_migrate_log_activity then were able to complete successfully.
2669/2669 [============================] 100% [notice] Processed 2669 items (2669 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_inventory'
463/463 [============================] 100% [notice] Processed 463 items (463 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_quantity_standard'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_log
468/468 [============================] 100% [notice] Processed 468 items (468 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_activity'
Further, some new stuff is now pretty much working;
And to summarize my current status;
