# Symbioquine’s 2.x Migration Testing Log Part 2

**URL:** <https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771>\
**Category:** Development\
**Created:** [April 21, 2021, 5:21pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771 "2021-04-21T17:21:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\
**Post date:** [April 21, 2021, 5:21pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/1 "2021-04-21T17:21:52Z")

</div>

Today I resume testing 2.x migration.

This post is a continuation of [Symbioquine's 2.x Migration Testing Log](https://farmos.discourse.group/t/symbioquines-2-x-migration-testing-log/660)

_Note: As with part 1, this post is not intended for direct use as a tutorial so don’t complain if these steps don’t work as-is for you - though I’m certainly happy to answer questions and take feedback that will be helpful to others._

In the past month there’s been a bunch of progress on farmOS 2.x and I want to make sure I’ve worked out any new migration hiccups…

---

<div class="post-metadata">

**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\
**Post date:** [April 21, 2021, 5:29pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/2 "2021-04-21T17:29:12Z")

</div>

I’ll start by looking at the diff between the version of the migration documentation which I referenced before and the current version;

> **[farmOS/farmOS](https://github.com/farmOS/farmOS/compare/6010d3af8f46b7bdce7f28087ca059375b6b94a8...farmOS:2.x#diff-3675db44044b43cbbf275cf977e6e9bca454b2891870f7f52bb97dfb77bdd0e6)**
>
> re/6010d3af8f46b7bdce7f28087ca059375b6b94a8...farmOS:2.x

Based on that, I need to make some updates to my **docker-compose.yml** file;

**docker-compose.yml**

```diff
         (
         su www-data -s /bin/bash <<'EOF'
           set -ex
 
           drush --root=/opt/drupal pm-enable --yes farm_group
           drush --root=/opt/drupal pm-enable --yes farm_migrate
- drush --root=/opt/drupal migrate:import --group=farm_migrate
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_config
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_role
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_user
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_file
           drush --root=/opt/drupal migrate:import --group=farm_migrate_taxonomy
           drush --root=/opt/drupal migrate:import --group=farm_migrate_asset
           drush --root=/opt/drupal migrate:import --group=farm_migrate_area
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_sensor_data
           drush --root=/opt/drupal migrate:import --group=farm_migrate_quantity
           drush --root=/opt/drupal migrate:import --group=farm_migrate_log
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_plan
          drush --root=/opt/drupal migrate:import --group=farm_migrate_reference
         
         EOF
         ) || echo "failed"

```

---

<div class="post-metadata">

**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\
**Post date:** [April 22, 2021, 4:34am UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/3 "2021-04-22T04:34:54Z")

</div>

Next (as before) I blew away my 2.x DB and farmOS containers then re-ran the whole migration. Everything seemed to go swimmingly until the log migration step.

The log migration step produced the following error;

```auto
   1/468 [>---------------------------] 0%
  31/468 [=>--------------------------] 6% [error] Error: Unsupported operand types in Drupal\migrate\Plugin\migrate\process\SubProcess->transform() (line 206 of /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php) #0 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(401): Drupal\migrate\Plugin\migrate\process\SubProcess->transform(Array, Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'quantity')
#1 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(207): Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#2 /opt/drupal/vendor/drush/drush/includes/drush.inc(206): Drupal\migrate\MigrateExecutable->import()
#3 /opt/drupal/vendor/drush/drush/includes/drush.inc(197): drush_call_user_func_array(Array, Array)
#4 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(846): drush_op(Array)
#5 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'farm_migrate_lo...', Array)
#6 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(319): array_walk(Array, Array, Array)
#7 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('', Array)
#8 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#9 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#10 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /opt/drupal/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(311): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /opt/drupal/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /opt/drupal/vendor/symfony/console/Application.php(1027): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /opt/drupal/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /opt/drupal/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(118): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(49): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /opt/drupal/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#19 /opt/drupal/vendor/drush/drush/drush(4): require('/opt/drupal/ven...')
#20 {main}. 
Error: Unsupported operand types in /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php on line 206 #0 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(401): Drupal\migrate\Plugin\migrate\process\SubProcess->transform(Array, Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'quantity')
#1 /opt/drupal/web/core/modules/migrate/src/MigrateExecutable.php(207): Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#2 /opt/drupal/vendor/drush/drush/includes/drush.inc(206): Drupal\migrate\MigrateExecutable->import()
#3 /opt/drupal/vendor/drush/drush/includes/drush.inc(197): drush_call_user_func_array(Array, Array)
#4 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(846): drush_op(Array)
#5 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'farm_migrate_lo...', Array)
#6 /opt/drupal/web/modules/migrate_tools/src/Commands/MigrateToolsCommands.php(319): array_walk(Array, Array, Array)
#7 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('', Array)
#8 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#9 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#10 /opt/drupal/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /opt/drupal/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(311): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /opt/drupal/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /opt/drupal/vendor/symfony/console/Application.php(1027): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /opt/drupal/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /opt/drupal/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(118): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /opt/drupal/vendor/drush/drush/src/Runtime/Runtime.php(49): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /opt/drupal/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#19 /opt/drupal/vendor/drush/drush/drush(4): require('/opt/drupal/ven...')
#20 {main}
Error: Unsupported operand types in Drupal\migrate\Plugin\migrate\process\SubProcess->transform() (line 206 of /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php).
 [warning] Drush command terminated abnormally.

```

It looks like that’s something to do with how log’s quantities are being migrated. I’ll have to dig deeper to see what’s going wrong though…

---

<div class="post-metadata">

**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\
**Post date:** [April 22, 2021, 12:05pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/4 "2021-04-22T12:05:20Z")

</div>

Hmm this looks similar to something that happened to me recently… although it seemed to resolve itself somehow… it made me suspicious, but I thought it might have just been an oddity of the way I was doing things (frequent code changes, rollbacks, migrations, not starting completely “fresh”). Now hearing you report the same thing makes me want to try again… I’ll re-run my local migrations to see if I hit the same thing… let me know if you discover any more clues!

---

<div class="post-metadata">

**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\
**Post date:** [April 22, 2021, 4:07pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/5 "2021-04-22T16:07:48Z")

</div>

> [@mstenta](#):
>
> let me know if you discover any more clues!

I updated [SubProcess.php](https://github.com/drupal/drupal/blob/aea84a7cc95a511bd29b30be7ba3210ac22cc7a7/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php#L206) to log each key/value/row as it is processed;

```diff
       foreach ($value as $key => $new_value) {
+ $migrate_executable->saveMessage("Transforming row value: " . print_r([$key, $new_value, $row], TRUE));
         $new_row = new Row($new_value + $source);

```

And monkey-patched it into the farmOS 2.x container in my **docker-compose.yml** ;

```diff
         ln -sf /opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml /opt/drupal/web/profiles/farm/modules/core/migrate/config/optional/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml
 
+ ln -sf /opt/testing_src_tree/SubProcess.php /opt/drupal/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
 
         wait_db_ready

```

```diff
     volumes:
       - './www2:/opt/drupal'
       - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
       - '../../farmOS_seed_asset/farmos_seed_asset:/farmos_dev_modules/farmos_seed_asset'
       - '../../farmOS_seed_asset/composer.json:/farmos_dev_modules/farmos_seed_asset/composer.json'
       - './sites/default/files:/opt/farmOS_1x_files:ro'
       #- '../../farmOS/modules:/opt/testing_src_tree'
+ - './SubProcess.php:/opt/testing_src_tree/SubProcess.php'
       - './migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:/opt/custom_migrations/migrate_plus.migration.farm_migrate_area_equipment_placeholder.yml:ro'

```

After blowing away my containers and re-running the migration, I discovered the log that’s failing has an inventory adjustment for a custom asset type which I haven’t ported to 2.x yet.

This manifests as a `target_id` in the source `field_farm_asset` field of the migration Row object;

```auto
[field_farm_asset] => Array(
    [0] => Array(
        [target_id] => 27
    )
)

```

but an empty array in the destination `asset` field of the row;

```auto
[asset] => Array(
    [0] => Array()
)

```

So the conclusion for this failure case is “expected behavior” - though I shouldn’t need to modify the migrate module source code to determine which row is failing and why…

---

<div class="post-metadata">

**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\
**Post date:** [April 22, 2021, 4:18pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/6 "2021-04-22T16:18:35Z")

</div>

Ah I was just going to ask if you were using inventory… that’s the only thing that changed with the quantity field migration on logs.

Can you make a simple stub migration for your custom asset type? Even if it just maps to an existing asset type for now, and just pulls in the name or something. That would at least allow for the mapping to happen, and serve as a placeholder for when you do port it.

This is pretty easy if you just copy one of the existing migration YML files like this one: [farmOS/migrate\_plus.migration.farm\_migrate\_asset\_compost.yml at 2.x · farmOS/farmOS · GitHub](https://github.com/farmOS/farmOS/blob/2.x/modules/core/migrate/config/optional/migrate_plus.migration.farm_migrate_asset_compost.yml)

That compost one is the simplest, it doesn’t have any additional fields on it, and easy to search/replace “compost”. 🙂

---

<div class="post-metadata">

**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\
**Post date:** [April 22, 2021, 7:15pm UTC](https://farmos.discourse.group/t/symbioquine-s-2-x-migration-testing-log-part-2/771/7 "2021-04-22T19:15:20Z")

</div>

> [@mstenta](#):
>
> Can you make a simple stub migration for your custom asset type? Even if it just maps to an existing asset type for now, and just pulls in the name or something. That would at least allow for the mapping to happen, and serve as a placeholder for when you do port it.

Easier to just port the [asset type](https://github.com/symbioquine/farmOS_money_asset).

With that and a bit of head-scratching about why the inventory migration wasn’t happening, I got it working;

**docker-compose.yml**

```diff
         su www-data -s /bin/bash <<'EOF'
- composer config repositories.farmos_dev_modules '{"type": "path", "url": "/farmos_dev_modules/farmos_seed_asset"}'
+ composer config repositories.farmos_dev_modules '{"type": "path", "url": "/farmos_dev_modules/*"}'
 
             composer require symbioquine/farmos_seed_asset @dev
             drush --root=/opt/drupal pm-enable farmos_seed_asset
 
+ composer require symbioquine/farmos_money_asset @dev
+ drush --root=/opt/drupal pm-enable farmos_money_asset
         EOF

```

```diff
           drush --root=/opt/drupal pm-enable --yes farm_group
+ drush --root=/opt/drupal pm-enable --yes farm_inventory
           drush --root=/opt/drupal pm-enable --yes farm_migrate
           drush --root=/opt/drupal migrate:import --group=farm_migrate_config

```

```diff
       - './php-custom.ini:/usr/local/etc/php/conf.d/php-custom.ini'
 
       - '../../farmOS_seed_asset/farmos_seed_asset:/farmos_dev_modules/farmos_seed_asset'
       - '../../farmOS_seed_asset/composer.json:/farmos_dev_modules/farmos_seed_asset/composer.json'
+ 
+ - '../../farmOS_money_asset/farmos_money_asset:/farmos_dev_modules/farmos_money_asset'
+ - '../../farmOS_money_asset/composer.json:/farmos_dev_modules/farmos_money_asset/composer.json'
+ 
       - './sites/default/files:/opt/farmOS_1x_files:ro'

```

The `farm_migrate_inventory` and `farm_migrate_log_activity` then were able to complete successfully.

```sh
 2669/2669 [============================] 100% [notice] Processed 2669 items (2669 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_inventory'

 463/463 [============================] 100% [notice] Processed 463 items (463 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_quantity_standard'
+ drush --root=/opt/drupal migrate:import --group=farm_migrate_log

 468/468 [============================] 100% [notice] Processed 468 items (468 created, 0 updated, 0 failed, 0 ignored) - done with 'farm_migrate_log_activity'

```

Further, some new stuff is now pretty much working;

- Inventory quantities - including those for our custom asset types (though not with the right units yet)
- Group membership

 ![image](https://global.discourse-cdn.com/free1/uploads/farm_os/original/1X/e3c143c9b97500792800d80663bc916222cd86eb.jpeg) ![image](https://global.discourse-cdn.com/free1/uploads/farm_os/original/1X/11cf956dd33a9e139aec6f14851bd2731fe2ea7f.png)

And to summarize my current status;

- Almost all data migrated - now including inventory and group membership
- Need to look into scripting some sort of parity mechanism to validate what data is missing
- Need to expand seed asset fields and the associated migration logic
- Need to port some additional manual site customization and custom tools into 2.x modules
- Need to look into a few observed quirks;
  - Seed asset migration 2459 vs 2460 count discrepancy - [Symbioquine's 2.x Migration Testing Log - #23 by Symbioquine](https://farmos.discourse.group/t/symbioquines-2-x-migration-testing-log/660/23)
  - Figure out how to change inventory units for migration of custom asset types (ref: [#farmOS:matrix.org discussion](https://matrix.to/#/!rxZezKXfxOKJxTgNGs:matrix.org/$161670874328997GKRXk:matrix.org?via=matrix.org))

- It would be nice to change all uploaded files to be private as part of the migration