Migration from V1 to V2

[ykir]

Good morning, everyone.

I installed on a hosted server farmos V1 in 2016 and being really happy with it, I have been using it until today for my farm.
(Actually having other apps on server I could not go beyond php version 7…)

however now that I have the right php version, I am migrating to the new version and as per the documentation, I am migrating from V1 to V2.

I am stuck at step 5 of the migration, where it is mentioned to install “farmOS Migrate”, if I go from web on the new version of farmos–Setup–Modules no migrate is mentioned, there are these available:

Land asset

Plant asset

Animal asset

Equipment asset

Structure asset

Water asset

Activity log

Observation log

Seeding log

Input log

Harvest log

Maintenance log

Standard quantity

farmOS Default Roles

farmOS Default Land Types

farmOS Default Structure Types

farmOS Inventory

Material asset

Seed asset

Sensor asset

Compost asset

Group asset

Transplanting log

Lab test log

Birth log

Medical log

farmOS Import CSV

farmOS KML

farmOS Import KML

farmOS Field Kit

farmOS Localization

farmOS Account Admin Role

Birth Quick Form

Group Quick Form

Movement Quick Form

Planting Quick Form

I should add that, I have SSH access to the server and I went to the FamOSv2/vendor/bin folder and gave the command ./drush farm_migrate:import and as a result it gave me, “There are no commands defined in the “farm_migrate” namespace.”

Also I will add, the settings.php file to be modified with the data from the V1 database is located at FarmOSv2/web/sites/default/settings.php location right?
I ask because according to the documentation in step 3 I should edit the data in the settings.php file, and if I do that, from the web I can then no longer access farmosV2, so I find it an inconsistency that makes me think I am editing the wrong file.
Or maybe I’m editing it the wrong way, but maybe that will come up later.

Thank you very much for your help, I tried to search for answers or guides but found nothing.

[Symbioquine]

Before you do anything else… please make a backup and describe your backup strategy here!

[Symbioquine]

Ideally, you’ve tested your backup restore strategy and confirmed that it yields a working/identical copy of your v1 instance on your local computer (or a temporary/ephemeral/development cloud instance if you like)

[mstenta]

Hi @ykir! First, I second what @Symbioquine said - always make sure you have a backup strategy that works.

That said, if done correctly, the migration from v1 to v2 is a process of setting up a separate v2 instance alongside your v1, and them “migrating” data from the v1 database to the v2. This means that you do not need to (and should not!) make any changes to your v1 farmOS. This is good too because it means you can keep that one running “as-is” while you experiment with migrating data to v2. Then once you are confident that everything works, you can wipe the v2 database and run a “final” migration, at which point you stop using your v1 database and start using your v2 as the “canonical” database moving forward.

To answer the two specific questions I heard:

@ykir

I am stuck at step 5 of the migration, where it is mentioned to install “farmOS Migrate”, if I go from web on the new version of farmos–Setup–Modules no migrate is mentioned

Yes good question… the “farmOS > Setup > Modules” list is actually a curated list of farmOS modules, but the farmOS Migrate module is not included there. In order to install it, you need to go to the big Drupal module list at /admin/modules (do not get tempted to install other modules on the list! They are not all tested with farmOS ). Another way to do it is via the command line using Drush with drush en farm_migrate.

@ykir

Also I will add, the settings.php file to be modified with the data from the V1 database is located at FarmOSv2/web/sites/default/settings.php location right?

Yes, but you will add an additional block of v1 database credentials, alongside your v2 database credentials. Sorry if that was unclear in the documentation.

Notice that the example codeblock is setting a variable called $databases['migrate']['default'], whereas the normal v2 database credentials are in a variable called $databases['default']['default']:

 $databases['migrate']['default'] = [
   'database' => 'my_farmos_1x_db',
   'username' => 'my-db-username',
   'password' => 'my-db-password',
   'prefix' => '',
   'host' => 'localhost',
   'port' => '3306',
   'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
   'driver' => 'mysql',
 ];

This is how Drupal is able to access multiple databases. And for the farmOS migration, it expects a database called migrate to be defined.

Hope that helps! As long as you are keeping you v1 and v2 databases and code separate, you should be able to experiment with the migration until you feel like it’s working properly for you. Then do a final migration and you’ll be ready to upgrade to farmOS v3!

One last question: have you made any customizations? Custom modules? Add-on modules? If so you will need to make sure they have an upgrade path to v2+ as well.

Good luck!

[ykir]

In addition to having an automatic daily, 7-day backup function of both the web space and the database active, before I get my hands on it I make a local copy of the database and the zip packages of the apps I have on the web space (one zip for farmosv1, one for farmosv2, etc.) and so far this procedure has taken me out of the mess as in an extreme case I delete everything and reload the backups.

I have tried in the past to run everything locally to work here first and then upload but unfortunately I can’t recreate settings identical to those on both the web space and the database and I always end up losing something.

I specify that I am a farmer with IT passion but I don’t have that kind of training

[ykir]

Yes good question… the “farmOS > Setup > Modules” list is actually a curated list of farmOS modules, but the farmOS Migrate module is not included there. In order to install it, you need to go to the big Drupal module list at /admin/modules (do not get tempted to install other modules on the list! They are not all tested with farmOS ). Another way to do it is via the command line using Drush with drush en farm_migrate .

Worked first time adding the module, thank you! I used the command line

@mstenta

Yes, but you will add an additional block of v1 database credentials, alongside your v2 database credentials. Sorry if that was unclear in the documentation.

Okay I had a doubt that it should be done this way, now I’ve done it. Thank you!

@mstenta

One last question: have you made any customizations? Custom modules? Add-on modules? If so you will need to make sure they have an upgrade path to v2+ as well.

No custom forms, I have always just used the ones already present!

[ykir]

After making the necessary corrections, I tried the
./drush farm_migrate:import
command again and got this result:

[notice] Importing migration group: farm_migrate_config
[warning] Undefined array key “tag” MigrateToolsCommands.php:463
[warning] Undefined array key “all” MigrateToolsCommands.php:464

In Connection.php line 196:

SQLSTATE[HY000] [1045] Access denied for user ‘Sql*******’@‘*********’ (using password: YES)

In Connection.php line 189:

SQLSTATE[HY000] [1045] Access denied for user ‘Sql*******’@‘*********’ (using password: YES)

I replaced my database user name after “Sql” and IP address with the * symbol to post it here.

I rechecked the data I entered in the settings.php file 3 times and it is correct.

It seems to me to be a problem due to my ssh credentials (I hope I am not saying the wrong thing but I cannot become root here as I am connected in ssh to the hosted server).

May I add that the IP address that comes up as a result before “(using password: YES)” is different from the one in my database ( and I don’t understand why)

[mstenta]

You’re on the right track now @ykir!

The credentials in settings.php need to be SQL credentials, not SSH credentials. Is that what you are doing?

The remote SQL server will need to be accessible to this host, which means you need to be able run mysql -u [db-user] -h [remote-host] [db-name] on your command line and successfully connect to it. Get that working before you try to use the migration commands at all. Because then you’ll know that the connection is usable as a first step.

There are various guides online for doing this. Here’s one I found quickly (but haven’t tried myself, so YMMV): How To Allow Remote Access to MySQL | DigitalOcean

Another, perhaps safer, option would be to set up a copy of your v1 database on the same host as you v2 database. Then you can point your settings.php to that local database instead of the remote one. And there is no need to allow remote access to your other database.

[ykir]

Meanwhile, thank you very much for your help @mstenta

Then the credentials I entered in the settings.php file are the database credentials, the database is unique for both V1 and V2 versions, they are on the same machine, same IP address, same username and password, only the database name changes.

Currently the connection between web space works, either because the version of farmosV1 I have used so far has always worked, or because on the same web space I have other apps connected to the same database that work.

I tried giving the command

@mstenta

mysql -u [db-user] -h [remote-host] [db-name]

, compiling it properly, and it gave me this result.

ERROR 1045 (28000): Access denied for user ‘Sql*******’@‘DIFFERENT_IP_OF_MY_DATABASE’ (using password: NO)

I read the guide you linked but I don’t have access to the file

/etc/mysql/mysql.conf.d/mysqld.cnf

from webspace I only have access to my home folder, the one the provider assigned me, from database I only have access to phpmyadmin

I understand that there seems to be a lack of communication between database and webspace but I don’t understand how come the other applications I have work fine, even the new farmosV2 installation communicates with the database without problems.

The only thing I notice strange is that the error results it gives me and which I have already written down, I get back with a slightly different database ip address than the right one.

[mstenta]

Hmm it sounds like you have everything set up correctly. Both v1 and v2 servers, and v1 and v2 databases are all on the same physical machine?

Perhaps your web host creates some kind of isolation between different applications? Maybe it would be worth asking for support from your host? It might be something simple with their configuration.

@ykir

The only thing I notice strange is that the error results it gives me and which I have already written down, I get back with a slightly different database ip address than the right one.

It may be that your host uses an internal IP address for the machine. This is often done to avoid sending traffic out to the internet and back to the local network, so instead it can just use the local network directly. So depending on where you are running a command from, it may use the external IP vs the internal IP. That may explain the difference, but that’s just a guess.

[ykir]

I am the classic example of the inattentive user.

I checked the sql data 3 TIMES, reversed an l with a 1 in the password …

I guess I need to make the characters on the screen a little bigger.

Ok I started the migration and got some results, it stopped at this row.

It remain stuck in import if re-run the import.

[mstenta]

Glad you got it figured out @ykir!

As for that error… I forgot all about it! It’s actually a known issue with MySQL migrations. Ultimately we decided not to fix it (because most people migrated to the recommended PostgreSQL instead, and no one offered to help fix the MySQL migration), but I can explain to you how to work around it…

Here is the bug report: Migrations failing on MySQL · Issue #506 · farmOS/farmOS · GitHub

And specifically, this comment shows the one line you need to comment out to work around it: Migrations failing on MySQL · Issue #506 · farmOS/farmOS · GitHub

Please note: by commenting this out, you will not get revision logs added to each record that says “Migrated from farmOS 1.x on …”. But if you don’t mind that, then this should work for you.

That comment thread also notes that they ran into another MySQL issue after this one… but I think it was related to the sensor module. Do you have that module installed on v1? If not, then you may not run into it.

Sorry the MySQL migration bugs were never fixed! I suppose it is worth asking: do you have the option to use PostgreSQL with farmOS v2? Now would be the best opportunity to switch, if so. But if not, that’s OK. We still test against MySQL/MariaDB… it was just the migration that had an issue I think.

[ykir]

Unfortunately, the MySQL database I already use for other apps and changing it would become complicated, if I could do that it would be great, otherwise I may consider buying another database space.

I had the sensor module active but never used it.

I disabled both the listener module and the sensor module I tried the migration.

I encountered an error regarding the translation module (localization) that another user here has already exposed and as indicated in the replies I disabled the localization modules and overcame the error…

…to land on another ( )

I find that in V2 is mentioned farm_migrate_plan

I found an explanation here —> Plans | farmOS

but i didn’t understand if it is missing data in my V1 or a missing module in V2

[mstenta]

You might be done @ykir!

The farm_migrate_plan migration group was reserved for migrating v1 plans to v2, but there weren’t many plan-type module that ended up providing a migration path from v1->v2. So if you didn’t have any add-on plan modules (like Crop Plan or Grazing Plan) installed in v1, then there is nothing to migrate.

Did all the other migrations proceed smoothly? Are you able to see all your data in the v2 instance? Be sure to check asset and log counts to make sure they are the same.

The migration command line tool will also show you how many of each were migrated when you run drush migrate:status --tag="farmOS 1.x"

[ykir]

Almost everything is there, only some files are missing that I actually don’t know why they are not even in the web space, but all the others (more than 300 images and pdfs) have been transferred perfectly.

It seems to be ok, really thanks for the help and again congratulations for the work you are and are doing!

I’m going to back everything up now and then start documenting for the upgrade to V3

[mstenta]

Great!! Glad it all worked!

@ykir

I’m going to back everything up now and then start documenting for the upgrade to V3

This should be pretty straightforward, but feel free to reach out with questions. We can update the title of this thread to “Migration from V1 to V2, update to V3” if we need to.

Here is the update guide: Updating farmOS | farmOS

[ykir]

I finally ran the update to version 3.2.1

everything went well, if anyone needs it I describe how I did it.

From ssh I downloaded with the wget command the latest release and extracted it to a test folder.
Inside the newly extracted folder I deleted the web/sites folder (as stated in the documentation) and used the cp command to copy the contents of the new release to the old release folder.

Specifically, the command was:

\cp -r 3.2.1/farmOS/* farmOS

finally with the drush command I performed the database update

vendor/bin/./drush updb

thanks @mstenta for the help i would never have been able to do it without it, but mainly because i have not kept farmos updated all these years, as in the end it was really easy.

Really wonderful V3 version, congratulations to you and all!