Upgrade from 3.5.1 to 4.0.0

I need some upgrade help/advice/guidance. I have a functioning 3.5.1 install which I believe now only uses farmOS modules installed in the correct location. My database is sqllite.

When I modify the docker compose file to use 4.0.0, then go to /update.php, I get the following the following error:

Incompatible modules
The following modules are installed, but they are incompatible with Drupal 11.3.5:

farmOS Asset Termination
Bee
Farm Eggs
Ledger
farmOS NRCS
farmOS Project plan
farmOS Weather

Are those modules not updated yet, or is that still an indication that something is still wrong on 3.5.1 before the upgrade is attempted? Is it safe to uninstall them to let the upgrade finish so I can get on version 4?

Thanks for helping out!

Troy

Hi @tlcarpenter69 - happy to help with this! Before doing anything else - be sure that you have a backup of your database that you can restore if you need to roll back! :slight_smile:

OK based on this I’m guessing that you have a docker-compose.yml file that is using the farmOS codebase from the Docker image, but is bind-mounting in a directory that you have manually downloaded modules into. Is this correct?

If so, you will need to update all of those modules at the same time as you update the codebase (which you are doing by changing the farmOS Docker image version).

But it would help to know exactly what you’re doing, because if you are bind-mounting the entire /opt/drupal/web/modules directory (including more than just the farmOS modules you listed) then you’ll need to do more… so please share your docker-compose.yml file in full here so that I can give you the right guidance.

All of the modules you listed have v4-compatible versions except farmOS Project Plan (but there is an open merge request for that which just needs @paul121’s review and merge: https://www.drupal.org/project/farm_project_plan/issues/3573622). So if you have Project plans you will need to wait for that module to provide an updated release. If you don’t have any Project plans, then you can uninstall that module and proceed with the upgrade…

It is NOT safe to uninstall them if you have any data that uses them! Specifically:

  • farmOS Asset Termination adds a boolean field to all logs to designate them as termination logs.
  • Bee adds a Beehive asset type.
  • Ledger adds Purchase and Sale log types.
  • Project Plan adds the Project plan type.

So if you have any of that kind of data, then you shouldn’t uninstall those modules.

The good news is: all of the modules you are using are simple ones, so your upgrade process won’t be too complicated.

Let me first take a moment to say: this is why the official recommendation in v4+ is to use Composer to manage your codebase and modules. This basically means you would maintain your own composer.json file that declares all of the modules you want and their versions, along with the farmOS version, and then use composer commands to automatically download them and put them into place. Composer ensures that all package versions are compatible with one another for you, so it would catch this kind of thing long before you tried to run drush updb / update.php.

Transitioning to this kind of setup isn’t too difficult, but it does require learning some things. Here is our official documentation, which I would recommend when you have time: Building farmOS with Composer | farmOS

But, you don’t need to do that right now if you don’t want to. You can update your modules manually by downloading the new versions as tarballs and unpacking them in place of the old versions (make sure you completely delete the old module directories first so there are no remnants). Once all the new v4-compatible versions are in place, you should be able to proceed with the drush updb / update.php procedure.

Also worth mentioning: farmOS v4 uses Drupal 11, which requires SQLite 3.45 or higher (see farmOS Server Requirements | farmOS).

See also our official blog post describing the major (and breaking) changes in farmOS v4: farmOS v4 (and v3.5) | farmOS

Hope that helps!

1 Like

Update: @paul121 just released a v4-compatible version of the Project Plan module, so you should be good to go @tlcarpenter69!

https://www.drupal.org/project/farm_project_plan/releases/2.0.0-beta1

Thanks for the guidance. As you may recall I somehow got my installation “non-standard” somewhere along the way. I thought I had fixed all the “external” volume issues where modules hung around persistently. I want to get back to a standard install and eventually move my database to postgres. But today I just want to get the 4.0 codebase running as a baseline.

Here’s my current docker config (ignore the formatting, it didn’t carry over in the paste):

services:
  farmos:
    container_name: farmos
    image: farmos/farmos:4.0.0
    # image: farmos/farmos:3.5.1

    volumes:
      - ./sites:/opt/drupal/web/sites
      #- ./modules:/opt/drupal/web/modules

    ports:
      481:80
    restart: always
networks: {}

You can see I commented out the older modules directory which caused problems in the past. That made me do a lot of fixing before the upgrade to get everything working in 3.5.1.

Let me try to follow your prescriptions. I have two folders where I’ve found modules, but not sure which is which. Both are in sites/all/modules. There is a farm and a farmos directory there, probably left from my cleanup efforts in 3.5.1. I will try to see which is which, unless you see and answer before I get that far.

1 Like

Great! Yea I bet the modules in sites/all/modules just need to be updated. Take a backup of the whole sites volume along with the database before you attempt it, so it’s easy to get back if you need to.

Let us know how it goes!

My first attempt was to upgrade them using composer update command as root in the container, but it just told me there was nothing to install, udpate, or remove. Is there no command to upgrade the modules automatically, or do I have to actually download the modules manually?

Ok, I think I found it. I need to use the composer require commands. I was able to do the farmos_asset_termination that way, now I’ll do the rest.

1 Like

Wait… you are not set up correctly to use composer require!

You are not maintaining your composer.json file outside of the container, so these will disappear next time you update.

I’ll write up more details soon, but in the meantime see: Building farmOS with Composer | farmOS

… or manually download and replace the ones in your sites/all/modules directory. Those are persisted outside of the container.

Actually, I think everything I would say is already explained in Building farmOS with Composer | farmOS - so just read that from beginning to end first. :slight_smile:

1 Like

I was hoping this process was still downloading the source into the persisted module directory at sites/all/modules/farmos.

Also, I haven’t been able to find the farm_bee module, and I got this error after Ledger:
”The installed version of the Ledger module is too old to update. Update to a version prior to 3.x first (missing updates: farm_ledger_post_update_farm_log_workflow).”

Composer downloads modules into /opt/drupal/web/modules inside the container.

It has no knowledge of sites/all/modules, and these WILL TAKE PRECEDENCE over /opt/drupal/web/modules, so you should remove those before you try to use Composer.

But see my next point first…

You need to manually update Ledger to it’s latest 3.x-compatible release before you update to 4.x.

Do this before you do anything else.

Here is the release you need: https://www.drupal.org/project/farm_ledger/releases/2.2.1

Ok. I need to go back to 3.5.1 and do ledger, then come back. Got it. and let me look over that link and see if I can do that while I’m doing all this.

First things first…I went back to farmOS 3.5.1, got the farmos_ledger 2.x code via git and ran it through the upgrade process.

Now I will do the same for the source for the other modules. I would like to get to the composer route, but I also would like to get my database switched from sqlite to postgres.

1 Like

All modules updated via git. the database update started for 42 updates.

However, I did get one failure, I think.

farm_role_account_admin theme

Update move_module

Failed: Drupal\Core\Entity\EntityStorageException: SQLSTATE[HY000]: General error: 1 no such table: oauth2_token: SELECT "base_table"."id" AS "id", "base_table"."id" AS "base_table_id" FROM "oauth2_token" "base_table" INNER JOIN "oauth2_token" "oauth2_token" ON "oauth2_token"."id" = "base_table"."id" WHERE ("oauth2_token"."auth_user_id" = :db_condition_placeholder_0) AND ("oauth2_token"."bundle" != :db_condition_placeholder_1); Array ( [:db_condition_placeholder_0] => 2 [:db_condition_placeholder_1] => refresh_token ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 815 of /opt/drupal/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

It looks like it may be related to keycloak module I had at one time? Maybe something modified in the user database for it? Just a guess.

I ran the update again via the web interface and it applied another 19 updates. However, I think that error above made it so that my admin user isn’t able to login.

EDIT:

The first problem is that I must have changed my password. When I used the correct password, I get a long error message about an unexpected error…again, related to the oauth stuff. Maybe there’s some modules I need to put back:

Drupal\Core\Entity\EntityStorageException: SQLSTATE[HY000]: General error: 1 no such table: oauth2_token: SELECT "base_table"."id" AS "id", "base_table"."id" AS "base_table_id" FROM "oauth2_token" "base_table" INNER JOIN "oauth2_token" "oauth2_token" ON "oauth2_token"."id" = "base_table"."id" WHERE ("oauth2_token"."auth_user_id" = :db_condition_placeholder_0) AND ("oauth2_token"."bundle" != :db_condition_placeholder_1); Array ( [:db_condition_placeholder_0] => 1 [:db_condition_placeholder_1] => refresh_token ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 815 of core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).
Drupal\sqlite\Driver\Database\sqlite\Statement->getStatement('SELECT "base_table"."id" AS "id", "base_table"."id" AS "base_table_id"
FROM
"oauth2_token" "base_table"
INNER JOIN "oauth2_token" "oauth2_token" ON "oauth2_token"."id" = "base_table"."id"
WHERE ("oauth2_token"."auth_user_id" = :db_condition_placeholder_0) AND ("oauth2_token"."bundle" != :db_condition_placeholder_1)', Array) (Line: 130)
Drupal\Core\Database\StatementPrefetchIterator->execute(Array, Array) (Line: 94)
Drupal\sqlite\Driver\Database\sqlite\Statement->execute(Array, Array) (Line: 666)
Drupal\Core\Database\Connection->query('SELECT "base_table"."id" AS "id", "base_table"."id" AS "base_table_id"
FROM
{oauth2_token} "base_table"
INNER JOIN {oauth2_token} "oauth2_token" ON [oauth2_token].[id] = [base_table].[id]
WHERE ("oauth2_token"."auth_user_id" = :db_condition_placeholder_0) AND ("oauth2_token"."bundle" != :db_condition_placeholder_1)', Array, Array) (Line: 521)
Drupal\Core\Database\Query\Select->execute() (Line: 288)
Drupal\Core\Entity\Query\Sql\Query->result() (Line: 85)
Drupal\Core\Entity\Query\Sql\Query->execute() (Line: 93)
Drupal\simple_oauth\ExpiredCollector->collectForAccount(Object) (Line: 55)
Drupal\simple_oauth\TokenExpiryTriggerHandler->handleUserUpdate(Object) (Line: 31)
simple_oauth_user_update(Object)
call_user_func_array('simple_oauth_user_update', Array) (Line: 389)
Drupal\Core\Extension\ModuleHandler->{closure:Drupal\Core\Extension\ModuleHandler::invokeAll():388}('simple_oauth_user_update', 'simple_oauth') (Line: 340)
Drupal\Core\Extension\ModuleHandler->invokeAllWith('user_update', Object) (Line: 388)
Drupal\Core\Extension\ModuleHandler->invokeAll('user_update', Array) (Line: 219)
Drupal\Core\Entity\EntityStorageBase->invokeHook('update', Object) (Line: 1038)
Drupal\Core\Entity\ContentEntityStorageBase->invokeHook('update', Object) (Line: 603)
Drupal\Core\Entity\EntityStorageBase->doPostSave(Object, 1) (Line: 918)
Drupal\Core\Entity\ContentEntityStorageBase->doPostSave(Object, 1) (Line: 528)
Drupal\Core\Entity\EntityStorageBase->save(Object) (Line: 804)
Drupal\Core\Entity\Sql\SqlContentEntityStorage->save(Object) (Line: 370)
Drupal\Core\Entity\EntityBase->save() (Line: 59)
Drupal\user\UserAuth->authenticate('admin', Object) (Line: 18)
Drupal\farm_login\UserAuth->authenticate('admin', 'xxxxxxx') (Line: 241)
Drupal\user\Form\UserLoginForm->validateAuthentication(Array, Object) (Line: 87)
Drupal\Core\Form\FormValidator->executeValidateHandlers(Array, Object) (Line: 280)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object, 'user_login_form') (Line: 123)
Drupal\Core\Form\FormValidator->validateForm('user_login_form', Array, Object) (Line: 611)
Drupal\Core\Form\FormBuilder->processForm('user_login_form', Array, Object) (Line: 347)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 73)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->{closure:Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber::wrapControllerExecutionInRenderContext():121}() (Line: 634)
Drupal\Core\Render\Renderer::{closure:Drupal\Core\Render\Renderer::executeInRenderContext():634}()
Fiber->resume() (Line: 649)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->{closure:Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber::onController():96}() (Line: 183)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 54)
Drupal\simple_oauth\HttpMiddleware\BasicAuthSwap->handle(Object, 1, 1) (Line: 53)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 28)
Drupal\Core\StackMiddleware\ContentLength->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 53)
Drupal\Core\StackMiddleware\AjaxPageState->handle(Object, 1, 1) (Line: 54)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 745)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

Wow. That took some digging to find the answer to. Here’s how I had to fix it:

   composer require drupal/devel_entity_updates
   drush en devel_entity_updates -y
   drush entup -y

I know that’s not saving anything permanently, but I only needed it to modify my database. By doing those steps above, the second drush command recognized all the database entries that were missing and then created them.

Then I had to make sure I deleted the modules before shutting down and testing:

drush pm:uninstall devel_entity_updates -y
composer remove drupal/devel_entity_updates

drush pm:uninstall devel -y
composer remove drupal/devel

drush cr

I’m sure those can be combined. If you don’t do the uninstall before shutting down, when you boot next the system will complain that those are installed but the code is missing.

I did those steps and now I’m able to log in as admin. I’ve looked at some of my data (assets/logs/etc) and I’ve not seen any errors. But I also can’t seem to find a GUI page that confirms what version of farmOS I’m running, but I do know from the docker config it’s 4.0.0.

EDIT: I found it under Administration → Reports → Status Report. There’s an entry for Installation Profile (it says farmOS (farm-4.0.0)).

Thanks for the pointers and the help. My first goal of getting to the 4.0 code base is now achieved.

Next I will look at the composer direction, and then ask for help when I’m ready to migrate my database from sqlite to postgres.

Huh. I’m not sure what that error was… but if you had the Keycloak module and did not properly uninstall it then it could certainly cause issues. Hard to say if everything’s OK now, but maybe it’s fine. If it were me I would roll back and try to figure that out. You shouldn’t need to use devel_entity_updates for anything. (Although there was a minor issue with a recent Consumers / OAuth2 module release that caused a “Mismatched entity” warning in /admin/status/reports, but I think they fixed that upstream recently, and it was harmless in the meantime I think).

Happy to help with the Composer setup.

I don’t have any experience migrating a database from SQLite to PostgreSQL, though. I tried helping someone migrate between MySQL and PostgreSQL and it was a bit of a rabbit hole - they ended up giving up I think.

I may not even start based on what you’ve said. If there’s no real reason to migrate the database, then forgetaboutit.

1 Like

PostgreSQL is going to be more performant, so if your site is going very slow then it might be worth figuring out. But otherwise its fine to stay on SQLite.