Looking for farmOS 2.x Migration Testers!

Development
1

Hi everyone and happy holidays! As many of you know, we’ve been hard at work on farmOS 2.x - the next major version - which brings a lot of improvements to the codebase, data model, API, and user interface. We’re very excited, and are aiming to tag a “beta” release this coming Spring, if all goes well. :slight_smile:

When it’s ready for release, the update process will involve “migrating” data from a version 1 database to a version 2 database. This will be an automated process. If you are hosting on Farmier it will happen automatically. If you are self-hosting then there will be instructions for installing 2.x, enabling the appropriate modules, connecting to your old database, and running the migration. The beauty of this is that your old 1.x database will not be touched in the process - it will only be read and copied from.

As with any large migration, we want to be sure that data goes from point A to B properly. This requires thorough testing. We are doing a lot of this testing ourselves, as we write the migration scripts, and we would like to enlist some interested parties to test with their data as well. The migration scripts are not terribly complicated, but we want to be sure there aren’t edge-cases we might miss.

We’re looking for folks who can spend some time going over their data with a fine-toothed comb - comparing their data in farmOS 1.x to 2.x side-by-side.

If you are interested in testing the 2.x migration on your own data, and you are hosting on Farmier, send me a private message and we can coordinate. If you are self-hosting, subscribe to this thread and I will post a link to instructions when they are ready.

Thanks!

5 Likes
2

Since I’ve just started to scratch the surface of what farmOS can do, I would like to test this migration.

My first reason for doing this is due to the fact that I am planning on starting a mushroom farm with a friend in the next couple of months and I noticed the Fungi module requires Drupal 9. I think this would be a great opportunity for me to learn more about the development process and see what is available in my target module. Are there any items/procedures/logs/inventory, etc that I can help with?

Who knows until I try?

:grin: :metal:

2 Likes
3

@shane_aldrich FWIW the farm_fungi module is actually a rename of the farm_mushroom module for farmOS 1.x - so you can start using that now! GitHub - farmOS/farm_mushroom: Provides features for managing mushroom production.

(Also, the farm_fungi module is not done for 2.x yet… see status of 2.x add-on modules here: https://www.drupal.org/project/farm/issues/3151247)

As for 2.x migration testing - if you have a 1.x database with data in it, I’d love to hear how the migration goes for you! There are instructions for setting up a local 2.x development environment here: Getting started - farmOS 2.x Docs - and the migration instructions are here: 1.x Migration - farmOS 2.x Docs

Or, if you’re just getting started and don’t have any data to test with, then it probably makes sense to just start using farmOS 1.x, and get ready to test/migrate a little later when we tag a beta release.

1 Like
4

@mstenta Thanks for this information!

Since I don’t have any real data yet, I’ll wait for the beta release and see how the migration goes from there.

Cheers! :beers:

1 Like
5

@mstenta
With alpha 2 now out and with over three years of data in our farmOS installation we are more than willing to help with migration tests.
I am a software developer (PHP, JavaScript and AWS) but unfortunately with zero knowledge of Drupal, so clear instructions are required for that. We are running the locally installed farmOS instance as a VM in a VMware vCenter-cluster. We use the capacity for hosting our CI/CO testing and other internal servers and have plenty of opportunities to spin up new servers and make snapshots.

Are the instructions in your post from Mar 17 up to date? (I guess I can adopt them to a VM and mysql database that we use instead?)
How to want the feedback?

2 Likes
6

Hi @mattias - great! And yes the links above are still up-to-date. The local environment setup instructions assume you will use Docker - but if you are comfortable setting up a Drupal/farmOS installation outside of Docker that will work too.

The migration instructions work with or without Docker. You need to use the Drush migrate commands to perform the migrations via the command line. This is included in vendor/bin/drush in the packaged release of farmOS.

farmOS 2.x works on MySQL/MariaDB/PostgreSQL/SQLite3 - so if you have MySQL already that should work! We’ve been using PostrgeSQL as the default database for all development, so you could also go that route if you want. But either should work.

As for feedback, anything and everything! We created this issue if you want to post comments/observations there: farmOS 2.x Alpha Feedback

The most important things to test right now is the actual migration mapping. We basically have logic that reads records from the old database and recreates them in the new database by mapping point A to point B (for each record and each field on each record). So we just want to make sure that all of those mappings are working correctly, and that no data is being lost/broken in the process. We’ve been doing lots of testing locally, but only with the datasets we have - so that’s why it’s important for others to test with their data and confirm that everything looks good post-migration.

Appreciate any amount of testing you can do!

7

We are on mysql so that will be what we’ll test on. I will be out sailing in the Baltic with limited Internet access for the coming week so no much will be happening until I get back home. With the amount of information we have stored in our installation we really want to be able to carry it forward to new releases, so help testing the migration part is at least 51% selfish :slight_smile:
Will be in touch!

1 Like
8

In context of this data migration test, i’ve just begun process of comparing .CSV file outputs of the same report (or: what should be same report) pulled out of both my (farmier-hosted) instances -current version 1.x and version 2 alpha- and i am getting some surprising results.

For example: Going to Assets/Plantings (only w/o the “…ing” in v2), w/o applying any filter but number or records (250, since i have less than that recorded), here’s the URL for reports from version 1.x and 2, respectively:

  1. https://vdl.farmos.net/farm/assets/plantings/list?name=&field_farm_move_to_tid=All&field_farm_group_target_id=All&archived_boolean=0&items_per_page=250
  2. https://vdl.farmos.dev/assets/plant?name=&is_location=All&is_fixed=All&status=active&items_per_page=250

… And here is the shape of .CSV files returned:

  1. farm_plantings-field_farm_move_to_tid_All-field_farm_group_target_id_All-2021-07-21T09-17-26.csv
Number of Rows, Columns =  (148, 11)
Column Headers:  Index(['Farm asset', 'Asset ID', 'Unnamed: 2', 'Name', 'Crop/variety',
       'Season', 'Description', 'Flags', 'Group', 'Location', 'Archived'],
      dtype='object')
Datatype of index =  <class 'int'>
  1. assets.csv
Number of Rows, Columns =  (197, 10)
Column Headers:  Index(['Bulk update', 'image_target_id', 'ID', 'Asset name', 'Asset type',
       'Flags', 'Parents', 'Group', 'Location', 'Status'],
      dtype='object')
Datatype of index =  <class 'int'>

Regarding the 3 noteworthy diffs here, i would say:

  1. Composition of filename: Version 1.x assigns a very descriptive filename, concatenating several attributes of the table/ query, while the v2 URL calls it simply “assets.csv” -which name might apply to other categories of assets besides those of type plant(ing). Why go from a more descriptive name to one that’s less so, i wonder?
  2. Column count and naming: I know there are good reasons for the schema change, so i’m not going to quibble about such details at this point, except insofar as it affects…
  3. Row count: the fact of v2 report having an extra 49 records on what was supposed to be an unfiltered report of plant(ing) assets is surprising enough… But what is moreover disconcerting is that the v2 report showed me a count of 145 total records, right next to the “Export CSV” button i pushed to generate this report of 197 records! Looks like something in those 2 db queries -the one presenting data in UI, and the one generating CSV output- must be different.

SO, in the light of such discrepancies on my first systematic look at data migration results, i have to ask @mstenta : does it make any sense for me to continue along this line of testing? is my method somehow flawed? or is it just too early, and i’d best just wait until certain development issues get sorted out first?

I should also add: with benefit of some patient support from @paul121, i’ve started to get some interesting data from the v2 API via farmOS.py library… But again: each of my first two experiments with the ,GET method revealed a bug (one in documentation of farmOS.py, and one in farmOS having an http link where it needed to be https)… So again, i have to ask: is this the best way to discover such bugs? Are there no automated tests to catch these things?

2 Likes
9

Thanks for taking the time to review and summarize your findings @walt ! This is exactly the type of feedback we’re looking for. :slight_smile:

Composition of filename
Why go from a more descriptive name to one that’s less so, i wonder?

Good observation - we can look into making the filenames more descriptive. This was just missed during the upgrade process.

It’s worth understanding that the development process for 2.x is essentially a process of rebuilding the 1.x features from scratch. This means that every decision that was made in 1.x needs to be replicated in 2.x. So there will definitely be little things like this that we missed/forget/overlooked. We’re relying on you and others in the community to point out the ones that matter to you. The side-result of this process is we also end up dropping things that didn’t actually matter - thereby simplifying things and maintaining less, in theory - so it’s a refinement process as well.

Column count and naming

There will definitely be differences with the columns. We’re taking a different approach with these listing pages in 2.x - building them somewhat “automatically” based on the fields available on each asset/log type. This means that there may be new columns that weren’t visible in 1.x… but it also may mean that there are columns missing in 2.x. Again, we’re taking the approach of “first pass then see what people notice” - so all feedback is welcome.

Row count

Ah ha! Good eye! This is definitely a bug. It appears that the filters are not working properly in the CSV export. If you look at the 2.x CSV, you’ll notice that it includes ALL asset types - not just Plants. I’ll file a bug report and we can look into this.

does it make any sense for me to continue along this line of testing? is my method somehow flawed?

Your method is perfect. Please do continue if you are able/interested. :slight_smile:

GET method revealed a bug (one in documentation of farmOS.py, and one in farmOS having an http link where it needed to be https)

The http bug is a known issue in my 2.x alpha testing server. It’s on my list to fix before migrating/hosting any real production instances. Let me know if this is preventing you from doing the testing you need, and I’ll try to bump it up in priority.

is this the best way to discover such bugs? Are there no automated tests to catch these things?

farmOS 2.x has automated test coverage for a LOT of things - but the bugs you found above are outside of its scope - and that will be true of MANY classes of bugs. So yes! This is the best (and only) way to discover such bugs. Thanks again for taking the time!

1 Like
10

the bugs you found above are outside of its scope

Clarification: I mean the bugs with farmOS.py documentation and Farmier alpha hosting server configuration. :slight_smile:

The bug you discovered with the CSV filters is within the scope of farmOS, and should have an automated test. We can add a test for that when we fix the bug, to prevent regressions.

1 Like
11

@walt here is the bug report for the CSV filtering, if you want to follow along: https://www.drupal.org/project/farm/issues/3224663

1 Like
12

[quote=“mstenta, post:9, topic:575”]

Good observation - we can look into making the filenames more descriptive. This was just missed during the upgrade process.

Good to hear; i always find (but especially when doing this sort of amateur “data science”) the more descriptive the filename, the easier it is to distinguish between files that might otherwise be mistaken for the same thing.

…We’re relying on you and others in the community to point out the ones that matter to you. The side-result of this process is we also end up dropping things that didn’t actually matter - thereby simplifying things and maintaining less, in theory - so it’s a refinement process as well.

Oh! <gulp!> So that means we’ll be losing features that just happened to slip thru the cracks of our test process?!

Column count and naming

…there may be new columns that weren’t visible in 1.x… but it also may mean that there are columns missing in 2.x. Again, we’re taking the approach of “first pass then see what people notice” - so all feedback is welcome.

2x<gulp!>. OK, then: at risk of erring on side of verbosity, i will say now that i find the absence of certain features i’m used to seeing in your typical industrial-strength db somewhat troublesome -namely date & authorship of records creation & modification. These features are needed, not only to confirm transactional integrity (esp where multi-user edit conflict is possible, but also in case of machine/ power failures) but also for other reasons. For example: right now my agronomist colleague may be creating records in one instance or the other, that put the two instances out of sync, so i need to be able to filter those out by date and/or authorship. In fact: this is so fundamental, i suspect those columns may exist at system level, but are hidden from the view i’m interacting with. Do i guess right?

Row count

Ah ha! Good eye! This is definitely a bug. It appears that the filters are not working properly in the CSV export. If you look at the 2.x CSV, you’ll notice that it includes ALL asset types - not just Plants. I’ll file a bug report and we can look into this.

Good to hear -but, do tell: is this one likely to affect all CSV outputs, until it is fixed?

does it make any sense for me to continue along this line of testing? is my method somehow flawed?

Your method is perfect. Please do continue if you are able/interested. :slight_smile:

Not only interested, @mstenta : i need this software to work -and soon!- or else i need to find another solution for farm record keeping. So: to whatever extent my responsibilities in the field will allow me time to do this extra work, i am keeping this atop my queue of back-office priorities.

1 Like
13

So that means we’ll be losing features that just happened to slip thru the cracks of our test process?!

Potentially yes - but hopefully only small ones. And to be clear: if you raise them, we’ll add them to the todo list!

Like I said, the process of upgrading farmOS to 2.x is a rebuild process (due to the nature of the upgrade from Drupal 7 to Drupal 9). Therefore: every additional todo requires more time and resources to do it, which are inherently limited. So we are trying to be as efficient as possible - focusing on the big features we know we need, but also allowing “small” things to fall to the wayside unless/until we hear demand for them. So this is a bit of a “culling” process too, which is good for the health of the overall project.

That said, don’t worry too much about it. If you see something missing, it doesn’t mean we won’t add it! Just bring it to our attention. Development is not going to stop once we release the first beta. :slight_smile:

date & authorship of records creation & modification
i suspect those columns may exist at system level, but are hidden from the view i’m interacting with. Do i guess right?

You guess right. :slight_smile:

In 1.x, the author and creation/last modified dates of assets and logs are stored in the database, but were not exposed in the UI.

The same is true in 2.x, but now it also adds the concept of “revisions”, which tracks when/who changed things at an even more granular level.

There is a “Revisions” tab next to “View” and “Edit” on asset and log records - although now that I look at it, it appears that the “Manager” role does not have access to that currently… that’s a simple permissions thing - I’ll make that change so you can see it shortly…

You can also access this metadata information via the API, which exposes ALL raw data that is stored in the database, generally speaking.

is this one likely to affect all CSV outputs, until it is fixed?

All asset and log CSV exports use the same base code/config, so yes.

i need this software to work -and soon!- or else i need to find another solution for farm record keeping

We’re doing the best we can with what we have @walt ! Appreciate your feedback and testing - but also understand if farmOS doesn’t meet your needs at this time.

1 Like
14

There is a “Revisions” tab next to “View” and “Edit” on asset and log records - although now that I look at it, it appears that the “Manager” role does not have access to that currently… that’s a simple permissions thing - I’ll make that change so you can see it shortly…

Less simple than I hoped. I opened an issue for this: https://www.drupal.org/project/farm/issues/3224684

1 Like
15

Don’t get me wrong, Mike: i am amazed at what you guys have done and are doing w/ so little in the way of resources!

That being said: there are practical decisions out here in the field that must be made, based on the reality of what tools we have available to us -and will have, and when. Right now i am focusing resources on this challenge of migrating our farm record keeping system over from spreadsheets to farmOS v2, and i need to know this will serve our operational purposes -including requirements of the agency that audits us for organic certification, b/t/w- so that’s why i use “need” verb, along w/ “soon” qualifier. (no pressure, of course; no worries mate! :slight_smile:

Seriously: i know the risk is all on me, and i won’t be asking you for any ETA on production release… But i do want you to know that i’m planning to use a Farmier-hosted instance of v2 as my operational data store as soon as it feels to me solid enough to trust -with backup, of course[1]- even before you may be ready to issue any guarantees.

[1] re Backup: Dunno if this is an appropriate forum to discuss the issue, but: i wonder what is the backup procedure on these Farmier instances, both production and test? And i wonder (since the risk during pre-production is all mine) if i might set up a script to pull records off on say a nightly basis via API?

2 Likes
16

100% understand @walt - and I hope the farmOS development can converge with your needs! All feedback you provide is very valuable - even if we can’t always implement requests immediately. So keep it coming and don’t worry about verbosity! :slight_smile: (or, if you are worried, make a dedicated forum topic and we can go as long as we need!)

re Backup: Dunno if this is an appropriate forum to discuss the issue, but: i wonder what is the backup procedure on these Farmier instances, both production and test?

Production instances (farmOS 1.x) are backed up automatically on my end (weekly snapshots stored in multiple physical locations). I also plan to provide the ability to download backups in the future! That will be easier to implement on my end after 1.x is behind us though.

The 2.x testing instances I set up for you and a few others are NOT backed up at all. I consider your 1.x instance to be the “canonical” database up until we do a “final” migration to 2.x. At which point it will be your “production” instance moving forward, with automated backups. And so from time to time I may “re-migrate” data from 1.x->2.x, as new features are added, bugs are fixed, etc - purely for testing purposes. This will wipe out everything on your current 2.x instance and replace it with data migrated from 1.x.

So to be clear: I DO NOT recommend that you use your Farmier-hosted 2.x instance for real data. Ideally you should wait until the “final” migration.

That said, if Farmier’s timeline isn’t sufficient, you can always host yourself! But of course that will bring its own headaches. Unfortunately this is just the “in-between” time we are in right now… waiting for 2.x to be ready. Can’t promise much at the moment, I’m afraid, but happy to provide advice/recommendations along the way!

17

Less simple than I hoped. I opened an issue for this: https://www.drupal.org/project/farm/issues/3224684

Update: I worked out the permission changes needed to grant access to the Revisions tab on assets/logs. This will be included in the alpha3 release. So @walt when alpha3 is released I will update your (and others’) dev instance so you’ll be able to try it out. I’m pushing to review/merge a few other things for alpha3 as well, including bulk actions, KML importers, and a few other things.

1 Like
18

It appears the farmos.dev server is down as of now -has been for some while already today- and so i’m to ask at this time: is this owning to an unanticipated failure, or is it a planned maintenance or upgrade event? In case of the latter case: would it be possible to get some warning in advance?

(i ask only because my work on this test instance involves co-ordinating with a colleague in a different time zone, and we’d like to plan our activities around any maintenance window that might be upcoming)

1 Like
19

Thanks for the heads up @walt. Should be back up and running now. This was unplanned. I’m investigating the cause. Please let me know via DM or email if you notice this again in the future.

If there is planned downtime I will let you know.

1 Like