Example farmOS.py CSV import script

On the last monthly call there was some interest in importing data from CSV/spreadsheets into farmOS - ping @walt @Kamitor @varunity ?

I hope this example can help serve as a starting point for other custom CSV importers using Python + farmOS.py. Code + installation and usage instructions are here: farmOS.py 2.x Import example · GitHub

This example reads a name, crop, variety and date from a CSV and creates a Plant asset with a seeding log on the specified date. The Plant asset’s plant_type is either loaded from an existing term, or a new plant_type will be created with a reference to the crop_family term (which is also created if one does not already exist). I think this incorporates both simple and more complicated concepts that will be run into when creating custom importers with Python:

  • Python
    • Command line arguments
    • Logging
    • CSV file loading + parsing
    • Date parsing
  • farmOS.py
    • Client instantiation + authorization
    • Create an asset with a plant_type/taxonomy term relationship
    • Create a log with asset relationship
    • Request existing entity with query filters (example is of taxonomy terms filtered by name)

While farmOS does have CSV importers, they require a strict format and are limited in what fields are supported. Even a fairly simple script allows for many more options: flexibility with the CSV format, the ability to create additional fields, multiple entities + entity relationships (assets + logs referencing assets), conditional logic, etc… but they also require customization & some programming knowledge!

Happy to answer any questions & would love to hear ideas for improvements!

2 Likes

Hey @paul121 : Excellent initiative! Being a near-total python n00b, i have my technical issues to sort out (1st related question posted to project repo), but at the higher business-case level, this sounds like a solid first step toward solving my big challenge of migrating my farm operational data from spreadsheets into farmOS.

So: my 1st hi-level question is: Could this script be modified and maintained with reasonable ease to work with v1.x API as well as v2?

I ask because, in light upcoming v2 release of farmOS, and these two related advisories @mstenta gave on last night’s Dev call- (1) to consider all data put into v2 Alfa instances as throwaway (it WILL be overwritten with each successive release); and (2) to not waste development effort against the v1.x API- i find myself in a bit of a quandry about strategy.

To explain: i am investing resources right now in this data migration challenge- by that i mean not farmOS v1.x to v2 migration, but rather from my existing spreadsheet-based model to farmOS- and for this project to be anything other than a bust, it needs to yield business results (i.e. farmOS as a useful tool in production, not just a R&D /playtime project). So: we need to be turning our existing spreadsheets into a CSV format that v1.x can understand now… But then the process of posting and updating new records is something i would really prefer to accomplish via API script.

SO: If you were in my position- i.e. a n00b w/o any in-house programming resources -i wonder how you might resolve this quandry? I guess the answer depends largely on how long it will be until farmOS comes out of alpha testing and into stable beta release (i know it’s bad form to ask this of FOSS developers, but even a “90% certain it will be before xyz date” would really help me in scoping this issue!).

2 Likes

Good question @walt. I think this script could be modified to work with 1.x pretty easily.

The 1.x client methods and structure of the data payloads and responses would look a little different, but the script would be structured the same. It makes sense that you will need this for your situation! I’ll try and draft up a 1.x version over the weekend.

Once you turn existing spreadsheets into a CSV format for 1.x, I think the same CSV should work with a 2.x server. This is because the farmOS data model is largely the same, it’s just the API changes that are different. So the structure of your script will largely stay the same as well, but farmOS.py client methods will change & return data in a slightly different format.

The mechanics of the 2.x API should be pretty stable, it’s just the farmOS data schemas that may change some before a beta release. So you should be fine to start developing scripts that work with a 2.x instance.

But yes, since you are hosted on Farmier and need to interact with your “canonical” database right now… my advice would be to start working on converting existing spreadsheets to a format that’s optimized for importing to farmOS. Then you could build a script for 1.x, and later on converting it to 2.x should be fairly easy. (Assuming this is not a one time thing and you’ll want to import/sync data in a similar format later on!)

2 Likes

If you can do that, @paul121 , that will be awesome!
Meanwhile, i will transform our spreadsheets into CSVs that farmOS 1.x can readily accept.
/w

1 Like

@walt I added an example import_plantings_1x.py file to the gist.

This led to me discovering a bug in the farmOS.py 1.0 beta that did not have the correct header for 1.x servers which prevented creation of records. That is fixed in a new 1.0.0b3 release - so you will need to install the latest version to create records on your 1.x server!

1 Like