Early Draft Post: Thinking about Organizing Agricultural Data with farmOS

I started sketching out another post for the blog that I was hoping to get some early feedback on. The later parts are just an outline and I’m not totally sure where I’m going with it. All feedback welcome. Please feel free to tear it apart :slight_smile:

Thinking about Organizing Agricultural Data with farmOS

Any kind of record keeping should serve a purpose. Record data itself is useless unless it can provide some insight, shape future decisions, or satisfy external requirements.

In agriculture some example drivers of record keeping are tracking the pedigree/lineage of animals/plants to help manage genetic diversity or breeding goals; tracking yields inputs and crop rotations to detect patterns in what practices are working; and gathering data required by regulatory bodies for permit or certification purposes.

Unfortunately, one can’t always predict what data will be needed for these things. Thus, a common strategy is to keep some sort of records even when the exact purposes to which they’ll be put aren’t yet known.

There are a lot of options when it comes to what tools that can be used for agricultural record keeping. Among these options is farmOS. farmOS lands in a sweet spot on the gradient between on one hand free-form tools like paper notebooks or spreadsheets and on the other hand purpose-built tools that target specific market/livestock/regulatory niches.

farmOS lands in that sweet spot by providing a structure of record primatives while remaining largely unopinionated about how data is organized within that structure. On top of that farmOS is highly open and extensible which allows for more opinionated consistency rules to be added on a use-case by use-case basis. It is also possible to use farmOS’ API to programmatically restructure large amounts of data as one’s recording keeping requirements evolve.

Where to Start?

It is clearly critical to determine what granularity of data to capture. Even if one could easily track the movements and growth of every macroscopic organism in an area with technology, the data wouldn’t be very useful by itself. Yet more technology could probably be used to infer some types of meaning from all that data, but the complexity of such a system would be enormous and very costly from a sensing-hardware, storage, and computational standpoint.

Fortunately, by working backwards from some hypothetical goals for the data, we can arrive at a much more reasonable strategy. We don’t need to track every ant and native shrub to have useful records for a market garden or a goat farm. The market garden might want to keep track of what is planted where, when it was planted, and harvest dates/yields. The goat farm might want to keep track of age, sex, breeding events, and so on about the goats.

It is easy to get overwhelmed imagining all the things that one might want to keep track of, however with farmOS it is easy to start small and - with a few reasonable assumptions - the patterns will scale to capturing more and more types of data as it is warranted.

Often it is helpful to look at one’s goals for the agricultural operation as a whole. If the goal is to raise animals, then perhaps tracking the individual animals or herds of animals is the core primitive to start with. Similarly if one’s goal is to grow vegetables perhaps the core primitive is plants or land areas (fields, or beds) of plants. In farmOS the first step would be to create assets for these core primitives.

Part of that process is determining how to connect between the physical reality and the records on the computer. Chances are there is already some sort of naming strategy and this tends to form the first layer of connection.

Asset Type and Granularity

  • When to use an existing asset type vs a custom one
  • Pros and cons of fine/coarse granularity assets. i.e. animals vs herds

Naming Assets

  • Ideas for designing naming patterns or adapting existing physical naming/tagging systems to farmOS.

Log Type, Purpose, and Frequency

  • When to use an existing log type vs a custom one
  • Tradeoffs with fine/coarse granularity logs. i.e. play-by-play activities/observations vs the minimum set of logs to capture only key changes

Naming Logs

  • Ideas for designing naming patterns or adapting existing task tracking/naming to log names

Next Steps

Allude to future posts about capturing data via quantities, inventory, sensors, and more

3 Likes

This is great @Symbioquine! I really like this description:

farmOS lands in that sweet spot by providing a structure of record primatives while remaining largely unopinionated about how data is organized within that structure. On top of that farmOS is highly open and extensible which allows for more opinionated consistency rules to be added on a use-case by use-case basis.

I also really like the structure you are sketching up for the asset type, log type, and naming stuff… these are some really good baseline things to explain and understand when you’re getting started, which we don’t really do a good job of covering elsewhere. This will be a very helpful blog post, I think.

Maybe worth mentioning strategies for taking quick notes in the field and then coming back to fill them in/add details afterwards? Or maybe that’s another post… :wink:

3 Likes

I think this is on target. FarmOS provides a very open and flexible canvas for record keeping. As a new farmer, I did not have experience and a baseline for what information I needed to track. I made a few assumptions and dove in. I found myself going down a given path for structuring the data and then would run into problems later with some of my assignments and choices. I had to go back and restructure some of the data entries. Bit by bit I got to something that was working. As I move to Version 2, I will need to do further restructuring to get my data assignments cleaner and more consistent.

For new users, there needs to be a set of Use Case template level examples based on different farm operations, like “Organic broad acre row crops” or such. These would help get users off to a good start. These templates would also aid future development by helping to keep end use cases in focus. Did the new proposal make a given use case easier to manage or harder? Right now my goal heading into Ver 2 is to define the template for my operation answering the questions in your outline.

3 Likes

@graffte thanks for the kind words about my draft and the ideas! I know we’ve talked about having some curated example data in the past, but I like your idea of having a few separate ones for different kinds of farms/operations.

Just to tie this together, I’ve previously demonstrated how sample data can be captured in a farmOS module and installed. I guess the next step would be to make a number of stand-alone modules installable via composer/drush which represent the curated examples… @mstenta Do you think it might make sense for those to live in core (assuming they demonstrate/exercise core functionality)? That would be cool because then they could be leveraged as part of testing and demo instances more trivially too.

Bringing it back to this post, I think I’ll keep the scope smaller and just address the general ideas around strategies for naming and asset/log granularity. We can always have future posts which introduce a given set of sample data and walks through the rational behind how the data is organized therein.

3 Likes

I’d probably start it in a separate module repo (eg: github.com/farmOS/farm_demo), perhaps with sub-modules for different use-cases. This would keep it isolated and allow additional dependencies to be pulled in if necessary (eg: Default Content | Drupal.org), without the considerations of maintaining that in core. If it proves to be well-maintained we can consider pulling it into core in the future.

1 Like