Consolidated Sensor Event

I have a flow meter I recently got connected to my irrigation system connected to FarmOS.

Right now it collects water/15minute and reports a data point to FarmOS every 15 minutes. This is acceptable and satisfies my needs.

However, I could program the sensor to be much “smarter” i.e. when I turn you on record the time I turn the irrigation pump on, then record the litres I put through you, then record the time I turn you off. Ultimately this would lend itself to a reporting event of the nature August 3, 3056 12:00 Pm to 12:15 Pm irrigated XYZ w/ 1000 litres.

Does anybody have a workflow in which I could take a sensor reading and report total reading over and “on” period and record it as an observation as I have described above? explicitly, turning a 15 minute periodic sensor reading into an observation of crop activity?

2 Likes

Very interesting @mbillion !

It does seems like it would make sense to record this as an observation or activity log with 2 quantities:

  • One for total water applied (L)
  • One for total duration (seconds/minutes)

The log timestamp could represent the “start” time, and the log would also give you the ability to reference areas & assets with the irrigation activity.

As far as implementing this… it might be easiest if you could “aggregate” the sensor readings outside of farmOS, such as on the sensor device itself, and then have it POST a log to farmOS. But of course that isn’t *always possible.

To do this all in farmOS I think you would need some kind of data storage & logic to listen for and save sequences of the sensor readings:

  • Save a start/power on timestamp
  • Save water flow value(s)
  • Once the next end/power off timestamp is received, autocreate a log with the previous start timestamp

That should be possible in farmOS 1.x, but it would require a lot of custom code.

This should become easier in farmOS 2.x with a concept of sensor “data streams” where a sensor’s data stream can actually reference the asset it is describing (some thoughts on this are described here but it’s still a work in progress!). I’m hoping that data streams can reduce the boiler-plate code required to implement common features such as POST and GET requests, as well as interfacing with sensor data storage. This should make it easier to provide custom sensor behaviors!

Certainly welcome any additional thoughts on the idea of data streams!

2 Likes

I can aggregate it outside on my device I am running a bunch of ESP8266’s connected via a central hub Raspberry Pi which handles and orchestrates events on the farm.

I am pretty new to farmOS and hadnt considered this would look like pushing a log event into the API instead of pushing sensor data to the sensor module. I will look into it

2 Likes

Just read the thread on data streams and what you are conceptualizing thinking about there.

You mentioned the idea of NDVI and sentinel 2. I was running the data analytics and warehousing shop at a very large organic grower a while back. I was able to do a lot of work with sentinel-2 for those guys as management viewed the cost of private purchasing custom imagery, drones, or some of the airlines take high-res off the bottom of airplanes and compile and sell that info as well. so I will speak to some of my experience there.

Flexibility is key - NDVI may be one users model, however, we were using the same data from sentinel + weather stations to do Degree Days calculations. You may want to have a place where you can associate sensor data to build a model, however, have the ability in general to create a model, but do not pre-load a specific model.

Also - clouds. Think how you are going to handle misreads or no-reads when sentinel-2 (or other device) passes over and does not get a read. Do you average, omit, use a calculus based smoothing or curve fitting routine? these types of things, in my experience, are awesome academic studies but they are rarely deployed in working farms as the practical deployment either needs to highly resilient (in the mathematical sense n-resilliency and failure modes in order to reach consensus), or you need to have a lot of mathematical modelling oomph behind a model to smooth and fit your curves.

Also - I know you mentioned drones - which leads me to believe you are thinking about flexibility, however, for me, an indoor hydroponic grower, I have a small gantry attached to my hydro beds w/ stepper motors attached to it is a camera which takes a picture of each of my crops on a daily basis. You are all quite responsible with flexibility, but I always call it out as some farmers might not be drone or gantry camera, etc sophisticated and might just be using a handheld camera for pictures as “sensor” data.

Similarly, not sure how far you have gotten into the Sentinel-2 API but it is remarkably vast, which, in turn means it can be quite the study in manipulating and extracting the data. Beyond sentinel-2 the ESA has a whole gang of satellites sensing all sorts of things with similar API access to sentinel-2. It can be quite difficult for the tinkerer or scientist-farmer to grapple with something like ESA data, but, once inside and poking around, you want to make sure the system is flexible enough to handle any number of data elements the user may throw at it as there is a whole ton of data that can be extracted from the system - as mentioned for other models - such as degree days.

Beyond that - my needs are mostly represented there - I think its always important to maintain the ability to do a “spot check” on a sensor to do a live look in, but having the ability to represent that as a time-series is important, and as you know I would like to also be able to take a start-time : end-time aggregate of sensor data and turn that into an aggregated observation.

2 Likes

Wow great thoughts @mbillion ! :smiley: That sounds like some interesting work you were doing for that grower.

I should mention that I don’t have much experience working with raw Sentinel or Drone data itself, I was simply being handed the results of what had already been processed (NDVI or NDWI mostly) and needed to incorporate them into the farmOS data model. But to all your points, the complexity of sentinel, drones, clouds, etc… it does seem like something that needs to be computed outside of farmOS.

I have tinkered a little with this library for Sentinel imagery that helps with downloading, masking, “removing” clouds, etc: GitHub - regen-network/regen-s2-ard: Regen Network Sentinel 2 Analysis Ready Data (ARD) Has some rough edges but is a step in the direction of making this sort of stuff easier! Even if a farmer could “simply” press enter to run a script and generate these calculations, that’s still a long ways from being all that practical.

Interesting that you mention growing degree days - that’s one thing I’m hoping to integrate this year as well. Again, I will just be handed the results of the GDD calculations (similarly derived from satellite + weather stations), and will need to save/represent in farmOS to make the data meaningful. Any tips, experience or otherwise you can share? GDD seems to be like it is quite meaningful & useful for decision making on the farm. I imagine a future where this has “first class” support in farmOS, as it’s reasonable to assume an asset’s GDD timeline would be useful in various contexts.

I haven’t given this all that much thought yet, but am planning on integrating GDD with a data stream that references 1 or many Assets. Likely 1 data stream for each weather station location - not that this data is directly coming from the weather station, but it’s just the most granularity we can get. In this way the data stream could reference all of the nearest Plantings/Fields. The end goal being to select a Planting/Field in farmOS, and be able to query/display the GDD values.

But yes back to you one of your points…

and as you know I would like to also be able to take a start-time : end-time aggregate of sensor data and turn that into an aggregated observation.

This is interesting. I’d like to further explore these use cases! What are some outputs you might want from this? Like we mentioned above, creating logs + quantities, but are there other times you just want a subset of the original sensor data?

One idea I have is that a sensor’s data stream could be split off into one or more data streams after the fact. I’m not sure it makes sense to break them up on a time basis, like every year or month, but more in terms of “event” based data streams. This might be too granular for your example of irrigation events, but might be useful for something like a Tractor’s Harvest/Yield mapping, where a sensor’s data stream is tied to one specific event. Since data streams are Drupal entities, it would even be possible to have a Harvest log reference a data stream!

1 Like

The GDD stuff is really cool and I believe it works really well. In part it depends on what it is you’ll be growing as there is a bit of an optimization period to adjust and much of it is based on the instrumentation quality and observation quality.

It took us a couple seasons to really get dialed in on an optimized number of degree days to achieve the perfect spec on product, further, it took us a while to kind of optimize out the measurement plans and get the math refined

Sentinel-2, much like many projects in our lifetimes, is collecting more data than has even begun to yield its ultimate value, the sheer volume of data could lend itself to a career in data science. The uncovered treasures of which are likely numerous.

As far as the time events I would think of anything that is not a point event would benefit from the ability to aggregate. Clearly as my example indicates, irrigation is really a time event. Right now - I have the ability to introduce a timestamp onto an input. Hydrologists and master irrigators would tell you the total amount of water, and the range of time it is applied depends on how deep your water penetrates.

I would think there are other events that would lend themselves to this type of thinking. Something like a danger event [getting too hot or too cold] would be nice to know how long heat was high or cold was low.

Further I can see this type of aggregated event being useful in other inputs or events as well - time to cool is big metric for me and my leafy greens. Quality increases significantly when the total uncooled time is kept low. This cannot really be thought of as a point event - rather the time between cut and cool needs to be treated as a collective unit.

In indoor agriculture I have a couple of other use cases where it could make sense as well - unlike your outdoor scenario - I can view light application times for purposes of GDD or other sunlight metrics as a block of time. For my analytics having the sun be a ramp with a temperature gradient simply doesnt apply so having this idea of on-off cycled block aggregates is a clean representation in the indoor world.

In my outdoors experience I can think of many other times when a timeblock would apply. For example, water applications and test time, presumptive tests, product wetting etc. I know there are scenarios where you cannot water or conduct other events like spray or applications based on certain time criterias. For example - If we test clean on our field of arugula - we cannot irrigate between now and harvest.

I shore this up almost like an execption - same thing if we sprayed - perhaps we have to wait a certain number of days before we can assume the insect repelling agent has taken affect. Similarly, perhaps a trap crop or other diversion technique is deemed effective for 30 days. Sure - the application date is the input date - but the event “crop protection” actually occurs for a thirty day period.

Kind of like a video game powerup type of thing.

I envision this allowing the FarmOS user several relationships to time. I think all farmers respect that we have a time based process, and events are not standalone point events. For the live look - as in sensors - it is logical to have that point-time relationship.

Conversely, I think about my time breaking new ground, or applying fish, compost, other nutrients, even GDD. We know that these have a point application Aug 3 2021 we applied the product. We also know that the efficacy of the application has a gradient, in my mind this means you need to have the concept of blocked time in order to represent - There will be x amount of this nutrient in the soil from the compost application y days from the point application.

I still dont exactly know how it is this SHOULD be dealt with. But I am now well into my journey on FarmOS, I have used Trimble Ag (big ag software), as well as the John Deere info stuff, ive seen farmers do it on excel, power bi etc etc etc, about any data capture scheme you can think of and the one thing I come back to all the systems missing:

The farm is an ecosystem, when I put nutrient in my hydro reservoir its not like my NPK is a square wave Nutriented or Un-Nutriented, testing and monitoring still only give you a point in time look at that, but what is really happening is time-blocks, this idea that an application isnt just a standalone event, it has a tip, a tail and geometry to it point in time analysis is simply unprepared to answer

This is one of the things that fascinated me so much about the intersection of big data and farming. Unlike my previous work in mortgage portfolios, big-banking, and financial markets, farms are organic and traditional mechanisms used to measure packetized loans, or stocks, or portfolios simply dont apply to the gradient-rich environment that is farming.

Anyways I will stop pontificating - if you are ever in Phoenix ill buy you a hot or cold beverage of your choice and we can nerd out on data-mining our farming operations

2 Likes

Very interesting… I agree that being able to represent these “gradients” would be quite useful in farmOS. Representing these in a bit more “abstract” way is the challenge… I still need to ponder on your thoughts… but one thing comes to mind!

It’s actually quite related to these ideas - the farm_rei module we created for Rothamsted research. This allows a Restricted Entry Interval (REI) to be defined for different materials/chemicals, commonly a # hours or days. When an input log is applied to an area with these materials (a point event), notices are shown in the farmOS dashboard, map, the area’s page, etc for the REI duration (gradient) so workers don’t enter without proper PPE.

I think there are (at least) two pieces to consider:

  1. How (and if) the “gradient” is represented in the data model (the REI interval is an additional field on the Material terms)
  2. The business logic that uses/displays the gradient (the REI module checks for active REI intervals in various places and alters the UI)

I’m not sure that #1 is always required. But it seems like #2 is always required. This makes it hard to replicate the behavior on other systems that use the farmOS API.

Would love to explore this further! Maybe we could all chat about these ideas in the upcoming monthly call? Monthly Call - farmOS.org

1 Like

For sure, I’m 100% into the call. Things have been a little tumultuous down on the farm and I missed last months call. But I intend on being there monthly. There’s only been one call since I brought farmOS to Munch.

my exposure is the farmier deployment @mstenta so kindly hosts; albeit begrudgingly, my board/co-owner were adamant that I was not allowed to self host. I probably deserve it, I’m great with data but not a networking or front end guy at all.

Alas that rei stuff is at its core exactly what I’m talking about. Representing events with a gradient instead and/or point events.

Talk soon

2 Likes