farmOS, NodeRed, Home Assistant etc

@mstenta, What is the best practice with regards to access/refresh tokens?

I’m considering a flow to monitor Google Calendar, so If I enter an event in the calendar with the phrase “Cow Serviced {{Cow Name}}” in it will create a " Cow Serviced {{Cow Name}}" Observation and a “Due Calving {{Cow Name}}” Observation.

As this flow will run unattended, should the flow use the Refresh Token with every request regardless of expiry? In practice with this use case I know It shouldn’t matter as the bull is only likely to be observed with a few cows per week, but for higher frequency automated tasks would this be the process?

And I notice that farmOS seems to hold all of the tokens generated, is this indefinitely or should they be pruned in some way after a period?

1 Like

Yes, getting a new refresh token with each request is fine.

And I notice that farmOS seems to hold all of the tokens generated, is this indefinitely or should they be pruned in some way after a period?

Oh yes I’m just realizing we need to document how to configure farmOS’s cron to run in the installation instructions

Basically you need to set up your system’s cron to run farmOS’s (Drupal’s) cron routine. This will handle the automatic cleanup of the tokens in the database.

Here is the upstream Drupal documentation: Configuring cron jobs using the cron command | Setting up cron | Drupal Wiki guide on Drupal.org

We need to add a simple bit to our documentation to mention this too…

1 Like

So its just run the URL provided in /admin/config/system/cron periodically?
Easy :cowboy_hat_face: image

Which brings up one more question, the link provided starts with http:// instead of https://, it is the same elsewhere like the sensor listeners. I am using https and the links with http:// won’t work until I add the ‘s’ which is fine but I am wondering if I’m missing something, perhaps in settings.php?

2 Likes

@Farmer-Ed are you using Nginx as a reverse proxy, like what’s described here: Local HTTPS | farmOS ?

There are two important bits that affect how Drupal automatically determines whether or not to use http or https for links:

In Nginx config:

      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

In settings.php:

$settings['reverse_proxy'] = TRUE;
$settings['reverse_proxy_addresses'] = [$_SERVER['REMOTE_ADDR']];
$settings['reverse_proxy_trusted_headers'] = \Symfony\Component\HttpFoundation\Request::HEADER_X_FORWARDED_ALL;

Check those and see if they help!

1 Like

Hmm… As far as I can see I have those options configured.
It’s a fairly minor issue as Https requests work and the http requests are refused. I’ll take another look at my settings later, maybe some other option in Nginx is overriding.

I’m working on a new Node Red flow. FarmOS-Node-Red-Flows/Cow-Service-Calendar.json at main · Farmer-Eds-Shed/FarmOS-Node-Red-Flows · GitHub

It Reads Google Calendar events starting with 'Cow Serviced {{Cow Name}}'

Then create 2 new farmOS logs 'Cow Serviced {{Cow Name}}' and 'Due Calving {{Cow Name}}'
Logs have the 'Animal Asset' attached and are marked with a 'Review Flag'
It also adds a 'Due Calving {{Cow Name}}' back to Google Calendar.
Notes added to calendar description are added to log notes.

It’s a bit rough, I’m sure it needs some refinement and some additional checks.

3 Likes

@Farmer-Ed Maybe you need to configure the trusted host settings? Trusted Host settings | Installing Drupal | Drupal Wiki guide on Drupal.org

Really cool to see these node-red scripts coming together!

2 Likes

Hi

@Farmer-Ed , I’m trying to follow your footsteps…

I’m working on this flow of yours: (Calving calendar)

I’ve cut the flow after the GET-node, just to see if I get that far, and to get familiar with the nodes.
I have authenticated, and the FarmOS url is sent to to the node.

But the GET-node produces these 2 errors:
Error: getaddrinfo EAI_AGAIN get get:80

Warning: msg properties can no longer override set node properties. See bit.ly/nr-override-msg-props

Object sent to GET
{"_msgid":"a975ad5e.8e4c3","payload":1644440749801,"topic":"","headers":{"Authorization":"Bearer __Stripped the token___ ..."},"url":"https://__farm__.farmos.net/api/log/observation","method":"GET"}

Did you encounter anything like this?

1 Like

Yes, it looks like you have sent the config for the http node in the message sent to the node and also set it in the node itself. It has to be one or the other but not both

For example, if you send the URL and/or method in the payload then don’t set it in the http node text box too.

1 Like

You were right @Farmer-Ed .
The errors disappeared. Got a bunch of HTML-code.
I’ll investigate further…

2 Likes

Working on a GUI for Importing/Exporting the CSV files using Node Red dashboard

Flows are borrowed and tweaked Node Red Repo

3 Likes

Seems there are probably a lot of new options coming online, I’ll definitely be keeping an eye on the JupyterLite thread as well as ideas from the CSV importers thread.

But for now anyway, I’ll keep chipping away at Node Red as handling CSV files, building basic GUI’s, and integration with HA and Google Calendars are all starting to come together for me.

I found a node openapi-red (node) - Node-RED which looks interesting as it can read the schema and return drop down boxes for making requests. (OpenAPI though not the default JSON:API)

I’m having some authentication issues with it though, so have only tested as an unauthenticated user, so not for the production server yet. It should work with the bearer token, but I’m not sure how it is meant to be passed, documentation says msg.openApiToken but it doesn’t accept the basic string format.

It also involves installing some openAPI modules for Drupal/farmOS.
- ‘./www/web/modules/openapi:/opt/drupal/web/modules/openapi’
- ‘./www/web/modules/openapi_jsonapi:/opt/drupal/web/modules/openapi_jsonapi’
- ‘./www/web/modules/schemata:/opt/drupal/web/modules/schemata’

2 Likes

Quick Demonstration of the CSV Importer

4 Likes

Looks awesome!!

2 Likes

Wow that is PRETTY COOL @Farmer-Ed !!

1 Like

Very intuitive workflow, @Farmer-Ed ; for all the fun i am having in Jupyter notebooks, i must admit: creating a CSV uploader so point-and-shoot simple & slick as yours is something i cannot even imagine doing in a .ipynb notebook. Congrats, mate!

2 Likes

Thanks @walt,
It does look like you have been having some fun with JupyterLite! I’m well impressed, it is well documented too, an area that I’m often lacking.

I think Node Red can do all I need for now.
There is still some work to be done and can be refined a bit.

The ability to use GUI features of Node Red Dashboard and merge with farmOS /Home Assistant makes a nice point click solution without needing any web development skills, worked out better than I’d expected after finding someone else’s file handling flow, Browsing files in dashboard - YouTube.

I have built in some checking of the existing records to avoid duplicates (Currently as long as there is less than 50 records)

I still have to deal with paginated responses from GET requests, I more or less know how I’m going to deal with it but I actually don’t have enough records in the database yet for it to be an issue. That will change quickly though as activity on the farm is about to pick up a bit.

My Simple CSV upload flow has grown legs a bit though!

From this:

To this:

2 Likes

Re your wiring diagram above: Yikes! Makes me all the more amazed, how you manage to hide all that complexity behind such a lovely interface -which is what’s really needed in the end, before we can say we’ve got a usable “Import CSV” feature.

3 Likes

Well 3/4 of that is the file handling and the GUI. The Interface tools are part of the Node Red Dashboard. The blue nodes represent a button or other UI item and are more or less pre built.
Home Assistant serves farmOS and the Node Red Dashboard in 2 iFrames to make them look like they are 2 different tabs of the same application.

3 Likes

This thing can really grow legs.

I’ve managed to get API access for my herd details held by the Irish Cattle Breeding Federation, they have much more detail than the CSV files from the Department of Agriculture, including genomic data, herd fertility, cattle weights, efficiency reports. It’s API is similar to that of farmOS so not much of a learning curve.

For the moment I’ve given it the same treatment as farmOS so that I can export CSV files and import them to farmOS as it was simple to more or less copy/paste flows and change a few variables. Ultimately I think it will be best to build a new flow from scratch that can sync my farmOS herd data with the ICBF version with minimum intervention. They also have some useful mobile apps for recording weights and breeding events, like heats, service/insemination etc.

2 Likes

Cool, @Farmer-Ed -i am so jealous! Ministry of Ag here in Portugal has lots of great info about locally grown fruits/ pests/ treatments, but no API access, alas.

The other genre of data i would love to correlate is from our local weather station, which is available via API, but it does carry a cost. Still: i think it’ll be worth paying for, once i get my database in tidy enough form to permit correlation of crops & weather along the line of time… So then integration via NodeRed will be a simple point&shoot operation i guess, huh? :grin:

1 Like