CSV Importers in v2.x

@Farmer-Ed Looks like you figured this out! I was going to say: yes Animal assets have a required “Animal type” field on them - which is a reference to a term in the “Species/breeds” vocabulary. So you must define an animal type before you can save an animal asset.

1 Like

Yes thanks, I had expected that but copied the wrong relationship :flushed: at least now I know where to read the error.

Is it possible to send an array of Assets or Logs? or just one update per POST?
I crashed my test server (a little Pi) by sending too many posts too quickly.

I have a working solution now with a 2s delay between posting. Just used it to add 96 terms to the animal taxonomy for the cattle breed codes used by Department of Ag with a CSV file.

2 Likes

Yes actually! farmOS includes the Subrequests module, which allows for multiple “subrequests” to be included in a single POST.

I haven’t used it myself (I believe @paul121 has), but the documentation on that module’s project page may be enough to get you started: https://www.drupal.org/project/subrequests

2 Likes

Hey @brunocacao and all, a few things:

(1)
We use (and build) surveystack.io (all open source also) for inputting data - it has a farmOS integration so you can build your API calls for saving logs and assets custom right there in the form. Happy to help you get it set up or record a video of how it works. This would be an alternative to ODK.

(2)
We’re thinking about creating a CSV importer for 2.x, we have a near term need to pull in a bunch of data from orgs in 2019 and 2020. A couple thoughts…

A partner (Digital Green) has been evaluating Dromo (Dromo Javascript Codesandbox) which is basically a pre-made csv import tool that can allow users to select data and point it to a data structure. Not sure if this is useful but it is for SurveyStack for sure… and excited about using it (maybe) as a general purpose lab CSV data importer. Any ideas on that?

Finally - does anyone feel like have a JS CSV importer would be useful? I ask because that would allow us to use it from inside surveystack and may be an easier implementation… not sure, but just curious if this would have value to others.

3 Likes

Hey @gbathree : good to hear! As a farmOS user with 1.x instance for some years now -and Grower Partner in RFC (by that token, a SurveyStack user) since last year- preparing now for data migration into 2.x, i am intrigued by the possibilities you raise. Specifically:

I don’t know anything about ODK, but if it’s like Kobo Toolbox (thanks for the pointer, @brunocacao), which i’ve explored a bit, this seems to me a neat toolkit for building mobile data collection apps… But for piping data thru farmOS API, i don’t see how that might work. Neither do i (yet!) see as how SurveyStack can be used to capture & upload @brunocacao’s post-harvest process logs, or my own (custom) observation logs, via farmOS API calls that “you… can build right there in the form.” Is this indeed running code that you can show us how to use now? If so, then deal me in on that demo! But not yet ported to v2.x API -right?

You and me both, @gbathree! How soon is this “near term need” having to be satisfied? :pleading_face:

Beauty! So easy to use -at least as far as mapping the fields to the given schema goes, tho i don’t yet know about the plumbing to move it from there thru farmOS API. What. might that involve, i wonder?

Am not a JS coder myself, but am guessing that such a thing would have the advantage of portability, so it could be deployed in SurveyStack and/or farmOS and/or your choice of mobile app-building kit. If that’s a fair assumption, then: Yes, that would seem a most useful thing, since this need to import CSV files into farmOS v2.x is what prompted me to kick off this conversation. :smile:

3 Likes

I can safely say everyone in the room is interested.

I had a very quick look at surveystack.io it looks very interesting but maybe not that intuitive certainly a video would be much appreciated. I was looking at building forms with Bubble.io earlier in the week and then a few other options were mentioned in posts above so I’ve been giving them some consideration too.

I’ve been working on a Node Red CSV importer which I now have functioning for my most basic needs, but I’m all for seeing how others implement CSV importing too, my current node red flows use the API but after talking with @paul121 yesterday it may well make sense to import farmOS.js into Node Red and possibly eventually build some custom nodes, so any JS based farmOS tools would be most welcome.

3 Likes

@walt, what are you looking for in a CSV importer?

So far I’ve replicated what I had with farmOS 1.x a simple importer, but I plan on developing it a bit further.

So my Herd Profile CSV importer will import animals form a CSV file from the Department of Agriculture register and create new animal assets. Currently there is a risk of creating duplicates if an asset already exists in farmOS, I think this was also the case in the CSV importer in farmOS 1.x

The Department of Agriculture Database should always be up to date as all birth registrations, movements to other farms/marts/abattoirs/knackeries etc need to be notified through their portal so my plan is that my complete profile should be able to be read each time and only update changes, so new assets added and missing assets marked for Archiving with a note for off farm movement.

1 Like

Quite simply, @Farmer-Ed: i am looking for a single end-user tool that i (or minimally-trained staff) can use to import CSV files generated by whatever tool we use for data capture, map that data to the farmOS schema as appropriate, and then POST or PUT the data (depending if records are new, or update) to my farmOS 2.x instance in a single operation from a net-connected computer desktop.

That’s not too much to ask, is it? :pleading_face: :pray:

I guess the devil is in the details of how to protect the database from introduction of incomplete/ duplicate/ otherwise wonky data, but that Dromo thing @gbathree shared does data validation in a nice way at the front end… And the API will reject illegal transactions anyway, if it comes to that (like the “Import CSV” provisioning in farmOS v1.x -which does also have that bothersome problem of creating duplicate records, which API would not allow, presumably).

Presuming that every approach thus far mentioned in this thread involves calls to the API (correct?), i wonder if Postman or some such tool might develop into the most flexible AND user-friendly tool, ultimately. It still requires more tech-savvy than i have at present, to build a custom “Import CSV” facility, though i have used it to GET some useful info out of farmOS v2. But i’ll bet it would be a snap for you, @Farmer-Ed ; have you ever given that one a try?

3 Likes

I hope not. I would ask for this too… :pray:

3 Likes

So do you want to be dealing with CSV files at all? Or just tools that integrate with farmOS?
This is where I think Node Red can be very useful in building the plumbing as you call it between different data sources and destinations, it wouldn’t be much different to pass data between API’s of different applications. If a few custom nodes were developed specially for farmOS it would be very straight forward to build flows with very little code.

There is only so much validation that can be handled at the front end, like checking its the expected sort of data for the fields, like is it a date or an e-mail or integer etc. The API will only accept correctly formatted data, but its not impossible to correctly format the wrong data and the API dose indeed allow you to create a duplicate record. My current CSV flow has this issue too, but I will when I have the time get around the issue by adapting it to search for an Animal Asset with the same tag ID before creating a new record.

This is the current complete flow, the first node is basically a button to start it, then it splits to simultaneously read the CSV file and retrieve the UUID’s of the breed codes from my farmOS, they then merge and are fed into a function to format the json objects (1 per asset to be created) before being sent one a time back to farmOS. Even though there is 13 nodes here only the function node really has a bit of code in it, the rest are pre built nodes or just contain a few variables for the HTTP requests. With some custom nodes built for farmOS this could be reduced to about 4 or 5 nodes.

If all of that sounded like gibberish, I’d say setup Node Red and import some flows, the green nodes are called debug nodes and can be connected at any point in the flow so you can see exactly how the messages are passed and change at each step. Obviously with a test server or you will import a lot of Irish cows :joy: I have a separate flow for handling farmOS authentication for all of my flows.

Yes I have used Postman, I used it only 2 days ago when I got a bit stuck with creating a new animal asset, initially when I didn’t know where to read the http response in Node Red I ran my json through it and it gave me the answer I needed. It is a very useful tool for exploring the API but not sure its going to be the most user friendly tool.

1 Like

@walt sorry if I’ve hijacked your thread, I can move my ramblings back to the node red thread if they are more suited there. I combined a lot of these nodes into sub flows this morning. Which will make it easier to follow.


So I’m building a subflow for each function I need, then these become reusable blocks which require no editing. Only the function node will require editing code to match the specific fields of the CSV to the farmOS API.

1 Like

Hey @Farmer-Ed : re your latest:

No worries, mate! In fact, as i so often get tunnel-vision in pursuit of a particular goal, i really appreciate it when somebody questions the goal itself, so long as they have a viable alternative to present, as you clearly do.

CSV is for me not the ideal (that would be seamless standards-based data interoperability across all platforms- but let’s not hold our breath waiting for that one!), but rather a simple expedient: CSV is the one universal LCD standard for serialisation of data matrices, it seems to me. Of course it would be great if all my tools would just auto-magically integrate with farmOS, but… How should that work? We’ve got a spreadsheet-based system (tool = LibreOffice) for crop planning & harvest logging, which needs to save files in either .ODS format or .CSV, so -latter being the more universal- it may as well be .CSV, right?

As it stands, based on my own experience, i would have to agree -yet the pace of development on that project is pretty awesome, it must be said. For all those actively involved in exploring APIs, Postman is clearly the most accessible, widely-adopted, documented & user-friendly tool available. And, tho i’ve not yet had time to dig deeper, a quick search on their site for “CSV upload” turns up 179 collections, including a good few of what looks to me like having pretty solid test-suites… What one probably needs to ensure integrity of database, right?

Am sorry i have not yet found time to dive into your Node Red scripts, Ed; i did visit your github repo, but absent some sort of “getting started” doc or video that lays out all the steps, this looks like a project for my next free weekend (this one being already booked, alas :frowning:

2 Likes

If Libre office is your primary tool for collecting data then CSV absolutely makes most sense. I was just wondering if you were using something like some of the form builder tools mentioned earlier and retrieving the data as CSV, as I’m sure some of them have their own API’s.

I probably haven’t looked at Postman enough, but I’m not surprised that there are CSV tools available, getting data from the CSV file is the easy bit, unless there is a plugin made for farmOS then some sort of scripting will still be needed to format the json for the API and fetch UUID’s when necessary from farmOS.

Well the thing about the flows saved in my github is they are saved in json form, so are not going to look like much until they are imported into Node Red, then they will create the visual blocks. I’m far from an expert in Node Red or any sort of software development but I like to tinker, I was considering doing some sort of tutorial on what I’m doing in my blog shortly.

3 Likes

Just FYI, sorry I said ODK I meant kobo - kobo is built on ODK.

Well I could definitely do a video if folks are interested. Sorry we don’t have better documentation, it’s a failing that needs to get fixed.

I can show for example how to connect a farmOS account (1.x at this point, but soon 2.x only) to a user, and then how to push data from a survey to those fields or plantings.

I could ask Octavio who’s been working on a file importer for SurveyStack also about using that same tooling I just described to make a CSV importer. We’ve been investigating Dromo.io also, which seems quite nice… this could be a great application for it.

5 Likes

Newsflash: This “JupyterLite and farmOS - Proof of Concept” by Don Blair opens up some interesting possibilities -i.e.:


Cribbed from chat w/ @mstenta :

  • [yesterday i was hacking around trying to wrap jupyterlite in a module - so you could just do [myfarmOSurl]/jupyterlite :-D]
  • (imagine people being able to share/import CSV importer scripts for specific use-cases!!)
  • still some kinks to iron out… but that would allow it to work without requiring cross-domain config
  • (and could enable some really neat ideas!!)
  • If this could be an alternative to building CSV importers in Drupal… wow!
  • although it is still kind of “advanced” to use… so we’ll have to see what shakes out[
  • (imagine people being able to share/import CSV importer scripts for specific use-cases!!)

It's that last possibility that really has my wheels turning -maybe you too, @Farmer-Ed , @pat , et al?

What i like about this JupyterLite solution is- if hosted on Gitub as Don has done in the above demo -the ability to commute access to anyone with a browser on the net for free, with no need to install any special software. Of course users must be trusted to keep their key combo secure- i.e. dump that browser cache, if there’s anybody else going to use that browser. But that is always the case anyway, right?

3 Likes

Wow JupyterLite is super cool as is Don Blair (ps how is he not in this discourse no @ mention!)…

I’m going to share this link with Octavio who’s doing a lot of this work with us… I have to say, a lot of ways to do this here, pretty neat stuff. Hoping Octavio can provide some thoughts from our scide (har har, tho that sounds like pesticide maybe not so good).

4 Likes

Interesting. Just having fast-viewed it yet., im not sure if I understand how simple this would be for the user tho…

In general, I think there should be an import feature in FarmOS that don’t require coding skills or difficult things to setup. Not every farm has developers at hand, or the time/interest in such things. Around me most farms have only 2 working hands.

Such one-time imports could perhaps be a paid service at Farmier.

3 Likes

Am +1 on both counts, @pat ; seems clear to me that our new 2.x platform need both:

  1. a user-friendly facility for uploading template-based CSVs; and
  2. an accessible way to engage template development and customisation services.

From my perspective (non-developer, but rather “enthusiast” about bending tech to serve my farm’s purposes), JupyterLite strikes a balance that can work for both purposes reasonably well… And moreover can facilitate the process of sharing data and insights more readily within our farmOS community.

One super-interesting thing along such lines that emerged from yesterday’s Dev chat (as a “tourist” in devs world, i sometimes lurk there :grin: ) is this Real-Time Collaboration tool, for JupyterLite that could enable “naive user” and more code-savvy tutor to join in a session that might be about how to navigate/ run/ customise these API scripts, in a very hands-on, interactive & efficient way.

Now i don’t know how close this is to becoming real for us, but- based on what i heard and saw of the ideas & actual code flying around yesterday -this looks pretty feasible in the near term… And suddenly the “lack of ‘Import CSV’ facility in v2.x” problem is looking like a real opportunity in disguise!

2 Likes

That could help creating some useful “special import” modules in the future. :grinning:

3 Likes

It’s great to be able to join the development and community discussions as well as exploring these ideas here, its probably one of the best aspects of farmOS. I’ve used cloud based farm managment packages for the past few years, they all cater for a particular type of farm but if you want to do anything off script its tough luck, but the open nature of farmOS is what is really appealing, the API along with farmOS.py and farmOS.js really have a lot of potential. I get @pat’s point though, not every farmer wants to sit down and code, but I’m regularly blown away by the amount of farmers who do, looking at some of the modules that were built for farmOS 1.x, I don’t think you’ll be waiting too long to see a whole array of solutions to some of these issues.

The Realtime Collabertion feature is really cool, if you have that code savey tuter available to you.

I actually think you could be right on the opportunity in disguise bit, I think if farmOS 2 had come with a CSV importer I would have made done with it, but now I’m building something exactly tailored to my particular use instead.

4 Likes