I’m very new to FarmOS and am busy uploading the many aspects of my farm. I plant hundreds of seeds, veggies and trees every week and am trying to streamline uploading of data onto the site. However, for some reason, I am unable to find the excel templates to import logs data.
I have activated the ‘Import KML’ section, but still have not found any templates. Hoping you have a link you can send me?
Thanks and apologies if it was sitting in front of me the whole time!
What logs are you trying to upload?
I presume you know kml is a markup language for geographical data?
And if so what form is your current geographical data in?
Hey Georgie: Welcome to the club! (I mean: both farmOS users club and that distinguished segment of us busy farmers with lots of data to upload
If i understand correctly your need, it is not simply geographical data (what KML is all about, as @Farmer-Ed points out) but also tabular data about your plant assets, seeding &/or transplanting logs, etc. -yes?
If so, you should know that- though there were “Import CSV” templates available for version 1.x of farmOS, those import templates have not been ported-over to current version 2.x of farmOS. As discussed in this recent thread, and this earlier one out of which it arose, there are other methods for importing data via API, which are certainly more flexible & powerful, but not so point&shoot easy as those old “Import CSV” templates.
One thing i can recommend, if you have any degree of scripting-savvy, there is this excellent farmOS.py script by @paul121 that can get you started uploading & downloading data to&from your farmOS instance via API without too much ado.
In any case @Georgie : i hope you find a way of uploading your farm data that works well for you. Do keep us posted as to how it goes!
Thank you for this…
Yes that’s exactly what I’m looking for to upload to FarmOS and download too.
I am definitely not tech savvy enough to do any coding… Was hoping for an easy ‘point&shoot’ option! Sorry to hear there’s no option yet for 2.x as this is crucial for my work I really don’t want to spend all day inputing each thing I’ve planted or seeded.
@Farmer-Ed yes, as you can see I’m all mixed up regarding the kml and csv and was really hoping it would be much clearer.
If anyone has any good ideas for this I’d be really grateful.
The lack of a CSV importer in 2.x is an issue for some alright, CSV export is possible at the moment though.
The farmOS API is much more flexible than an integrated CSV importer will be as scripts can be tailored for exactly how you want the data presented for import and export. Having said that I’m sure a CSV import module would be welcomed by many.
Not sure exactly how non tech savvy you are, but the API is still going to be your best bet at the moment, you could always ask or even hire someone to write a few short scripts that would cover your needs.
Or perhaps, if you really wanted to learn enough to import a CSV then there is plenty of help/guidance here.
Edit: I’ve renamed the thread to "Templates for CSV uploads?"
Gotcha, @Georgie. I don’t know to what extent you might have explored possibilities on the download side of things, but i would say that the “Export CSV” link that appears at bottom of most list views in farmOS has enabled me to satisfy all of my download needs thus far, in that i can select/ filter/ sort a set of asset or log records pretty much any way i can think of, and then download in tabular (i.e. CSV) form. To upload a set of records tho… Not so simple in v2.x, alas.
I feel your pain, Georgie -something that i have struggled with for longer than i care to recall. Two most important things that i have learned from this struggle are:
- Give more focus to all those aspects of farm record keeping that farmOS is and can easily be helping me with right now. KML export AND import of farm maps are two of them, b/t/w; didn’t fully realise that myself until just last week (duh! always more goodness to discover here); and
- Regarding those other farm record-keeping challenges that are not so easily managed in farmOS: try to give a certain amount of time to sharing and discussing those challenges here, because (a) whatever the problem may be, you are probably not alone with it (certainly not, in this particular case!); and (b) whatever the Developer team on this project lacks in terms of size, it more than makes up for in terms of passion/ commitment/ willingness to not only tolerate dumb n00b questions, but moreover to value them, and do all in their power to help.
That’s my €0.02 worth, Georgie, as a busy farmer who- like you, i gather -has not enough time to enter line-by-line data about all my veg crop developments and orchard operations and animal movements and whatnot, let alone enough time to learn the basic coding skills i wish i knew. Still: this software is right up there with the handful of tools that i can no longer imagine farming without. Spreadsheets are great, as far as they go, but at a certain point, you need a database… And, when it comes to that, farmOS has the most farmer-friendly structure and interface(S*) that i’ve yet to encounter.
Now as to those interfaceS: besides the standard GUI, there are plugin modules (e.g. ‘farmOS KML’ and ‘farmOS Import KML’), API scripts (e.g. farmOS.py), Node-Red scripts (hat-tip to @Farmer-Ed ) and more, all of which can certainly do with some refinement… Which is i guess what we’re talking about here, right?
Cheers @walt, if you, @Georgie or anyone else want an example of the simplest CSV importer in Node-Red, just share a sample CSV file (just a few lines) and the target log type and I can produce one and explain its operation. (It will be the most basic produce a log for each line of the CSV flow).
OK @Farmer-Ed : i append below in CSV form a little list of transplants we did just last week, what i’ve yet to upload as Plant Assets to our farmOS instance -just the essential columns of data that we find necessary to manage. Easy enough to import via API…
BUT: The trick is then at harvest time the uploading of harvest log records in such a way as to be correlated each one with the asset record to which it pertains. NB: each ‘Plant asset’ (i.e. a crop planted in a particular bed on a particular week) normally yields a goodly number of harvests… And for these records to be correlated in the database, each 'Plant asset" name needs to be guaranteed unique. This is what adds significantly to the complexity of our “Crop importer” and “Harvest log importer” scripts…
… And this is also why the simple “Import CSV” affordances that were available in v1.x were of limited value, because there was no provision for guaranteeing uniqueness of uploaded records -meaning that duplicate records were all too easily created. The workaround i found was to flag all records so uploaded with “Needs review,” then go through, review and delete all those dupe records -a real PITA, which is why import via API is the superior method, if properly scripted.
Name,Description,Crop/variety,Season,length FB,Nr plants FB,Area/Location
2022-W17 Zucchini HVW11-N,Length (m): 16; Nr of plants: 20,Zucchini,SPRING,16,20,HortasDoVale-West (HDVW) - bed 11 north section
2022-W17 Tomato roma HVW12,Length (m): 28.6; Nr of plants: 70,Tomato roma,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 12
2022-W17 Tomato roma HVW10,Length (m): 28.6; Nr of plants: 70,Tomato roma,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 10
2022-W17 Peppers HVW12,Length (m): 28.6; Nr of plants: 70,Peppers,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 12
2022-W17 Peppers HVW10,Length (m): 28.6; Nr of plants: 70,Peppers,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 10
2022-W17 Hot peppers HVW12-S,Length (m): 8.91; Nr of plants: 22,Hot peppers,SPRING,8.91,22,HortasDoVale-West (HDVW) - bed 12 south section
2022-W17 Green beans dwarf normal HVW11-S,Length (m): 27.6; Nr of plants: 124,Green beans dwarf normal,SPRING,27.6,124,HortasDoVale-West (HDVW) - bed 11 south section
2022-W17 Cucumber HVW11-N,Length (m): 11.6; Nr of plants: 52,Cucumber,SPRING,11.6,52,HortasDoVale-West (HDVW) - bed 11 north section
Well I probably asked for that
Note: basic CSV importer = farmOS 1.x style importer which I will do anyway for the simple example.
But yes you are absolutely correct about the limitations of such a simple CSV importer. With a little further development it is absolutely possible using the API to check for duplicate entries and link assets etc. would possibly need to know more about your particular workflow to implement fully.
BTW I think your CSV is missing a heading.
Never mind, I think Libre Office was taking a semicolon as a delimiter as well commas
Sorry, @Farmer-Ed , if that reads like reverse Polish notation (with a bit of Portuguese thrown in)… But we do need to pack a lot of facts into those fields, especially ‘Name’ (which concatenates data from 3 spreadsheet fields) and ‘Description’ (which concatenates both bed length and number of plants, plus text to make those integers comprehensible at a glance).
If you like, Ed: that would bee cool to see
“Absolutely possible” for you, maybe (/me = just another cargo-cult code leech
You think? I don’t know, but maybe the mix of punctuation (e.g. colons & semicolons in the Description field, slashes in col’s 3 & 7) were too confusing. Here’s a simple copy/paste of header row & 1st row of data, parsed as HTML for clarity:
|Name||Description||Crop/variety||Season||length FB||Nr plants FB||Area/Location|
|2022-W17 Zucchini HVW11-N||Length (m): 16; Nr of plants: 20||Zucchini||SPRING||16||20||HortasDoVale-West (HDVW) - bed 11 north section|
I’m not a plant farmer so not sure I understand the logic here fully.
From the CSV you’d like to create a new asset? then a transplant log?
does an asset not exist already from an initial seeding?
is the name the Asset name and do you derive the log name from that in some way?
excuse the ignorance of a simple beef farmer
Good questions, @Farmer-Ed -to which i would answer mostly with a yes, except about the initial seeding, which step we are not presently recording in farmOS; as far as farmOS is concerned, we just go straight to transplant (although we do have a greenhouse operation where we germinate our own seedlings, and only occasionally buy-in seedlings for transplant).
So: the “Crop importer” script developed by @paul121 for this case checks to see if there’s already a ‘Plant asset’ in my farmOS database that matches the one i’ve asked it to import, and if not, then it creates the asset, and then a transplant log that looks like the following copy/paste from GUI:
|Status||ID||TimestampSort ascending||Log name||Assets||Location||Quantity||Equipment used||Flags||Log category||Assigned to|
|Update this item||Done||3174||2022-04-25||Transplanting log 3174||2022-W17 Zucchini HVW11-N||HortasDoVale-West (HDVW) - bed 11 north section||* Length FB ( Length/depth ) 16||* Nr plants FB ( Count ) 20|
Does that help?
If “ignorance is bliss,” as they say… I envy you, mate! At times i am tempted to just turn all these annual annual crops back into soil, chuck all this vegetative complexity in favour of what sure seems like a simpler life, running a herd of cattle (tho i guess it’s not all “ti yi yippy yi yay” all day, eh?
yep, that all makes more sense now.
It’s the reason why a little bit of scripting is not a bad thing as no 2 farms are the same no matter whether they deal in crops or animals. I’ll use your example anyway just to have some plant and animal templates for anyone looking for similar.
OK @walt probably not as simple as promised but hopefully useful just the same. It gave me something different to test my farmOS Nodes on anyway. Obviously you already have a solution from @paul121, but this may serve as a template for anyone else looking for similar, perhaps a link to the python script you are using may be useful too so there are a few options on here? (minus any hardcoded credentials etc.)
I’ve saved the Node Red flow here in json form: (Honestly its simpler than it looks )
When imported will look like:
To keep it simple(-ish) for understanding, I’ve split the task up into separate flow segments.
Authentication - Authenticates and retrieves taxonomy and land UUID’s
Create asset/Store CSV file - Reads the CSV, stores all fields and Creates the Assets.
Create Quantities - Quantities need to be created before the logs
Disclaimer: This flow is far from complete and could do with some additional error checks including possible duplication of logs/assets. I’d recommend anyone trying it to have a test environment that can be easily reset. Also the sections can be linked up but could probably do with a little additional logic to ensure each section is complete before the next starts to avoid sending too many http requests at once.
Wow -for a simple cattle man, you got pretty deep into the plant-based data model here, @Farmer-Ed. I appreciate your taking such an interest in this UseCase!
Yes: as soon as i get a bit of time away from mission-critical stuff here on farm, i will clean up and share those scripts we’ve got running in some form (either gColab or Jupyter Notebook) that i hope will be useful to others.
In fact Ed: this way of modelling dataflows appears to me beautifully simple, on the face of it, though it must be hiding a fair bit of complexity… And if the user can run these scripts without ever having to see need should arise), then that sounds like a very good thing!
Well thank you very much for the sample data it was good to test the nodes I built are flexible enough for different uses than my own. I think they did Ok but also helped me see a few areas for tweaking.
I think it would be good to have a collection of options here, at least until the new tutorials site is launched anyway. I’m sure this question will come up again and again, at least until a CSV module is made, although it may be a hard task to have a simple CSV importer with the flexibility of the API.
It is very nice that a lot of common tasks are available as drag and drop blocks already such as the CSV nodes, there is good and bad to the complexity being hidden away though as sometimes you can be constrained to the inputs and outputs available to a particular node. I was considering building a few more farmOS specific nodes to simplify handling CSV’s even more but there is a danger of losing some flexibility but maybe no harm if there were enough use case templates included.