Hi there,
I’m very new to FarmOS and am busy uploading the many aspects of my farm. I plant hundreds of seeds, veggies and trees every week and am trying to streamline uploading of data onto the site. However, for some reason, I am unable to find the excel templates to import logs data.
I have activated the ‘Import KML’ section, but still have not found any templates. Hoping you have a link you can send me?
Thanks and apologies if it was sitting in front of me the whole time!
1 Like
What logs are you trying to upload?
I presume you know kml is a markup language for geographical data?
And if so what form is your current geographical data in?
1 Like
Hey Georgie: Welcome to the club! (I mean: both farmOS users club and that distinguished segment of us busy farmers with lots of data to upload 
If i understand correctly your need, it is not simply geographical data (what KML is all about, as @Farmer-Ed points out) but also tabular data about your plant assets, seeding &/or transplanting logs, etc. -yes?
If so, you should know that- though there were “Import CSV” templates available for version 1.x of farmOS, those import templates have not been ported-over to current version 2.x of farmOS. As discussed in this recent thread, and this earlier one out of which it arose, there are other methods for importing data via API, which are certainly more flexible & powerful, but not so point&shoot easy as those old “Import CSV” templates.
One thing i can recommend, if you have any degree of scripting-savvy, there is this excellent farmOS.py script by @paul121 that can get you started uploading & downloading data to&from your farmOS instance via API without too much ado.
I have also found Postman to be a pretty useful tool for API exploration and development of some reports i want to look at regularly, though it takes a bit of javascript-savvy to do anything complicated -like a request with some conditional logic (e.g. first GET to see if some assets exist, before you POST a new asset, then POST a transplanting log to that asset; that sorta thing).
In any case @Georgie : i hope you find a way of uploading your farm data that works well for you. Do keep us posted as to how it goes!
/walt
3 Likes
Hi @walt
Thank you for this…
Yes that’s exactly what I’m looking for to upload to FarmOS and download too.
I am definitely not tech savvy enough to do any coding… Was hoping for an easy ‘point&shoot’ option! Sorry to hear there’s no option yet for 2.x as this is crucial for my work 
I really don’t want to spend all day inputing each thing I’ve planted or seeded.
@Farmer-Ed yes, as you can see I’m all mixed up regarding the kml and csv and was really hoping it would be much clearer.
If anyone has any good ideas for this I’d be really grateful.
Thanks!
2 Likes
@Georgie
The lack of a CSV importer in 2.x is an issue for some alright, CSV export is possible at the moment though.
The farmOS API is much more flexible than an integrated CSV importer will be as scripts can be tailored for exactly how you want the data presented for import and export. Having said that I’m sure a CSV import module would be welcomed by many.
Not sure exactly how non tech savvy you are, but the API is still going to be your best bet at the moment, you could always ask or even hire someone to write a few short scripts that would cover your needs.
Or perhaps, if you really wanted to learn enough to import a CSV then there is plenty of help/guidance here.
I use Node-Red for these tasks which is a “low code” programing environment (some drag/drop blocks with a small amount of javascript). It is far from the only option though.
Edit: I’ve renamed the thread to "Templates for CSV uploads?"
2 Likes
Gotcha, @Georgie. I don’t know to what extent you might have explored possibilities on the download side of things, but i would say that the “Export CSV” link that appears at bottom of most list views in farmOS has enabled me to satisfy all of my download needs thus far, in that i can select/ filter/ sort a set of asset or log records pretty much any way i can think of, and then download in tabular (i.e. CSV) form. To upload a set of records tho… Not so simple in v2.x, alas.
I feel your pain, Georgie -something that i have struggled with for longer than i care to recall. Two most important things that i have learned from this struggle are:
- Give more focus to all those aspects of farm record keeping that farmOS is and can easily be helping me with right now. KML export AND import of farm maps are two of them, b/t/w; didn’t fully realise that myself until just last week (duh! always more goodness to discover here); and
- Regarding those other farm record-keeping challenges that are not so easily managed in farmOS: try to give a certain amount of time to sharing and discussing those challenges here, because (a) whatever the problem may be, you are probably not alone with it (certainly not, in this particular case!); and (b) whatever the Developer team on this project lacks in terms of size, it more than makes up for in terms of passion/ commitment/ willingness to not only tolerate dumb n00b questions, but moreover to value them, and do all in their power to help.
That’s my €0.02 worth, Georgie, as a busy farmer who- like you, i gather -has not enough time to enter line-by-line data about all my veg crop developments and orchard operations and animal movements and whatnot, let alone enough time to learn the basic coding skills i wish i knew. Still: this software is right up there with the handful of tools that i can no longer imagine farming without. Spreadsheets are great, as far as they go, but at a certain point, you need a database… And, when it comes to that, farmOS has the most farmer-friendly structure and interface(S*) that i’ve yet to encounter.
Now as to those interfaceS: besides the standard GUI, there are plugin modules (e.g. ‘farmOS KML’ and ‘farmOS Import KML’), API scripts (e.g. farmOS.py), Node-Red scripts (hat-tip to @Farmer-Ed ) and more, all of which can certainly do with some refinement… Which is i guess what we’re talking about here, right? 
3 Likes
Cheers @walt, if you, @Georgie or anyone else want an example of the simplest CSV importer in Node-Red, just share a sample CSV file (just a few lines) and the target log type and I can produce one and explain its operation. (It will be the most basic produce a log for each line of the CSV flow).
2 Likes
OK @Farmer-Ed : i append below in CSV form a little list of transplants we did just last week, what i’ve yet to upload as Plant Assets to our farmOS instance -just the essential columns of data that we find necessary to manage. Easy enough to import via API…
BUT: The trick is then at harvest time the uploading of harvest log records in such a way as to be correlated each one with the asset record to which it pertains. NB: each ‘Plant asset’ (i.e. a crop planted in a particular bed on a particular week) normally yields a goodly number of harvests… And for these records to be correlated in the database, each 'Plant asset" name needs to be guaranteed unique. This is what adds significantly to the complexity of our “Crop importer” and “Harvest log importer” scripts…
… And this is also why the simple “Import CSV” affordances that were available in v1.x were of limited value, because there was no provision for guaranteeing uniqueness of uploaded records -meaning that duplicate records were all too easily created. The workaround i found was to flag all records so uploaded with “Needs review,” then go through, review and delete all those dupe records -a real PITA, which is why import via API is the superior method, if properly scripted.
8<-----(snip)----->8
Name,Description,Crop/variety,Season,length FB,Nr plants FB,Area/Location
2022-W17 Zucchini HVW11-N,Length (m): 16; Nr of plants: 20,Zucchini,SPRING,16,20,HortasDoVale-West (HDVW) - bed 11 north section
2022-W17 Tomato roma HVW12,Length (m): 28.6; Nr of plants: 70,Tomato roma,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 12
2022-W17 Tomato roma HVW10,Length (m): 28.6; Nr of plants: 70,Tomato roma,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 10
2022-W17 Peppers HVW12,Length (m): 28.6; Nr of plants: 70,Peppers,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 12
2022-W17 Peppers HVW10,Length (m): 28.6; Nr of plants: 70,Peppers,SPRING,28.6,70,HortasDoVale-West (HDVW) - bed 10
2022-W17 Hot peppers HVW12-S,Length (m): 8.91; Nr of plants: 22,Hot peppers,SPRING,8.91,22,HortasDoVale-West (HDVW) - bed 12 south section
2022-W17 Green beans dwarf normal HVW11-S,Length (m): 27.6; Nr of plants: 124,Green beans dwarf normal,SPRING,27.6,124,HortasDoVale-West (HDVW) - bed 11 south section
2022-W17 Cucumber HVW11-N,Length (m): 11.6; Nr of plants: 52,Cucumber,SPRING,11.6,52,HortasDoVale-West (HDVW) - bed 11 north section
8<-----(snip)----->8
2 Likes
Well I probably asked for that 
Note: basic CSV importer = farmOS 1.x style importer 
which I will do anyway for the simple example.
But yes you are absolutely correct about the limitations of such a simple CSV importer. With a little further development it is absolutely possible using the API to check for duplicate entries and link assets etc. would possibly need to know more about your particular workflow to implement fully.
BTW I think your CSV is missing a heading.
2 Likes
Never mind, I think Libre Office was taking a semicolon as a delimiter as well commas
2 Likes
Sorry, @Farmer-Ed , if that reads like reverse Polish notation (with a bit of Portuguese thrown in)… But we do need to pack a lot of facts into those fields, especially ‘Name’ (which concatenates data from 3 spreadsheet fields) and ‘Description’ (which concatenates both bed length and number of plants, plus text to make those integers comprehensible at a glance).
If you like, Ed: that would bee cool to see 
“Absolutely possible” for you, maybe (/me = just another cargo-cult code leech 
You think? I don’t know, but maybe the mix of punctuation (e.g. colons & semicolons in the Description field, slashes in col’s 3 & 7) were too confusing. Here’s a simple copy/paste of header row & 1st row of data, parsed as HTML for clarity:
| Name | Description | Crop/variety | Season | length FB | Nr plants FB | Area/Location |
|---|---|---|---|---|---|---|
| 2022-W17 Zucchini HVW11-N | Length (m): 16; Nr of plants: 20 | Zucchini | SPRING | 16 | 20 | HortasDoVale-West (HDVW) - bed 11 north section |
/walt
2 Likes
I’m not a plant farmer so not sure I understand the logic here fully.
From the CSV you’d like to create a new asset? then a transplant log?
does an asset not exist already from an initial seeding?
is the name the Asset name and do you derive the log name from that in some way?
excuse the ignorance of a simple beef farmer 
1 Like
Good questions, @Farmer-Ed -to which i would answer mostly with a yes, except about the initial seeding, which step we are not presently recording in farmOS; as far as farmOS is concerned, we just go straight to transplant (although we do have a greenhouse operation where we germinate our own seedlings, and only occasionally buy-in seedlings for transplant).
So: the “Crop importer” script developed by @paul121 for this case checks to see if there’s already a ‘Plant asset’ in my farmOS database that matches the one i’ve asked it to import, and if not, then it creates the asset, and then a transplant log that looks like the following copy/paste from GUI:
| Status | ID | TimestampSort ascending | Log name | Assets | Location | Quantity | Equipment used | Flags | Log category | Assigned to | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Update this item | Done | 3174 | 2022-04-25 | Transplanting log 3174 | 2022-W17 Zucchini HVW11-N | HortasDoVale-West (HDVW) - bed 11 north section | * Length FB ( Length/depth ) 16 | * Nr plants FB ( Count ) 20 |
Does that help?
If “ignorance is bliss,” as they say… I envy you, mate! At times i am tempted to just turn all these annual annual crops back into soil, chuck all this vegetative complexity in favour of what sure seems like a simpler life, running a herd of cattle (tho i guess it’s not all “ti yi yippy yi yay” all day, eh? 
2 Likes
yep, that all makes more sense now.
It’s the reason why a little bit of scripting is not a bad thing as no 2 farms are the same no matter whether they deal in crops or animals. I’ll use your example anyway just to have some plant and animal templates for anyone looking for similar.
1 Like
OK @walt probably not as simple as promised 
but hopefully useful just the same. It gave me something different to test my farmOS Nodes on anyway. Obviously you already have a solution from @paul121, but this may serve as a template for anyone else looking for similar, perhaps a link to the python script you are using may be useful too so there are a few options on here? (minus any hardcoded credentials etc.)
I’ve saved the Node Red flow here in json form: (Honestly its simpler than it looks )
When imported will look like:
To keep it simple(-ish) for understanding, I’ve split the task up into separate flow segments.
Authentication - Authenticates and retrieves taxonomy and land UUID’s
Create asset/Store CSV file - Reads the CSV, stores all fields and Creates the Assets.
Create Quantities - Quantities need to be created before the logs
Create Logs
Disclaimer: This flow is far from complete and could do with some additional error checks including possible duplication of logs/assets. I’d recommend anyone trying it to have a test environment that can be easily reset. Also the sections can be linked up but could probably do with a little additional logic to ensure each section is complete before the next starts to avoid sending too many http requests at once.
3 Likes
Wow -for a simple cattle man, you got pretty deep into the plant-based data model here, @Farmer-Ed. I appreciate your taking such an interest in this UseCase!
Yes: as soon as i get a bit of time away from mission-critical stuff here on farm, i will clean up and share those scripts we’ve got running in some form (either gColab or Jupyter Notebook) that i hope will be useful to others.
In fact Ed: this way of modelling dataflows appears to me beautifully simple, on the face of it, though it must be hiding a fair bit of complexity… And if the user can run these scripts without ever having to see need should arise), then that sounds like a very good thing!
2 Likes
Well thank you very much for the sample data it was good to test the nodes I built are flexible enough for different uses than my own. I think they did Ok but also helped me see a few areas for tweaking.
I think it would be good to have a collection of options here, at least until the new tutorials site is launched anyway. I’m sure this question will come up again and again, at least until a CSV module is made, although it may be a hard task to have a simple CSV importer with the flexibility of the API.
It is very nice that a lot of common tasks are available as drag and drop blocks already such as the CSV nodes, there is good and bad to the complexity being hidden away though as sometimes you can be constrained to the inputs and outputs available to a particular node. I was considering building a few more farmOS specific nodes to simplify handling CSV’s even more but there is a danger of losing some flexibility but maybe no harm if there were enough use case templates included.
1 Like
About a year into my Node-Red / farmOS experimenting and about to do some tidying up on my GitHub Repo and I came across this flow based on @walt’s transplantings. It works but it’s not very efficient in how it deals with quantities , I was in 2 minds about deleting it or fixing it.
Having recently figured out subrequests while working on a soil samples CSV importer with @pat which has 11 quantities per log, I decided to update the transplanting flow too, so every log including quantities is created in a single request.
Both CSV upload templates can be found in my repo and I will add a few more after tidying them up a bit.
They are obviously Node-Red flows, but could be relevant for any programing language, just spin up a Node-Red docker image and import to view. I’ve also included Subrequest Code below:
Create Quantities + Log:
//Declare variables
var unit = "m"
//Declare Variables from CSV
var length = msg.payload["length FB"];
var nr = msg.payload["Nr plants FB"];
var name = msg.payload.Name;
var description = msg.payload.Description;
var land = msg.payload["Area/Location"];
//Get flow Variables
var unit_terms = flow.get("unit_terms");
var plant_uuid = flow.get("plant_uuid");
var land_uuid = flow.get("land_uuid");
//Declare empty variables
var plant_id = null;
var land_id = null;
var unit_id = null;
//Generate UUID's
var Luuid = uuid.v1();
var Nruuid = uuid.v1();
//search for plant uuid
plant_uuid.forEach(data => {
if (data.name == name) {
plant_id = data.id;
}
})
//Search for land uuid
land_uuid.forEach(data => {
if (data.name == land) {
land_id = data.id;
}
})
//Search for plant term uuid
unit_terms.forEach(data => {
if (data.name == unit) {
unit_id = data.id;
}
})
if (plant_id != null && land_id != null && unit_id != null) {
var numerator = length;
var denominator = 1;
while (numerator % 1) numerator *= 10;
denominator = numerator / length;
//Create Quntity
var len = {
"data": {
"type": "quantity--standard",
"id": Luuid,
"attributes": {
"measure": "length",
"value": {
"numerator":numerator,
"denominator":denominator
},
"label": "length FB"
},
"relationships": {
"units": {
"data": {
"type": "taxonomy_term--unit",
"id": unit_id
}
}
}
}
}
msg.length = JSON.stringify(len);
//Create Quantity
var nrp = {
"data": {
"type": "quantity--standard",
"id": Nruuid,
"attributes": {
"measure": "count",
"value": {
"numerator": nr,
"denominator": 1
},
"label": "Nr plants FB"
}
}
}
msg.nr = JSON.stringify(nrp);
//Create Asset Object
var log = {
"data": {
"type": "log--transplanting",
"attributes": {
"data": name,
"status": "done",
"notes": {
"value": description
},
"flag": [],
"is_movement": true
},
"relationships": {
"location": {
"data": [
{
"type": "asset--land",
"id": land_id
}
]
},
"asset": {
"data": [
{
"type": "asset--plant",
"id": plant_id
}
]
},
"quantity": {
"data": [
{
"type": "quantity--standard",
"id": Luuid,
},
{
"type": "quantity--standard",
"id": Nruuid,
}
]
}
}
}
}
msg.log = JSON.stringify(log);
}
return msg;
Combine in a single Subrequest:
var token = global.get("token");
msg.payload = [
{
"requestId": "req-1",
"uri": "/api/quantity/standard",
"action": "create",
"body": msg.length,
"headers": {
"Accept": "application/vnd.api+json",
"Content-Type": "application/vnd.api+json",
"Authorization": "Bearer " + token
}
},
{
"requestId": "req-2",
"uri": "/api/quantity/standard",
"action": "create",
"body": msg.nr,
"headers": {
"Accept": "application/vnd.api+json",
"Content-Type": "application/vnd.api+json",
"Authorization": "Bearer " + token
}
},
{
"requestId": "req-3",
"waitFor": ["req-1","req-2"],
"uri": "/api/log/transplanting",
"action": "create",
"body": msg.log,
"headers": {
"Accept": "application/vnd.api+json",
"Content-Type": "application/vnd.api+json",
"Authorization": "Bearer " + token
}
}
]
msg.headers = {};
msg.headers['Authorization'] = "Bearer " + token;
msg.headers['Accept'] = "application/vnd.api+json";
msg.headers['content-type'] = 'application/vnd.api+json';
msg.url = global.get('farmOSurl').concat('/subrequests?_format=json')
msg.method = "POST";
return msg;
2 Likes
I’m glad you sorted out the subrequests. It makes some stuff more effective, and the flows will look more tidy.
2 Likes