At Rothamsted we regularly want to download all the data associated with an asset (plants, fields but also experiments). Simplifying this data export is probably one of the biggest bug bears with our user community across both the farm and research teams.
At Rothamsted we have two main issues with the current ‘export CSV’ and ‘export quantities CSV’ options that are currently listed in FarmOS core.
- The logs all have different headers, so we need to do manual exports for each log type in order to extract all the data.
- For a similar reason, we can’t easily extract material quantities that include the material type for a large number of assets (for us this is most challenging when it comes to experiments, which can have between 20 and 2000 plots, and we want the quantities associated with all plots).
@paul121 and I were talking about this yesterday, and how we might improve the UI for exporting logs and quantities from these assets. One idea we had is to create a single ‘export data’ option from the Actions dropdown list at the bottom of each asset type, replacing the current ‘export CSV’ and ‘export quantities CSV’ options that are currently in FarmOS core.
In it’s simplest form, you’d filter to find the logs that you want and then select “Extract Data” from the actions dropdown.
This ‘Extract Data’ button would open a new screen where you can choose what you want to extract. At Rothamsted we’d want a minimum of two options:
1. Export logs: at Rothamsted we would want the export to provide multiple csv’s separated by type, so you get one csv for operations, one for harvests, one for inputs, etc. This gets around the issue of different headers in the csv.
2. Export Quantities: again you get two csv’s, one for standard quantities and one for material quantities. We don’t use test quantities because most of our lab test data lives in electronic research notebooks (ERN’s) but theoretically if there were test quantities, you’d get a third csv.
Depending which of the two options you choose, you get to select which columns you want from each export type but with more granularity (e.g. you can be specific about which columns you want from an input log and those can be different from what you want in the harvest logs - so basically fully customisable).
@paul121 pointed out that this comes very close to what you want in a reporting module, where you might want to save those settings so you can apply them multiple times over either to the same asset(s) or different assets. I think that is very much a secondary phase, but I thought I’d mention it here because it’s worth thinking about these things in that context.
We have to get a first version of this built for Rothamsted before our grant funding ends in March, but I’d love comments from the community more broadly so we can try build something relevant to all that might later be incorporated into FarmOS core at a later date (if we had more time I’d be happy to open this as a Core request, but our funder deadline in March makes it difficult to the full community scoping work before we run out of money).
One last point: data that I am have left out of this initial spec but which others might like to comment on:
- exports of csv’s associated with different asset types (e.g. if you want to download all the data associated with plant assets and/or land assets). Currently we are pretty happy with this functionality in FarmOS core.
- exports of experiment metadata. The Rothamsted experiment module is not currently compatible with FarmOS core, although we are applying for funding to take that next step and being able to export experiment metadata would be part of that later grant work.