Mapping issue: Area classification

Development
1

I don’t know if this is a Development issue, properly speaking, but since @mstenta flagged it as such a couple of months ago, i thought this might be the context in which to bring it up again.

To put a finer point on the issue: I want to be able to view landplots in farmOS as filtered by certain classification criteria, such as land-use, irrigation regime, management, etc. I’ve been managing such info in QGIS, but the only connection i’ve been able to make between these two systems is KML export from QGIS -import of which files is how i set up areas in farmOS initially, but that is quite a limited format: really just geolocated vector maps with a Title and Description defined.

In Mike’s summarisation of the issue linked above, he proposed adding a field for land-use to the farmOS schema, but- given that (a) there are a good few other attributes of a landplot that we also are relevant to farmOS application; and (b) that farmOS is not now, nor does it ever aim to be a GIS- my question is: what is the best way to get richer information about landplots that i manage in QGIS to show up in farmOS?

A related issue which i brought up in sidebar chat on last week’s Developer call (thinking it might be relevant to the Mapping issues on @Symbioquine 's agenda, but those were very specific outstanding PRs, i now realise; sorry tor the distraction!) is how to replace the raster basemap image that is farmOS default with more useful imagery that we may have. For example, i’ve got a better (i.e. more current and granular) GeoTIFF from recent aerial drone survey that i’d like to use, but it is 2.5+ gb, and not (yet) online accessible. Did a bit of digging into that COG technology that @paul121 mentioned, but it would involve having a GeoServer instance online that i don’t yet have… But if there’s a low-cost way of bringing such a pipeline for getting both vector AND raster data from my desktop GIS into farmOS, i would love to know about it!

1 Like
2

Hi Walt, these are great questions!

There are a couple viable strategies which I’ll classify in terms of where the data is stored;

  1. farmOS only: In this strategy all data is stored in farmOS. Where you need to store asset/geometry/attribute types which don’t already exist in the farmOS data model, these would need to be added via custom farmOS modules. The data can still be accessed/modified via My WFS module.
  2. farmOS + GIS hybrid: In this strategy the core definitions/geometries of assets are stored in farmOS and additional data that is only needed on the GIS side is stored outside farmOS. My WFS module could be used to bring the assets into QGIS and then QGIS features such as joins or auxiliary data tables could be used to associate the additional data. This has the advantage that it can be done in a point-and-click fashion without additional custom farmOS modules, but it can be fragile and has the downside that the data thus added would only be available on the GIS side.
  3. farmOS + GIS copied data: This strategy is essentially continuing doing what I think you’re doing now - copying certain data into farmOS and manually keeping them in sync. This requires no custom development and has the lowest up-front cost. This also has the advantage of a certain kind of redundancy, but has the cost of potentially introducing errors and higher maintenance cost.

For now, this requires a custom farmOS module to add the new “base layers”. (Such a module would look kind of like https://github.com/farmOS/farm_map_no) On the call we discussed the possibility of a more generic module being possible which would make this more of a point-and-click administrative task, but as far as I know, nobody has volunteered to pick up the dev work to make that a reality.

Actually, the point of the “Cloud Optimized GeoTIFF” (in my understanding) is that it does not require a special GIS server - like GeoServer. Instead, it is supposed to work over simpler file hosting channels - as long as they support certain relatively common HTTP request features.

In practice, this means that you might be able to host the files very simply via something like an AWS S3 bucket (Of course, things get more complicated if you don’t want the files publicly exposed on the internet, but that’s a different topic.) or even possibly uploading them to the same server where farmOS is running.

3 Likes
3

(To add to @Symbioquine’s great reply… I admit I wrote this before reading so apologies for some redundancy) :slight_smile:

@walt The way that I would accomplish this, if I were doing it for myself personally, would be to write a custom module that adds ALL the fields I want to Land assets, and then use farmOS exclusively for managing this data (that is assuming that name + geometry + these fields is the only thing you’re using QGIS for currently). This would be a pretty quick module to spin up as a proof-of-concept. I’d be happy to try that out with you on a dev call sometime, and install it on your 2.x testing instance for you to try out.

If you are able to host that GeoTIFF imagery somewhere (either a self-hosted GeoServer/MapServer or via a paid tile hosting service) then we could probably help you get that into the same custom module so it shows in your farmOS 2.x instance.

We also talked on the call about adding a general-purpose configuration UI to farmOS 2.x that would allow any hosted URL to be configured for inclusion in the map. This would be the best long-term solution IMO. But we can certainly get something up and running fast in your module as a first step.

Hosting is the big question mark though. I would probably recommend just looking for a paid layer hosting service to get yourself started quick. Maybe others have good recommendations for that - I’ve never explored it.

Agreed! And just to be clear for @walt this illustrates the two sides of the problem: 1) hosting and 2) adding it to the map. You might be able to just upload a GeoTIFF to an asset in farmOS (if that made sense), and that would might care of hosting it. But you would still need a module that says “the URL of the GeoTIFF is here, add it to the map”.

2 Likes
4

Thanks @Symbioquine & @mstenta for your engagement on this issue. I’ve also got email from @paul121 in which he set me straight on my misconceptions about COGs -that they are much more liberal in what they require of a server is great!- and you’ve all helped me to better understand important realities about this.

So, in view of all this, i’d like to explore further the strategy that Morgan described as #1 above -which, when paired with his WFS Module makes all relevant data available on both sides of the farmOS | QGIS divide.

@walt The way that I would accomplish this, if I were doing it for myself personally, would be to write a custom module that adds ALL the fields I want to Land assets, and then use farmOS exclusively for managing this data (that is assuming that name + geometry + these fields is the only thing you’re using QGIS for currently). This would be a pretty quick module to spin up as a proof-of-concept. I’d be happy to try that out with you on a dev call sometime, and install it on your 2.x testing instance for you to try out.

Excellent! I will put some thought into new land-asset fields we’d be wanting to add to these vector objects, and send them to you directly, Mike.

Now as to the raster side of this equation:

YES! If raster data could be hosted on my (Farmier) instance, that would completely obviate the need for any trafficking of image data via internet in excess of what the server is already trafficking with the default map image. I can do the transformation of this ~2.5 gb GeoTIFF into tiles, using affordances built right into QGIS, if that helps… Presuming that the hosting proposition is in principle agreeable to you, which i gathered from your response to Morgan:

Agreed! And just to be clear for @walt this illustrates the two sides of the problem: 1) hosting and 2) adding it to the map. You might be able to just upload a GeoTIFF to an asset in farmOS (if that made sense), and that would might care of hosting it. But you would still need a module that says “the URL of the GeoTIFF is here, add it to the map”.

So… This needs a custom module to be written? Would that be just for me, or is it something that others would also be using? If the former, then i would ask if you can write it for me and on what terms… And if there’s any additional charges related to bandwidth (not much i expect, as mine will not be a very hi-volume app), then i expect we can work something out.

SO: tho farmOS may still lay no claim to the title of GIS, it is beginning to sound like it can serve that subset of functions that is really needed for farm management -an interesting prospect!

2 Likes
5

Farmier has a file upload size limit of 100 mb, so I don’t think this is a feasible approach. I’m not set up to offer hosting of large image sets like this. I do not host the MapBox or Google satellite imagery on my servers - these come from MapBox/Google servers, and are just referenced by the map layer in farmOS. And technically I have API keys with those providers that measure how much traffic is used and I pay extra if I go over their (generous) free levels. Hosting large imagery directly on my server would be a different set of calculus for me, which I would need to think about with regard to pricing etc. In all honesty I’m not interested in exploring that, since there are so many other options available already, and it would be another thing I need to manage.

So if you are working with imagery that large then perhaps S3 would be the best option. If you’re unfamiliar, S3 is Amazon AWS’s “simple storage service” which is aimed specifically at hosting static files that do not change frequently (like images). You would pay Amazon for the cost of storage and traffic directly.

1 Like
6

Yeah, makes sense that these files could be too much. But do you think GeoTIFFs that fit the upload size limit would be OK? Obviously this may be more “imagery traffic” than you get now (none? lol) but COGs are optimized so the browser only requests what it needs. Paired with browser caching it might be pretty minimal? (these data provider docs are kinda relevant: Provider's Guide to COG)

I imagine @walt’s file is so large because it’s high resolution. I’m not sure, but I think removing some “X% of the resolution” might reduce file size by more than X% (if that makes sense?) So getting a 100MB file (that is still better than satellite) might be feasible.

Out of curiosity I checked sentinel-2 specs: each tile is a 100km x 100km square @ ~600MB. Most don’t need the entire tile though, so once it’s clipped to even something like 100km2, the file would likely be < 10MB.

7

Yea if the files are reasonably sized and caching is figured out (to reduce the traffic footprint), then I’d definitely be open to offering something like the “Imagery asset” you proposed on Farmier!

Didn’t mean to sound like I was shooting down any ideas! I just have a different set of considerations for hosting/managing lots of farmOS instances. :slight_smile:

3 Likes
8

Am very happy to hear this, @mstenta ! :+1:

Of course Mike: this is why you get the big bucks for hosting service :laughing:

Seriously tho: I only jumped on that possibility when @Symbioquine mentioned it, because i gathered (perhaps mistakenly? *) that COGs tiling tech was enough of a game-changer, the diff in bandwidth vs status quo on server side would be minimal… And to whatever extent that is not the case, it’s only fair that client should pay the diff (including not just bandwidth, but admin/ management overhead).

Also: this is in fact a super-hi res GeoTIFF i’m talking about, suitable for A1 printout, which could i guess be reduced as @paul121 suggests without affecting the level of detail at screen resolution appreciably. I don’t know much about this stuff, but i would be happy to work w/ y’all on it!

1 Like
9

What is the max. caching size allowed for the Field-kit for when is going off-line and can’t stream map data? (for us, this is crucial for grazing hill paddocks in which there is no reception)

1 Like