Farm_input_file_nnnnnnnn.dat

Using farmOS
1

On one of my input log entries I attached a number of files for OMRI certificates, invoices, material tags, and other info related to the input applied. I opened the record and all of the attachments seem to have disappeared. All I see are entries "farm_input_file_ (a long number) .dat with a 0 byte file size. What happened? Did I forget to hit save or something?

(By the way… The UI needs to have a SAVE reminder so a user doesn’t exit the data entry process and record without saving. Adding every file is a multi step process of selecting and uploading. The save button frequently is not visible off the bottom of the screen. very annoying to leave the screen and realize everything needs to be redone because I missed the save button.)

1 Like
2

Hi @graffte - Huh… if you try re-uploading the files does it work?

Trying to think what might cause this…

What types of files are they? farmOS has a set of “allowed” file extensions… but it would warn you if you tried to upload something that wasn’t in that set, so I doubt that’s the issue…

With regard to the “Save” button… if you didn’t click “Save” at the bottom of the log itself, then it wouldn’t save anything at all - there would be no files associated with the log afterwards. The “Upload” button next to the file upload field will handle uploading the file and then the “Save” button will save the log record, along with the references to the uploaded files. If you clicked “Upload”, but not “Save”, then files would get stored on the server, but they would be marked as “temporary” in the database table that manages files. The “Save” button on the log is what marks them as “permanent” and creates the link to the log itself. This allows “temporary” (orphaned) uploaded files to be automatically cleaned up/deleted. But that’s not what happened here… because you wouldn’t see any files associated with the log at all… (just talking this through as I try to figure out what might be happening here…)

The UI needs to have a SAVE reminder so a user doesn’t exit the data entry process and record without saving

I agree! This actually came up as a request a long time ago, now that I think about it, but never happened. I would support adding a simple JavaScript to prompt the user to confirm that they are navigating away from an unsaved form. This is pretty common in other form-based websites… I say we open a feature request for this and add it.

Getting back to the issue at hand though… the file name itself is curious: farm_input_file_X.dat

I assume you did not name the file that yourself? Also curious because .data is not one of the “allowed” file types. So I’m a bit perplexed as to what’s going on.

Curious: how big are the files you are trying to upload? There is a limit to how many megabytes can be transferred in a single request (currently 50 mb on Farmier-hosted instances). But that shouldn’t be an issue if you are uploading them one at a time through the web UI (because each file upload is processed in a separate request), assuming each individual file is under the limit.

I’d be happy to take a closer look with you if you want to arrange a time to do so over Zoom or telephone?

3

The UI needs to have a SAVE reminder so a user doesn’t exit the data entry process and record without saving

Oh I found the old feature request for this: https://www.drupal.org/project/farm/issues/3022515

4

Ah ha! I did some digging and have a new theory…

Are you using Field Kit to upload these files?

I found that the restws_file module (which farmOS uses to process files uploaded via the API) has some fallback code in it that will result in a file named [log-type]_file_[timestamp].dat. This is the only code in farmOS that can result in that filename pattern (that I can find).

@graffte does that sound right? Were you uploading files through Field Kit or through the normal farmOS web UI?

I attached a number of files for OMRI certificates, invoices, material tags, and other info related to the input applied

Were these photos of documents? Or were they other file types like PDFs? Field Kit only supports photo uploads right now. And these photos get attached to the log’s “Photos” field, not its “Files” field. The “Photos” field only supports files ending in one of the following: png gif jpg jpeg

how big are the files you are trying to upload? There is a limit to how many megabytes can be transferred in a single request (currently 50 mb on Farmier-hosted instances). But that shouldn’t be an issue if you are uploading them one at a time through the web UI (because each file upload is processed in a separate request), assuming each individual file is under the limit.

This makes me realize… if you are using Field Kit, then this could also be an issue. I’m just remembering that we ran into this exact limitation in the past: Log not Synching to Database - #12 by mstenta

Field Kit needs some work on the file handling side of things, and that’s on our radar. Field Kit development has turned focus to farmOS 2.x, though, so that is likely where it will be fixed/improved.

If this does seem to be the cause of this issue for you, then at this point my best advice is to not use Field Kit for uploading files.

Instead, if you need to upload files from your phone, then I would recommend using farmOS directly in your phone’s web browser. It works the same way as it does on desktop, and will adapt to the screen size relatively well. farmOS itself will handle issue with the file uploads and provide feedback/warnings if anything goes wrong (this type of feedback still needs work in Field Kit).

The only real limitation to using farmOS directly on your phone is that it requires an internet connection. (This is one of the main limitations that we hope Field Kit aims to solve - but “offline-first” is not easy.)

Hope that all makes sense - and gets to the root of this issue for you! And hopefully it won’t be too hard to re-upload the files in question. Let me know if I can help or provide any more guidance!

5

Yes, I use field kit. I don’t recall if I used it for this entry. Might have entered a quick log entry through the field kit and used my PC to attach the pdf document files later. I definitely did not use field kit to upload the files.

6

Ah ok… then maybe it was a combination!

I found that the restws_file module (which farmOS uses to process files uploaded via the API) has some fallback code in it that will result in a file named [log-type]_file_[timestamp].dat . This is the only code in farmOS that can result in that filename pattern (that I can find).

The only way that the Farm_input_file_nnnnnnnn.dat can possibly be created is via API (Field Kit). So those were probably created from photos uploaded via Field Kit…

Did I forget to hit save or something?

And maybe you forgot to save for the other files (uploaded via desktop)?

That combination of events would explain the symptoms. Does that sound plausible?

7

This is the log entry in question.
https://rightmindag.farmos.net/log/159

There is one photo in the log entry which means I definitely used the Field Kit to make the original log entry. I was in the field when I noticed the tracks confirming the fertilizer had been spread.

I added the pdf files to the log entry this week when I saw the files I thought I had entered were missing. The .dat files with 0 bytes are the ones that I would have entered from my PC previously and where I may have forgotten to hit SAVE.

So seems to agree with your hypothesis that log was made by Field Kit and that I neglected to SAVE when I added the pdf files to the existing log entry causing the uploaded but not saved files to end up with the farm_input_file_x.dat and zero data.

1 Like
8

Thanks for sharing that @graffte ! I think we’re honing in on a plausible explanation. :slight_smile:

There’s still one thing that confuses me… those *.dat files are attached to the “Files” field, not the “Photos” field. But Field Kit only has the ability to upload to the “Photos” field currently… So I’m still perplexed as to how those files were created.

I found that the restws_file module (which farmOS uses to process files uploaded via the API) has some fallback code in it that will result in a file named [log-type]_file_[timestamp].dat. This is the only code in farmOS that can result in that filename pattern (that I can find).

Here is the line of code in the restws_file module that creates that naming pattern: restws_file.module · 3be9c228f1187c460a9516e3aad5da7632871a9a · project / restws_file · GitLab

And below that you can see that it also logs a notice when it’s unable to figure out the type of file (and therefore results in .dat extension):

That restws_file code is only used to upload files via the API (and in your case I assume Field Kit is the only app that’s using the API (right?)).

I checked the Drupal logs for your farmOS and sure enough that notice is in there referencing log 159. So those files were definitely created via the API request.

9

Ah ha! I talked a bit with @jgaehring and we have a new hypothesis (a possible bug with Field Kit and/or the farmOS API):

If a log already has files attached to it, and you load it into Field Kit, make some changes (to other data on the log), then sync it back, it results in these *.dat files being created.

I haven’t tested this yet… that’s just the hypothesis… I’ll try to test this myself when I have a chance - please feel free to do the same @graffte if you are able!

10

Oof. OK I just tested this and that’s definitely what’s happening. And it’s worse that I expected. When a log that has files attached to it is loaded into Field Kit, and then synced back to the server, all files get overwritten.

This explains everything you experienced @graffte - You didn’t forget to click save.

I created a bug report: Files attached to logs get overwritten on sync · Issue #444 · farmOS/farmOS-client · GitHub

@graffte Do not use Field Kit to sync logs again until we get this fixed…

1 Like
11

Hey everyone, I think we’ve got this fixed. I just rolled out the change to v1.farmos.app as version 1.0.1, so if you get prompted to install the update, please do, and let us know if you’re still experiencing any issues related to this bug.

I’ve posted a little further explanation here:

2 Likes
12

I added the Files (pdf) from my PC to the Field Kit created log. The Field Kit created log would have had a picture or two. So no I have not entered Files in Field Kit only Photos (from camera).

1 Like
13

@graffte - Makes sense!

One thing to note about the v1.0.1 release that “fixes” this bug… it actually is a bit of a temporary “workaround” to prevent the issue from happening, but in order to do so it adds a new limitation: you can only add photos to NEW logs in Field Kit (logs that you create in Field Kit). Once a log is synced to Field Kit (or if a log is pulled in from the server), it will no longer allow you to add photos to it in Field Kit.

This was the quickest way to get a fix out to prevent any further data loss (for you and everyone else who might not be following this thread). We are planning to overhaul how photos/files are managed in Field Kit version 2, which @jgaehring is working on. So that is most likely when the full functionality will be restored.

This means you may need to adapt your strategy for using Field Kit in the meantime… If you want to use Field Kit to add photos to logs, you need to make sure that those logs are created FIRST in Field Kit, and not synced to the server until you are done adding photos to them.

This may disrupt your current workflow - so I wanted you to be aware of it so you can adapt accordingly.

It is worth mentioning that you can also use farmOS directly in your phone browser to create logs and upload photos - without any of these limitations. So you may find that to be a more flexible solution in the short term. It does require internet access, of course, but if you take the photos on your phone, then get back to wifi range, you can create logs + upload photos directly from your phone, without needing Field Kit at all.

Just want to be sure you know all the options and considerations. Hope that helps! :slight_smile: