Are there limits to amount of data that can be uploaded from sensors?
I have data uploading to a sensor that is 8 different measurements per timestamp in 10 minute increments. After a certain point ( not sure when) every other timestamp starts disappearing.
The first time I saw it I was only seeing the :00, :20, and :40 minute points. So i created a new sensor, started uploading the same data (historical). I checked the first 2 points, which were a :00 and :10 minute points, all good. So i let it keep uploading data and eventually I was left with the :10, :30 and :50 minute points for all hours. Even the very first point was gone.
Huh - not sure what would cause that! There isn’t any limit on how much data can be sent/stored (apart from the rate limit of 1 post per sensor per minute on Farmier hosting).
Are you able to see any response code errors from the requests you’re sending?
If you want to try to find a time when we’re both online, I could take a look at logs on my end and we can see if we can pinpoint it. Ping me in the farmOS chat!
Thanks for offer @mstenta. I will try a couple more things, because i am seeing its not as consistent as i thought, what points stay and which get lost. I am taking my script and modifying it to look at all timestamps individually in my record and upload if they are not present. I want to see if after multiple executions if data continues to disappear or not.
I did see that every data point i pushed did make it to the server, but then eventually it would disappear.
So i adjusted my script to check each timestamp locally to see if it exists on the server and if not to then push the local data for that timestamp to the server, instead of just checking the latest like i was before. Something interesting i am seeing is that with each run i do, every other data point that was previously uploaded needs to be reuploaded. So on the first run lets say i upload 100 timestamps. On the next run, i need to upload every other data point. Then the next run every 4th data point needs to reuploaded, then every 8th, then every 16th, and so on until they are all present. Its very strange.
But if i checked that data point immediately after uploading, to see if it was present, it would be.
Farmier hosting enforces a limit of 1 POST per endpoint per minute.
Your script.
I’d like to understand a bit more how your script is working…
Is it POSTing a single request with 1 timestamp and 8 values?
How often is it sending a POST? If it is trying to send one every 60 seconds, this may be too close for the Farmier-imposed limit (#2 above), which could explain why every other one is dropped.
I did see that every data point i pushed did make it to the server, but then eventually it would disappear.
Are you saying that data was on the server, and then it was lost?? Or are you saying that over time not all points make it to the server? I was a bit confused by your descriptions.
@applecreekacres Looking at Farmier server logs I do see a LOT of data points hitting the 1 minute limit for your instance. So that is the issue I think.
How often is it sending a POST ? If it is trying to send one every 60 seconds, this may be too close for the Farmier-imposed limit (#2 above), which could explain why every other one is dropped.
This might be the most important question to figure this out ^
@mstenta i will adjust my timer to go to 70 seconds between uploads instead of 60. I was seeing data points visible on the server and then eventually they disappeared. I have not been able to properly diagnose when that disappearance occurs though.
One thing i am going to do to is at one point i was seeing only certain named values from a particular data point coming back on the API read but i could see them on the FarmOS page.
It makes sense that “60 seconds” would be a bit error prone like this - simply due to the slight differences in server clocks. There is not such thing as perfect precision.
at one point i was seeing only certain named values from a particular data point coming back on the API read but i could see them on the FarmOS page.
Oh interesting - well keep an eye on this! It’s important to note that the code that generates the list of data in the farmOS UI is different from the code that generates the data that is returned by the endpoint via GET request. So if you are seeing differences that could indicate a bug in one or the other (more likely in the API endpoint since that is custom code, whereas the UI is generated by Drupal Views, which is just a simple database query).
Let me know if you notice any inconsistencies there moving forward!
Thanks @mstenta. Do the Farmier hosted instances have the ability to delete data via the API? Since i am guessing we wouldn’t be able to hook to the SQL database?
The reason i ask is that some of my data points doubled up for certain timestamps because if the discrepancy in uploads. So for one timestamp i get back 4 instances of the same named value. It would be nice to clear that out to save space and not have to worry about it when i pull data out in scripts.
farmOS doesn’t have an API method for deleting data points, no (and so neither do Farmier-hosted instances). That could be a good feature request!
Alternatively, if you delete a sensor asset, it will delete all data associated with it in the database. So one option would be: once you know everything is working, delete the sensor assets you were testing with and start fresh.
How does the throttling work? If it’s using a standard throttling mechanism perhaps the burst rate can be set to 2/minute and the 5 or 10 minute average set to 1/minute? That would allow for minor deviations such as those from latency variations while still enforcing the same overall rate…
How does the throttling work? If it’s using a standard throttling mechanism perhaps the burst rate can be set to 2/minute and the 5 or 10 minute average set to 1/minute?
That’s a great idea @Symbioquine - I use Nginx’s limit_req directive, so adding the burst option to that could solve this. I will look into that.
I had considered that. I had actually already created a second instance to test out my script so I may do that again now that it has been working for me.
