# Proposal for farmOS 3.x core datamodel improvements: Immutable Quantities & Soft Entity Deletion **URL:** \ **Category:** Development\ **Tags:** farmos-enhancement-p\ **Created:** [January 22, 2023, 2:03am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490 "2023-01-22T02:03:15Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\ **Post date:** [January 22, 2023, 2:03am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/1 "2023-01-22T02:03:15Z")
I wanted to open this thread to test the waters for a couple of (I feel) key improvements to the core farmOS datamodel that should be considered for a version 3 (or 4?) of farmOS. ## Specifically: ### Immutable Quantities Make quantities immutable and enforce that a single quantity is referenced by at most one log. **Rational:** This makes changes to quantities more atomic and guarantees that the log’s `changed` timestamp is always updated when one or more of its quantities are changed. ### Soft Entity Deletion Make entity deletion an admin only action and provide a soft-delete (a.k.a. [tombstone](https://en.wikipedia.org/wiki/Tombstone_(data_store))) mechanism for regular users (`farm_manager`, `farm_worker`). The logical choice here seems to be introducing `deleted` as a new `status` value - in addition to `active` and `archived`. (I’m open to other ideas - such as a dedicated boolean field too.) **Rational:** This makes it easier for API clients to determine why an entity no longer “exists” - i.e. replication issue, database rollback, actual deletion, etc. It also makes data-loss easier to avoid in the case of a rogue user - i.e. rather than needing to get the data from a recent backup, it just becomes a matter of rolling the entity/entities back to an earlier - not “deleted” - revision. ## Next Steps: On a recent call I remember we talked about maybe having a more formal proposal mechanism - like [PEP](https://peps.python.org/pep-0000/). “farmOS Enhancement Proposals” (FEP) anyone 🤓? I’d be happy to pilot such a mechanism if there’s interest… regardless of that I’d welcome any/all feedback on the above proposals. ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [January 22, 2023, 2:53pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/2 "2023-01-22T14:53:19Z")
Love both of these proposals @Symbioquine! Thanks for getting the conversation started. 🙂 > [@Symbioquine](#): > > On a recent call I remember we talked about maybe having a more formal proposal mechanism - like [PEP](https://peps.python.org/pep-0000/). “farmOS Enhancement Proposals” (FEP) anyone 🤓? Yes!! 😄 At the risk of [bikeshedding](https://en.wiktionary.org/wiki/bikeshedding)… “FIP” is another option (“improvement” vs “enhancement”, eg: [BIPs](https://github.com/bitcoin/bips)). Either way! I love the idea of formalizing this process for core data model changes moving forward. > [@Symbioquine](#): > > Make quantities immutable and enforce that a single quantity is referenced by at most one log. This might allow us to remove our dependency on [Entity Reference Revisions](https://www.drupal.org/project/entity_reference_revisions), and the extra complexity (eg: [Issue #3267304](https://www.drupal.org/project/entity_reference_revisions/issues/3267304)) that it adds for API requests. Feels like there will be a lot of code considerations with this change… curious to start thinking it all through… > [@Symbioquine](#): > > Make entity deletion an admin only action and provide a soft-delete (a.k.a. [tombstone](https://en.wikipedia.org/wiki/Tombstone_(data_store))) mechanism for regular users This has been something I’ve wanted to see for a long time. 👍 It will also require a fair bit of thought and extra logic to hide “deleted” status entities throughout the UI (and API?) ---
**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\ **Post date:** [January 24, 2023, 4:29pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/3 "2023-01-24T16:29:14Z")
Thanks for the enthusiasm! > [@mstenta](#): > > Feels like there will be a lot of code considerations with this change… curious to start thinking it all through… Agreed > [@mstenta](#): > > It will also require a fair bit of thought and extra logic to hide “deleted” status entities throughout the UI (and API?) Yeah, I’m thinking it might not be too bad if we use the permission system… > [@mstenta](#): > > At the risk of [bikeshedding](https://en.wiktionary.org/wiki/bikeshedding)… “FIP” is another option (“improvement” vs “enhancement”, eg: [BIPs](https://github.com/bitcoin/bips)). Either way! I love the idea of formalizing this process for core data model changes moving forward. Okay, I’ll bite… - **GIIFT** : **G** reat **I** dea to **I** mprove **f** armOS **T** oday - **FIELD** : **f** armOS **I** mmediate **E** nhancement **L** ightweight ( **D** esign| **D** raft) - **FEAR** : **f** armOS **E** nhancement **A** ction **R** ecommendation - **FBI** : **f** armOS **B** etterment **I** dea ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [January 24, 2023, 4:31pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/4 "2023-01-24T16:31:39Z")
> [@Symbioquine](#): > > Okay, I’ll bite… > > - **GIIFT** : **G** reat **I** dea to **I** mprove **f** armOS **T** oday > - **FIELD** : **f** armOS **I** mmediate **E** nhancement **L** ightweight ( **D** esign| **D** raft) > - **FEAR** : **f** armOS **E** nhancement **A** ction **R** ecommendation > - **FBI** : **f** armOS **B** etterment **I** dea I think we’ll need to start a commission to decide what makes the most sense. /s 🤣 ---
**Author:** ![paul121](https://avatars.discourse-cdn.com/v4/letter/p/54ee81/32.png) [@paul121](https://farmos.discourse.group/u/paul121)\ **Post date:** [January 25, 2023, 5:58pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/5 "2023-01-25T17:58:16Z")
> [@mstenta](#): > > This might allow us to remove our dependency on [Entity Reference Revisions](https://www.drupal.org/project/entity_reference_revisions), and the extra complexity (eg: [Issue #3267304](https://www.drupal.org/project/entity_reference_revisions/issues/3267304)) that it adds for API requests. Curious to understand how this helps with complexity… removing ERR would be nice, but do we create something more complex in its place? Right now to modify a quantity you must update the `log.quantity` `revision_id` after modifying the quantity entity. With this proposal you would need to create a new quantity as a “copy” of the “old” quantity with intended changes, remove the old quantity from the `log.quantity` field, and add the new quantity to `log.quantity`. Does that sound right? ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [January 25, 2023, 9:57pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/6 "2023-01-25T21:57:55Z")
> [@paul121](#): > > Curious to understand how this helps with complexity… removing ERR would be nice, but do we create something more complex in its place? I was only referring to the extra “API surface complexity” that the ERR module adds by requiring the revision ID be specified in addition to the entity ID. That is an outlier in terms of the rest of our data model. Only requiring the entity ID would bring it in line with everything else, but there’s a lot more complexity added in other code to enforce immutability I agree. Whether or not that’s justified is up for discussion. ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [January 31, 2023, 10:20pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/7 "2023-01-31T22:20:56Z")
Quick note for future reference/detail: @paul121 and I were discussing in chat about some boilerplate PHP methods on the Asset/Log/(others?) entity type classes like `getName()` and `getOwner()` that might be worth deprecating in 2.x and removing in 3.x. They were created initially because it’s standard practice in custom Drupal entity types, but they aren’t really necessary and aren’t really justified - we could reduce the API surface a bit by removing them. Plus the `getOwner()` field is misleading… it returns the `uid` of the entity, not the `owner` field values that we add via the `farm_owner` module. ---
**Author:** ![Symbioquine](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/symbioquine/32/284_2.png) [@Symbioquine](https://farmos.discourse.group/u/Symbioquine)\ **Post date:** [February 4, 2023, 1:14am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/8 "2023-02-04T01:14:19Z")
> [@paul121](#): > > With this proposal you would need to create a new quantity as a “copy” of the “old” quantity with intended changes, remove the old quantity from the `log.quantity` field, and add the new quantity to `log.quantity`. Does that sound right? That’s definitely what I was suggesting - though typically some convenience mechanism would be provided in the write parts of the API to simplify doing so. > [@paul121](#): > > Right now to modify a quantity you must update the `log.quantity` `revision_id` after modifying the quantity entity. Thanks for pointing this out @paul121! I’ll need to play with it a bit more, but I think this actually may address most of my motivation for that part of this proposal. Assuming: - That this guarantees that all well-formed quantity views/calculations will never see a partially modified set of quantities - That this guarantees that the log `changed` timestamp will always be updated when any quantity associated with the log is changed Given the above, I think the only bit of this proposal regarding quantities left would be to enforce that a single quantity entity can’t be re-used for multiple logs. > [@mstenta](#): > > I was only referring to the extra “API surface complexity” that the ERR module adds by requiring the revision ID be specified in addition to the entity ID. That is an outlier in terms of the rest of our data model. Only requiring the entity ID would bring it in line with everything else, but there’s a lot more complexity added in other code to enforce immutability I agree. Whether or not that’s justified is up for discussion. Yeah, I’m not stuck on it. There’s a lot to be said for leveraging existing platform patterns, even if we might build it differently ourselves otherwise. ---
**Author:** ![rmattb](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rmattb](https://farmos.discourse.group/u/rmattb)\ **Post date:** [February 11, 2023, 2:23am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/9 "2023-02-11T02:23:07Z")
Is farmOS currently making use of the Published boolean for Quantities? > **[Create EntityPublishedInterface and use for Node and Comment](https://www.drupal.org/node/2789315)** > > Problem/Motivation In \[#2810381\] we introduced EntityPublishedTrait, however this does not yet have a common interface. Proposed resolution - Add \\Drupal\\Core\\Entity\\EntityPublishedInterface - Add a published entity key - Only add the base field via... ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [February 12, 2023, 12:55pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/10 "2023-02-12T12:55:55Z")
> [@rmattb](#): > > Is farmOS currently making use of the Published boolean for Quantities? No, quantities are their own entity type, and they don’t have a concept of “published”. farmOS doesn’t use Drupal node or comment entities for anything currently. The available quantity attributes and relationships are described here: [Quantities | farmOS](https://farmos.org/model/type/quantity) ---
**Author:** ![rmattb](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rmattb](https://farmos.discourse.group/u/rmattb)\ **Post date:** [February 13, 2023, 5:47am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/11 "2023-02-13T05:47:34Z")
I suggest that if something is not removed from the database, it should not be called “deleted” - brainstorming around alternatives: deprecated archived unpublished shelved retracted retired internal private ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [March 11, 2023, 7:57pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/12 "2023-03-11T19:57:33Z")
FYI I started a meta issue to track all the breaking changes we might want to propose/consider for farmOS 3.x: [[META] farmOS 3.x Breaking Changes [#3347412] | Drupal.org](https://www.drupal.org/project/farm/issues/3347412) ---
**Author:** ![Guillermo](https://avatars.discourse-cdn.com/v4/letter/g/eb8c5e/32.png) [@Guillermo](https://farmos.discourse.group/u/Guillermo)\ **Post date:** [April 7, 2023, 8:07am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/13 "2023-04-07T08:07:51Z")
In my humble opinion, being new to FarmOS but having a long stance as a farmer trying to go digital, I have to say that a farm manager or dedicated farm worker would like to see a button labeled [delete] associated to such user function. Farmers are radical in regard to the hard facts of life. For example, when they hung a sheep for eating, they have “killed” the sheep, not “archived” it. Therefore, they would like to press a button labeled [dead] [killed] [shot] or, close enough, [deleted], reserving [archived] for when declaring they have put its carcass in the freezer only 🙂 Beyond joke, archived signifies to me that the thing is still there, somewhere. When a farm manager or diligent farm worker erases an asset because the flood took it away or because it was a mistake entry all together, they want to press the sure [delete] button. For programmers, the action behind the label could be called soft delete, archived, or whatever the committee votes the best option. But I propose the UI label should be hard and clear of doubts —for it was done and recorded as such. ---
**Author:** ![Farmer-Ed](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/farmer-ed/32/662_2.png) [@Farmer-Ed](https://farmos.discourse.group/u/Farmer-Ed)\ **Post date:** [April 7, 2023, 8:41am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/14 "2023-04-07T08:41:09Z")
I’m a Beef farmer and i disagree with a lot of that. I don’t think animals sold or slaughtered should be considered deleted, there are a few reasons why an animal needs to remain in an achieved state, such as: 1. Compliance, records such as drugs given to the animal post termination needed for Inspectors. 2. Lineage, animals used in breeding still have relationships with animals still on farm. 3. Profit monitoring, weights prices for animals sold remain important. Perhaps better language could be used for a UI button to Archive, but I believe the current mechanism is universal to all Asset types, so slaughter/kill might not make sense to a Tillage/veg farmers assets. Also slaughter is not always the right answer for a Livestock farmer either, it could be Slaughtered / Sold / Dropped Dead or even only temporarily archived in the case of Contract Rearing or simply an animal sold and bought back in future. There is currently a delete Asset option (which can remove an Asset created in error) Archive Asset is different and better describes an Asset that did exist but is no longer on farm. Personally I think there is room for a few additional Log types, including Animal Death, and Animal Sales, and Animal Purchases and I have added these to my own self hosted instance through a custom module. Also @wotnak has made a nice termination module which can be used to make logs terminating (ie logs can archive the asset when they are created) ---
**Author:** ![Guillermo](https://avatars.discourse-cdn.com/v4/letter/g/eb8c5e/32.png) [@Guillermo](https://farmos.discourse.group/u/Guillermo)\ **Post date:** [April 7, 2023, 9:15am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/15 "2023-04-07T09:15:18Z")
Hi Ed, I agree 100% with what you said above (I am learning here) I am interested in the custom module you developed for your self hosted beef farm instance. Perhaps there is room for same-industry farmers to collaborate in building modules such as yours, specific for a particular industry. ---
**Author:** ![Farmer-Ed](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/farmer-ed/32/662_2.png) [@Farmer-Ed](https://farmos.discourse.group/u/Farmer-Ed)\ **Post date:** [April 7, 2023, 9:31am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/16 "2023-04-07T09:31:11Z")
Always happy to share and/or collaborate, not an expert myself but getting to grips with the ins and outs thanks to the help I have received here. Not much development needed to create new log types, Its a cut paste and minor modification of existing types. If you are self hosting your own farmOS instance make sure to have a separate Dev instance for experimenting with before commiting anything to your production instance. Feel free to start a new topic to discuss what you’d like to achieve or just add to any relevant threads already started. ---
**Author:** ![aislinnpearson](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@aislinnpearson](https://farmos.discourse.group/u/aislinnpearson)\ **Post date:** [January 22, 2025, 10:45am UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/17 "2025-01-22T10:45:22Z")
Hi Mike. We had an instance this week where someone deleted a log for one of our experiments and it restarted a discussion about recycle bins/ not being able to delete data out of the system. I am begining to think that might be the ideal solution as at Rothamsted we have customised our instances so deletions ar restricted to Data Admins, and yet we still lost the data. Which just goes to show this can happen even with restrictions in place. Fortunately we were able to retrieve the data from the paper records we keep but talking to @paul121 yesterday he mentioned you do regular backups but these aren’t the kind of backups you go to in order to retrieve one lost record. They are there in case the system is hacked and needs a full reboot. I mentioned this to our QA manager asked me to document this process internally and share it with our ITS team, so I was wondering if you already have some online documentation/ help pages I can link to. Our QA manager was specifically asking about how regularly the data is backed up, what the process would be if we wanted to request the backup data (I don’t think we do, but good to know how we would access it if we needed it). That sort of thing. Apologies if I am posting this in the wrong place - the last post is about 2 years old, but I wasn’t sure where the chat had moved to. Feel free to e-mail me if anything about my question is confidential. ---
**Author:** ![mstenta](https://yyz2.discourse-cdn.com/free1/user_avatar/farmos.discourse.group/mstenta/32/4_2.png) [@mstenta](https://farmos.discourse.group/u/mstenta)\ **Post date:** [January 22, 2025, 7:23pm UTC](https://farmos.discourse.group/t/proposal-for-farmos-3-x-core-datamodel-improvements-immutable-quantities-soft-entity-deletion/1490/18 "2025-01-22T19:23:58Z")
Hi @aislinnpearson - yea I think it would be great to add this functionality to farmOS core (“soft deletion”) to prevent that kind of issue. I’ll follow up with you separately about Farmier-specific hosting backups and how they work, since that’s a different topic. 👍