Conventions as config entities

@paul121 and I just had a great discussion around conventions, quick forms, plans, etc, and how they relate to some of the recent discussions we’ve had with others, including @gbathree (on the forum and in phone calls), with regard to how we can “codify” conventions in a way that enables both “enforcement” (on the data input side, to ensure new records meet the requirements of a convention) and “filtering” (on the data output side, to find records that match a convention).

The outcome was:

What if we created a “convention” config entity type, to represent farmOS conventions?

Drupal has a concept of “config entities” which are basically small chunks of configuration that can be imported/exported as YML files, and then used throughout the system to build logic around them. We already use config entities for a lot of different things in farmOS.

Imagine a “convention” config entity that defines all the various aspects of a convention, including what fields/quantities are required on it, what ontologies are referenced, whether or not multiple linked records are needed, etc etc.

Lots of possibilities open up from this basic idea:

  • Individual records (assets, logs) could reference which conventions they intend to follow via a convention base field on them.
  • Multiple conventions can be “layered” on individual records, to indicate that they follow a general convention, but also have some farm-specific convention elements as well.
  • Conventions could be exported as YML files and shared in modules, or in a community repository.
  • Quick forms and Plans could fill in the convention field when they create assets/logs, to indicate which convention(s) they adhere to.
  • Filters can be added to the UI so you can say “show me all records that match this convention”.
  • API consumers can filter records by convention.
  • Reports can be built that only include records that follow a particular convention (eg: the animal weight report in 1.x basically looks for weight logs using a convention).
  • Audit code could be written that looks at the “intended” convention of a record and checks to see if there are any discrepancies in the record itself. This could be flagged to the user so they can make adjustments.
  • The asset/log forms themselves could even go so far as to offer an option upfront when you click “Add asset/log”, which asks: “would you like this record to follow a convention?” And if you select one, then it changes the form (showing/hiding fields, making certain fields required, renaming labels, creating a fixed set of quantities, etc etc) to make the form more specific (and simple).
  • … ???

The biggest question is: what would these config entities include? What is necessary to accurately represent a convention? We need to start organizing these thoughts and collecting all the use-cases we can think of in order to spec this out.

This is where the rubber meets the road. :slight_smile:

1 Like

Yes! I want to have that weedy conversation!

A few thoughts:

  1. Using drupal components is awesome, but I think there’s still a lot of broader value to conventions producing a final modified schema which encompasses the conventions modifications. Can this still be an output, and can we capture (to your point) all the relevant elements of a convention in that schema? I thought about those elements a bit here and made a list: it’s on gitlab but I can’t post the link so it’s gitlab url + /our-sci/partners/research-farm-roadmap/-/wikis/farmOS-Conventions-notes.

  2. I still think (I know this is controversial) that we need a default on option… I like the idea of a button that says ‘apply this convention’ when creating a log, but also want to make sure that we can default apply conventions - so you click the button and you can turn off conventions if you want. This is absolutely critical for the mass base-level user who is just trying to not break shit.

I would love to have a conversation on this topic and post the notes here or record it if that helps, is there a time that might happen?

1 Like

I think there’s still a lot of broader value to conventions producing a final modified schema which encompasses the conventions modifications. Can this still be an output, and can we capture (to your point) all the relevant elements of a convention in that schema?

I hope so! We’ll have to dig in to lots of specific use cases to explore all the considerations I think.

You and I have talked about somehow potentially extending the JSON Schema to accommodate conventions - I assume this is what you’re referring to by “schema” in this context. I like the idea (or at least I agree with the end goal of having something that can be used to “check” records against). Whether or not that can be done with JSON Schema, I’m less certain. I haven’t quite connected A to B in my head yet, so it’s still a fuzzy idea. But I think this discussion/exploration might help to work through that.

FWIW, config entities actually have two YML files involved. One to represent specific convention configs (one for each), and one to represent the schema for how convention configs work more generally (one for all). So there are multiple levels of “schema” in all of this too. I’d like to focus first on defining what a “convention” is in terms of a YML object, and the schema for that (which makes it a standardized and machine-readable format). We may find that this actually covers the same goal as what you’re hoping to achieve with JSON Schema, but let’s dig in and see… I think we’re on the same page in terms of the end goal either way.

I still think (I know this is controversial) that we need a default on option

Ha! I’m not against this! But first step is to define what a convention is… (as YML) then we can add the UI support… and THEN we can debate making it the “default”. Your upfront point is noted. :wink:

I thought about those elements a bit here and made a list: it’s on gitlab but I can’t post the link so it’s gitlab url + /our-sci/partners/research-farm-roadmap/-/wikis/farmOS-Conventions-notes.

(Ah yea I increased your Discourse user’s trust level so you shouldn’t have that problem anymore… automatic spam prevention I guess.)

Here’s the link for others: farmOS Conventions notes · Wiki · our-sci / partners / Research Farm Roadmap · GitLab

This would be a great place to start!

Ultimately I think what we should start sketching out are VERY LOW LEVEL definitions of conventions. Here is a very simple and rough example:

Recording Egg harvests

The “Egg harvest quick form” is a good example of an existing UI feature that basically implements a simple convention: “how to record an egg harvest”. Perhaps the convention can be summarized in English as:

Egg harvests shall be recorded with a Harvest log. The Harvest log MUST have a Quantity entity with a Count measure and a Units of “egg(s)” with a numeric positive integer value. An egg harvest log MAY reference an Animal asset or a Group asset.

From this written description, we may be able to devise a YML definition that captures all of these requirements…

farm.convention.egg_harvest.yml:

# ...standard config entity stuff
log:
  type: harvest
  fields:
    quantity:
      - measure: "count"
        unit: "egg(s)"
        value:
          constraints:
            - required
            - positive_integer
    asset:
      constraints:
        type:
          - animal
          - group

That is VERY rough and I can already see a lot of things wrong with it… so consider it just a seed for starting the conversation.

It might be worth starting a repository where we just add a bunch of files that include the written descriptions of conventions that we already know we need. From that, we can start building requirements for the schema definition of conventions more generally, and then begin to build YML config entities for each.

There are going to be a lot of things to consider with all this… I expect there will be many rounds of revisions to the schema definition of “what a convention can include” before we ever write code that uses these conventions. But this is the horse, and that is the cart.

I know I’m coming to this discussion late, but have you seen this first draft of a specification we worked on a while back, @mstenta?

The idea was to develop a very barebones specification for the convention itself, agnostic of any particular implementation, and with a mind for how they could be published and versioned for others to use. I think my bullet points in the README got a little out of wack, but the gist was you just need some key identifiers, to link the convention with a specific entity and bundle, and then an identifier and version number for the convention itself.

I think, and correct me here if I’m wrong @gbathree, but JSON Schema itself would not need to be extended. JSON Schema would simply be one possible way of describing the constraints that make up the convention (I think these are usually called Interface Description Languages, or IDL’s? We also discussed with Protocol Buffers and the OpenAPI Spec as possible alternatives). And really, all a constraint is, is a schema that the data can be validated against. It just happens to be more strict than what might be specified by the farmOS Data Model. So, for instance, if the farmOS Data Model (FDM?) says that the attribute status can be any string, a convention might apply the additional constraint that it can only be the strings 'in_progress' or 'done'. So the constraint expressed in JSON Schema might look something like this:

{
  "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
  "$id": "https://example.com/conventions/my_convention/constraints/status",
  "enum": ["done", "in_progress"]
}

A convention would then be a collection of these constraints, which might look something like:

{
  "id": "https://example.com/conventions/my_convention",
  "version": "1.0.0",
  "entity": "log",
  "bundle": "log--activity",
  "constraints": [
    {
      "field": "status",
      "constraint": {
        "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
        "$id": "https://example.com/conventions/my_convention/constraints/status",
        "enum": ["done", "in_progress"]
      }
    },
    {
      "field": "name",
      "constraint": {
        "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
        "$id": "https://example.com/conventions/my_convention/constraints/name",
        "maxLength": 32
      }
    }
  ],
}

The convention itself isn’t anything special, just JSON, not JSON Schema, and of course that could all be done in YAML or any other data format. The critical thing is just that the identifiers are in place, and it’s in some kind of document format that could be pushed to a public repo or other means of publication. Hell, I suppose it could be markdown, for that matter. No reason to be any more strict than we have to. Only the constraint itself complies to JSON Schema. Perhaps different IDL’s would be better suited for certain constraints, as they got more complex, but I think for our initial purposes JSON Schema would work just fine out of the box, with its enum, format, and pattern keywords. I assume OpenAPI and Protocol Buffers even more options we could explore down the line.