Conventions as config entities

Development
1

@paul121 and I just had a great discussion around conventions, quick forms, plans, etc, and how they relate to some of the recent discussions we’ve had with others, including @gbathree (on the forum and in phone calls), with regard to how we can “codify” conventions in a way that enables both “enforcement” (on the data input side, to ensure new records meet the requirements of a convention) and “filtering” (on the data output side, to find records that match a convention).

The outcome was:

What if we created a “convention” config entity type, to represent farmOS conventions?

Drupal has a concept of “config entities” which are basically small chunks of configuration that can be imported/exported as YML files, and then used throughout the system to build logic around them. We already use config entities for a lot of different things in farmOS.

Imagine a “convention” config entity that defines all the various aspects of a convention, including what fields/quantities are required on it, what ontologies are referenced, whether or not multiple linked records are needed, etc etc.

Lots of possibilities open up from this basic idea:

  • Individual records (assets, logs) could reference which conventions they intend to follow via a convention base field on them.
  • Multiple conventions can be “layered” on individual records, to indicate that they follow a general convention, but also have some farm-specific convention elements as well.
  • Conventions could be exported as YML files and shared in modules, or in a community repository.
  • Quick forms and Plans could fill in the convention field when they create assets/logs, to indicate which convention(s) they adhere to.
  • Filters can be added to the UI so you can say “show me all records that match this convention”.
  • API consumers can filter records by convention.
  • Reports can be built that only include records that follow a particular convention (eg: the animal weight report in 1.x basically looks for weight logs using a convention).
  • Audit code could be written that looks at the “intended” convention of a record and checks to see if there are any discrepancies in the record itself. This could be flagged to the user so they can make adjustments.
  • The asset/log forms themselves could even go so far as to offer an option upfront when you click “Add asset/log”, which asks: “would you like this record to follow a convention?” And if you select one, then it changes the form (showing/hiding fields, making certain fields required, renaming labels, creating a fixed set of quantities, etc etc) to make the form more specific (and simple).
  • … ???

The biggest question is: what would these config entities include? What is necessary to accurately represent a convention? We need to start organizing these thoughts and collecting all the use-cases we can think of in order to spec this out.

This is where the rubber meets the road. :slight_smile:

2 Likes
2

Yes! I want to have that weedy conversation!

A few thoughts:

  1. Using drupal components is awesome, but I think there’s still a lot of broader value to conventions producing a final modified schema which encompasses the conventions modifications. Can this still be an output, and can we capture (to your point) all the relevant elements of a convention in that schema? I thought about those elements a bit here and made a list: it’s on gitlab but I can’t post the link so it’s gitlab url + /our-sci/partners/research-farm-roadmap/-/wikis/farmOS-Conventions-notes.

  2. I still think (I know this is controversial) that we need a default on option… I like the idea of a button that says ‘apply this convention’ when creating a log, but also want to make sure that we can default apply conventions - so you click the button and you can turn off conventions if you want. This is absolutely critical for the mass base-level user who is just trying to not break shit.

I would love to have a conversation on this topic and post the notes here or record it if that helps, is there a time that might happen?

1 Like
3

I think there’s still a lot of broader value to conventions producing a final modified schema which encompasses the conventions modifications. Can this still be an output, and can we capture (to your point) all the relevant elements of a convention in that schema?

I hope so! We’ll have to dig in to lots of specific use cases to explore all the considerations I think.

You and I have talked about somehow potentially extending the JSON Schema to accommodate conventions - I assume this is what you’re referring to by “schema” in this context. I like the idea (or at least I agree with the end goal of having something that can be used to “check” records against). Whether or not that can be done with JSON Schema, I’m less certain. I haven’t quite connected A to B in my head yet, so it’s still a fuzzy idea. But I think this discussion/exploration might help to work through that.

FWIW, config entities actually have two YML files involved. One to represent specific convention configs (one for each), and one to represent the schema for how convention configs work more generally (one for all). So there are multiple levels of “schema” in all of this too. I’d like to focus first on defining what a “convention” is in terms of a YML object, and the schema for that (which makes it a standardized and machine-readable format). We may find that this actually covers the same goal as what you’re hoping to achieve with JSON Schema, but let’s dig in and see… I think we’re on the same page in terms of the end goal either way.

I still think (I know this is controversial) that we need a default on option

Ha! I’m not against this! But first step is to define what a convention is… (as YML) then we can add the UI support… and THEN we can debate making it the “default”. Your upfront point is noted. :wink:

I thought about those elements a bit here and made a list: it’s on gitlab but I can’t post the link so it’s gitlab url + /our-sci/partners/research-farm-roadmap/-/wikis/farmOS-Conventions-notes.

(Ah yea I increased your Discourse user’s trust level so you shouldn’t have that problem anymore… automatic spam prevention I guess.)

Here’s the link for others: farmOS Conventions notes · Wiki · our-sci / partners / Research Farm Roadmap · GitLab

This would be a great place to start!

Ultimately I think what we should start sketching out are VERY LOW LEVEL definitions of conventions. Here is a very simple and rough example:

Recording Egg harvests

The “Egg harvest quick form” is a good example of an existing UI feature that basically implements a simple convention: “how to record an egg harvest”. Perhaps the convention can be summarized in English as:

Egg harvests shall be recorded with a Harvest log. The Harvest log MUST have a Quantity entity with a Count measure and a Units of “egg(s)” with a numeric positive integer value. An egg harvest log MAY reference an Animal asset or a Group asset.

From this written description, we may be able to devise a YML definition that captures all of these requirements…

farm.convention.egg_harvest.yml:

# ...standard config entity stuff
log:
  type: harvest
  fields:
    quantity:
      - measure: "count"
        unit: "egg(s)"
        value:
          constraints:
            - required
            - positive_integer
    asset:
      constraints:
        type:
          - animal
          - group

That is VERY rough and I can already see a lot of things wrong with it… so consider it just a seed for starting the conversation.

It might be worth starting a repository where we just add a bunch of files that include the written descriptions of conventions that we already know we need. From that, we can start building requirements for the schema definition of conventions more generally, and then begin to build YML config entities for each.

There are going to be a lot of things to consider with all this… I expect there will be many rounds of revisions to the schema definition of “what a convention can include” before we ever write code that uses these conventions. But this is the horse, and that is the cart.

4

I know I’m coming to this discussion late, but have you seen this first draft of a specification we worked on a while back, @mstenta?

https://gitlab.com/OpenTEAM1/OpenTEAM-Technology/-/tree/master/conventions

The idea was to develop a very barebones specification for the convention itself, agnostic of any particular implementation, and with a mind for how they could be published and versioned for others to use. I think my bullet points in the README got a little out of wack, but the gist was you just need some key identifiers, to link the convention with a specific entity and bundle, and then an identifier and version number for the convention itself.

I think, and correct me here if I’m wrong @gbathree, but JSON Schema itself would not need to be extended. JSON Schema would simply be one possible way of describing the constraints that make up the convention (I think these are usually called Interface Description Languages, or IDL’s? We also discussed with Protocol Buffers and the OpenAPI Spec as possible alternatives). And really, all a constraint is, is a schema that the data can be validated against. It just happens to be more strict than what might be specified by the farmOS Data Model. So, for instance, if the farmOS Data Model (FDM?) says that the attribute status can be any string, a convention might apply the additional constraint that it can only be the strings 'in_progress' or 'done'. So the constraint expressed in JSON Schema might look something like this:

{
  "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
  "$id": "https://example.com/conventions/my_convention/constraints/status",
  "enum": ["done", "in_progress"]
}

A convention would then be a collection of these constraints, which might look something like:

{
  "id": "https://example.com/conventions/my_convention",
  "version": "1.0.0",
  "entity": "log",
  "bundle": "log--activity",
  "constraints": [
    {
      "field": "status",
      "constraint": {
        "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
        "$id": "https://example.com/conventions/my_convention/constraints/status",
        "enum": ["done", "in_progress"]
      }
    },
    {
      "field": "name",
      "constraint": {
        "$schema": "https://json-schema.org/draft/2019-09/hyper-schema",
        "$id": "https://example.com/conventions/my_convention/constraints/name",
        "maxLength": 32
      }
    }
  ],
}

The convention itself isn’t anything special, just JSON, not JSON Schema, and of course that could all be done in YAML or any other data format. The critical thing is just that the identifiers are in place, and it’s in some kind of document format that could be pushed to a public repo or other means of publication. Hell, I suppose it could be markdown, for that matter. No reason to be any more strict than we have to. Only the constraint itself complies to JSON Schema. Perhaps different IDL’s would be better suited for certain constraints, as they got more complex, but I think for our initial purposes JSON Schema would work just fine out of the box, with its enum, format, and pattern keywords. I assume OpenAPI and Protocol Buffers even more options we could explore down the line.

1 Like
5

@gbathree and I had a great discussion, with special guest @sudokita.

I won’t go into too much detail here, but I wanted to document the kind of workflow I envisioned between farmOS, farmOS.js and SurveyStack, which I put into this gist:

TL;DR: SurveyStack gets a farmOS log schema via farmOS.js. It creates a convention based on that schema, stores it and publishes it, then later can create logs that comply with that schema, push them back to farmOS, and later third-parties can retrieve them from farmOS directly.

2 Likes
6

Awesome gist @jgaehring! Sounds like a good plan! SurveyStack (and the Coffee Shop) is a great place to incubate/experiment with these convention ideas. The lessons learned will help to inform our process of defining these ideas as more standardized specifications (where possible).

FWIW I’ve started drafting some longer-term ideas involving RDF/JSON-LD as well - perhaps as an alternative to this “config entity” approach idea. I’ll try to get these ideas into a forum topic for further discussion. It will be nice to think about all of these things in parallel, and get more voices and use-cases included!

In either case I’m excited to see things start to take shape in SurveyStack as a first step! It makes logical sense to start there.

1 Like
7

Thanks all, great discussions and helpful, got me set on the right track I think :slight_smile:

here’s where I kind of lay it out → https://miro.com/app/board/o9J_lGv1SWk=/?invite_link_id=457106734132

I’m going to try to connect with Tibet and Tom from Hylo also this coming week to make sure the ‘farm profile’ component has a clear space. Personally I like the farm profile component as an early RDF component, as one would expect it’ll get used in a range of spaces on the internet, which seems like a good use.

2 Likes
8

This is pretty tangential, but I’ve been thinking back to this presentation by Scott Wlaschin ever since our chat the other day. I think @sudokita has been discussing UML as a way of communicating to users what these conventions are. Espeically when we need to discuss what actual fields are included in a farmOS record, like a log, and how those fields might be constrained. The talk is called Domain Modeling Made Functional:

He has a brief write-up on his blog with links to the slides, too, if that’s more helpful. Essentially, it’s rooted in Domain Driven Design (DDD) and the concept of “ubiquitous language”, using F#'s Hindley-Milner type system as a shared language between developers and users for describing a particular domain. Hindley-Milner sounds super scary, especially for its associations with Haskell, but I think it could be a nice alternative to UML. I find it pretty intuitive, and it’s nice that anyone with a text editor can effectively create or modify a given model.

Here’s the example of a type module that Wlaschin provides in his talk, which I think best illustrates all this:

module CardGame =

  type Suit = Club | Diamond | Spade | Heart

  type Rank = Two | Three | Four | Five | Six | Seven | Eight
              | Nine | Ten | Jack | Queen | King | Ace

  type Card = Suit * Rank

  type Hand = Card list
  type Deck = Card list

  type Player = {Name:string; Hand:Hand}
  type Game = {Deck:Deck; Players: Player list}

  type Deal = Deck -> (Deck * Card)

  type PickupCard = (Hand * Card) -> Hand

He points out that any developer (and most users) would not only be able to understand what this module is describing, but could even point out any potential errors, like if we were missing a rank or something. And to boot, this module will actually compile and type check against potential errors that could arise elsewhere in the codebase, so you can be sure everything is adhering to the domain as described. Pretty nifty!

Anyways, I realize we’re not adopting F# or Haskell anytime soon, but it seems like a nice alternative to consider, and may have fewer barriers (ie, less software tooling) to getting up and running than UML.

1 Like
9

Minor update on this front; I’m now publishing the “cleaned-up” schema documents for core farmOS entity bundles:

https://raw.githubusercontent.com/farmOS/farmOS.js/100a908fb76296a36fd5b57f54e9687a33160cbf/core_schemata/log/activity.json

It’s a bit of the hack for the time being, taking advantage of GitHub’s raw file URL’s, but the nice thing is this doesn’t contain any references to Drupal-specific fields, and gets rid of some unnecessary nested fields and references as well. Curious what you think, @gbathree.

Bigger updates coming in the next couple days, so keep an eye out!

:wink:

1 Like