Declarative data entry conventions

[mstenta]

(Quite the obtuse topic name, but we have to start somewhere! )

Spinning this off as a dedicated topic from:

@mstenta

This got my gears turning…

Wouldn’t it be cool if you could define a “data entry pattern” in a declarative way (eg: via YAML), and then have that automatically generate both a UI (quick form and/or Asset Link plugin), API (JSON:API resource or custom REST endpoint, along with JSON Schema), and CSV importer? The same could be used to build a local TUI.

This “data entry pattern” would essentially be a codified convention.

This is a fun (and functional) idea that relates to the “Documenting Conventions” discussion.

Documenting conventions

I’d like to propose we establish some standard processes for documenting conventions used in farmOS, in the form of a new Git repository managed by the farmOS community. For background on what we mean by “conventions”, see: Conventions | farmOS To quote from an earlier discussion: A lot of the discussions thusfar have been focused on how to implement conventions in different ways (eg: with quick forms, config entities, JSON schemas, RDF/JSON-LD, tooling for validation, etc). But as a first s…

The first step to exploring this might be brainstorming what the YML might look like for representing a “data entry pattern” (aka a “declarative data entry convention”)…

This is essentially an ETL (Extract, transform, load - Wikipedia) specification. We are starting with some semi-structured data (eg: CSV columns, simplified form/question inputs, etc) and transforming them into records in the farmOS data model.

With that in mind, the first place to start might be Drupal’s core Migrate API, which already provides a YML schema for declaring ETL process flows.

Our documentation provides an example for an “Eggs harvest” CSV importer:

docs/development/module/csv.md?plain=1

Could something like this be used to generate a data entry form (eg: Quick Form, Asset Link plugin, TUI prompt, etc), in addition to a CSV importer?

The example “Harvest Quick Form” in our documentation is not that different from the Eggs Harvest CSV importer example, in terms of what it collects and how it processes it:

docs/development/module/quick.md?plain=1

Migrate API uses “process plugins” for transformation logic. In theory, these could serve the same role as the Form API submit logic we currently use in quick forms.

That’s an interesting idea… refactoring core quick form submit logic into Migrate API plugins. Maybe easier said than done. And quick forms (Form API) perform more than just submit logic, they can also validate the input. Migrate API can check entity/field constraints, but it doesn’t provide any mechanism for direct input validation. Also, Form API provides features for customizing how a form looks/behaves, too. In theory you could use hook_form_alter() to modify a form that’s generated from a declaration after the fact (not suggesting that’s a good idea, just thinking out loud here). So it’s probably unrealistic to think that this idea could replace quick form code entirely (you’ll always have cases where more customization is needed), but it could provide another mechanism for generating simple forms. And if that also gives you a CSV importer and API and … (etc) … that could be very useful.

That said, a lot of the Migrate API YML schema is very specific to Migrate API (not surprising). But maybe we could build a thin and simplified layer of our own on top of that, which then translates down to full Migrate API YML.

Fun to think about… curious what this stirs up in others’ brains…