JSON Schema Distribution Tools: Next steps.
The current situation is:
- We have a system that enables a user to write very complex JSON
schemas leveraging the FarmOs format using high level functions with
meaningful errors and methods. - The system allows easy documentation, publication and sharing with
minimum effort. - The system allows to share standalone JS code validators, usable
both in the browser and on a server, with 0 dependencies. These, as
all JSON schenma, provide meaningful errors to suers wanting to
validate. - All the functionality is condensed in a GitLab monorepo.
- Consistency is validated via a non standard JSON schema property,
but we still are able to offer both the schemata and our validators.
Improving Code Separation
This library has been built with the intention of obtaining a maximum
profit of the standardization work the user performs in a unique set of
files contained in the folder `definitions`. There should always be a
way to leverage all features by only populating the intended set of
conventions. This is always coupled to abundant data examples to keep
everything constantly tested and provide full feedback to the users of
the extent any modification they perform might have. Of course, that
design does not stop any user from modifying the open source code
providing all the features in any way they want. As a minimum, having a
full implementation can inform any new developers and serve as a working
example.
We want to split this monorepo into up to several independent repos,
which we believe brings a lot of advantages. There are up to three
deliverables we might want to split from the current monolithic
offer.
- The `convention~builder~` library, which provides the interface to
build complex linked schemata involving many FarmOS entities. - The `repo~structure~` which enables the easy and automated
publication of schemata, documentation and code validator libraries. - The `open~teamconvention~`, our first proposed set of conventions,
which can be used by other organizations as a base and is also a
concept validator for us while we develop the whole system.
What's being pulled out into an NPM Package?
-
Stage 1: The `convention~builder~` library.
There are many reasons to do this, no fucntionality will be lost,
and a typical user does not want to have a local copy of the code
intertwined with their `definitions`. Turning the library into an
NPM package which can be installed and version controlled removes no
freedom from the users and in fact adds several advantages.- They can stick to a version of the code or choose always the
latest. Since each repo has it's own tests based on examples,
they can check which examples get broken by each updated and
why, and choose accordingly how to act regarding the update,
including reporting their issues to any developer. - If they want to fork the `convention~builder~` library, it
will be way easier this way, as they will be able to keep track
of their changes and constantly compare with other versions. - The repo size will be smaller and the folder complexity will be
greatly reduced. - A lot of redundant testing will be avoided, as there is no need
to test all the functions again each time a new
example/convention is added. - It will be easier to offer this library and the whole format to
other projects, which might want to adopt the convention
building workflow, but not the other features.
- They can stick to a version of the code or choose always the
-
Stage 2: The `repo~structure~`.
Currently, we are leveraging a lot of CI features and combining code
in several languages, so the only way to get a functioning replica
of our repo is to clone it and modify it incrementally. This can be
cumbersome for several reasons, among others because it mixes up the
convention development flow with the software development
flow which ideally is not even mandatory for users, as the only
thing we expect is the writing of `definitions`. A good way to
allow users to get a functional repo and keep it up to day is to
offer a second `installer` library. It will grab some needed
environment and CI variables and structure all needed folders, CI
files, etc. A user woul no longer need to clone somebody else's
repo in order to get a functional first step in which all
functionality is correctly orchestrated and, if needed, we might
offer some parameters to them allowing some customization, (for
example, opting between publishing their documentation in docusaurus
or mkdocs, two backeds we have currently used and tested).- Since forking isn't happening anymore, we need to provide a new
and better way of obtaining conventions from other repos. This
is also sketched already and is technically simple. Since repos
publish their conventions as NPm packages, we can make them
dependencies, or copy the gitLab artifacts when available.
- Since forking isn't happening anymore, we need to provide a new
-
Stage 3 (no extra work): Open Team Convention Set
After separating step 1 and step 2, what remains in our repo is
exclusively a set of `definitions`. These also form a useful
package, which would now be clearly separated and available to
anybody who wants them as a staring point, learning example,
upstream source, literal copy, etc.
How many packages are created?
Up to three, as described in the previous section.
Who can contribute to each package?
Being hosted in public git repositories, anybody can read the code,
suggest changes, ask for help, etc. Better separating concerns across
repositories probably makes it way easier, as less is happening in each
repo, each one is more focused and adequate documentation can be exposed
in the central spot for each one.
When will updats happen?
With this new separation, updates will happen less often for each part,
and the user will have the option to test how his conventions deal with
the new code before updating, or even choosing if they want to update at
all. The user's set of conventions, as well as the sources they've
chose, will have the most frequent updates, as effectively they are
required to change each time the conventions change in any way. The
structure should eventually achieve all the first intended
functionalities and depending on the userbase's intention could become
quite static soon. The `convention~builder~` library will be updated
only when new features need to be added or bugs need to be fixed.
How is the process of creating a new Farm Convention changed? what's the new process?
We currently propose forking our repo and incrementally applying
changes. This is easy and fast, but has proven to be counterintuitive.
The new process implies a fairly standard development procedure.
- Start a new Git repo. (`git init`)
- Start an NPM module in that repo. ( `npm init` ).
- Install the two involved libraries, `npm install
convention~builder~ convention~schemapublisher~` ). - Initialize the repo calling the convention~schemapublisher~ helper
functions, as it is currently done to initialize projects in React,
Vue or docusaurus. This will provide all the basic configuration,
structure and files. - Set up the required CI variables (repo name, npm publication
credentials, convention set name, etc). Of particular interest is
the source of the basic FarmOS schemata, which can either be a farm
or another repo. - (optional) If you want to inherit conventions from another repo,
list them in a very simple JSON or YAML config file, which will have
a format in which for each repo a user can either request all
conventions, all conventions of several types (like "everything
starting with 'log–activity' and everything starting with
'asset') or cherrypick individual conventions. - Writing definitions as explained in our tutorial and commiting them
into either `staging` or `main` will trigger the CI process,
test that the examples are adequate, publish the validation code,
publish the conventions and publish the wiki containing all
documentation. The links will also be clearly exhibited for the
user.
<!-- -->
-
Updates which need to happen.
- We are totally ready for stage 1.
- We are [almost]{.underline} ready for stage 2, I already started
separating the folder structure in a way that enables it to
happen, but haven't written the actual structure helper.
Writing it is a task of a medium to low complexity and medium
time requirement. I might do it in a full day or even less. - The third fase would be a natural consequence of the first two,
but we would probably improve the documentation to better
reflect the status of our open team convention as first class
deliverables of the project.
If that affects Juliet, what does she need to do?
Juliette or any other user wouldn't need to do much: all their work is
exclusively in the "definitions" folder. They could follow the (rather
fast and easy) process described in the preceding section to initialize
an empty/new project, copy the definitions there and get all the
advantages of the new system without rewriting a line of their
preexisting work.