Work Proposal: Scaffold a new Gatsby site for farmOS.org & 2.x docs

Work Proposal

Title: Scaffold a new Gatsby site for farmOS.org & 2.x docs
Submitted by: Jamie Gaehring
OpenCollective account: Jamie Gaehring - Open Collective
Cost: $600
Completion Date: 30 Apr 2021

Description

The goal of this project is to scaffold a new website for farmos.org, which can be used to host documentation from multiple GitHub repositories. Eventually the hope is that this site could also house more diverse content in the form of a blog, tutorials and other community-generated materials, as well as a richer, more marketable landing page to attract new users. Although those larger goals are not within the scope of this immediate proposal, part of the aim of this scaffolding project is to enable the gradual development of those features as a series of progressive enhancements, while we approach the launch of farmOS 2.0. I will build the new site using the Gatsby framework, as a more powerful and flexible static-site generator (SSG), along with Netlify for hosting, which combined should be able to achieve these short-term and long-term goals. Such a progression would be difficult or impossible to achieve with our current tool-chain of MkDocs and GitHub Pages.

For more background, see farmOS.org Redesign.

Deliverables & Acceptance Criteria

  • A public repository with the source code for the new site
  • The ability to use the farmOS 2.x branch as a content source for documentation
  • A theme for the documentation pages that provides basic navigation, table of contents and search, on par with the current docs.farmos.org
  • A preview of the site hosted on Netlify

Poll

Do you support funding this proposal?
  • Yes
  • No

0 voters

1 Like

Iā€™ve got a first draft available for preview:
https://gracious-brattain-bdd606.netlify.app/

Source repository:

Iā€™m pretty pleased with it. Iā€™ve tried to keep parity with what we have currently at docs.farmos.org. However, Iā€™ve tweaked the colors to be a little more on-brand, and shrunk the width of the body text to be a little easier to read.

Some features still lacking:

  • Search
  • Proper slugs and/or a different home page
  • SEO
  • Copy button code blocks
  • Center text when thereā€™s no ToC
  • farmOS icon

Of course, the big one is search, and Iā€™m hoping to spend tomorrow evening working on that. The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because Iā€™m not sure how we want to structure things once we start pulling docs from multiple source repos. Right now Iā€™ve got a pretty sloppy redirect in place, which is painfully obvious the first time you load the page and get a flash of the Gatsby starter page, but thatā€™s just a temporary fix. SEO should be made pretty easy with this plugin, Iā€™ll try to bang that out before tackling search tomorrow. The rest of the features struck me as relatively minor and/or not worth pursuing seriously until we have a full rebranding effort in place.

Looking forward to hearing feedback!

3 Likes

A couple little issues Iā€™ve noticedā€¦

Some inline code blocks overflow the main content container, as can be seen here:
https://gracious-brattain-bdd606.netlify.app/farmos/docs/development/module/fields/#structure-type

I also realized while getting the above link that I need to adjust the scroll on those heading links to compensate for the app bar.

Edit: One more thing is the main menu on mobile, you have to click on the links, it doesnā€™t work if you click elsewhere in the <li>'s box.

1 Like

I walked back on this a little, widening the main container from 75 chars to 90 chars, to allow for longer text strings like the one I mentioned here:

I also added word breaks for inline code blocks just to be safe.

Fixed.

Added some really basic meta tags, nothing fancy. But the structureā€™s in place. I borrowed pretty liberally from this tutorial and put all the details in the gatsby-config.js file in root, for anyone interested. We can revisit this to tweak the description and keywords etc at a later date. Testing out the link, preview now:

https://gracious-brattain-bdd606.netlify.app/farmos/docs

Looks like the preview image is still broken, but we can tweak that.

I hoped to get to search tonight, but donā€™t think thatā€™ll happen. I did find a nice guide, though, while looking for SEO stuff. Maybe Iā€™ll give it another go later tonight or tomorrow.

1 Like

Just fixed a major issue where all absolute links in markdown files were being prefixed with the site domain. Oops!

Iā€™ll note that the check I wrote could probably be more robust, if we want to support more protocols than http and https; Iā€™m just checking if the string starts with 'http'. I canā€™t think of any other protocols we would want to link to in our markdown files, but perhaps Iā€™m overlooking something?

1 Like

Great progress @jgaehring! Exciting to see this start to take shape!

Quick thoughts/observations/questions:

  • First, I love how you were able to keep things so similar to the current docs.farmos.org! Thatā€™s impressive in itself! :smiley:
  • Search would be great - but I could also see that being its own follow-up with a separate budget, if it seems like it might eat up too much this round. The link navigation is enough for now.
  • I noticed that ā€œData modelā€ section is omitted? Is this intentional (do you have future plans in mind - perhaps in a more top-level section)?
  • The left nav items donā€™t stay ā€œopenā€ when you click through to one of them, and it doesnā€™t highlight the one youā€™re currently looking at. Not sure how easy this would be, but the navigation is more cumbersome without it.
  • I like the ā€œNextā€ and ā€œBackā€ footer features that MkDocs+Material provided, but obviously not critical for this first pass.
  • Needs footer with Creative Commons and trademark text.

The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because Iā€™m not sure how we want to structure things once we start pulling docs from multiple source repos.

Happy to help think this though. Could you elaborate a bit on what the considerations are?

Yea, Iā€™d at least like to take a stab at the core functionality, b/c I think weā€™ll need to do that sooner or later, although I may skimp a bit on the styling. I actually really like the way Material for MkDocs styles and handles UX for search, but itā€™s a lot of CSS complexity that might be hard to emulate or cleanly copy & paste. Probably not worth the trouble right now.

Ah, yes, I forgot to mention this. Up until now, Iā€™ve been using a stub to reproduce the data from the mkdocs.yml file, without being entirely dependent on it. Because I havenā€™t updated this stub since early on, it is still missing the Data Model and other sections. As I think Iā€™ve mentioned, Iā€™d like to get to the point where instead of the mkdocs.yml file, we use Frontmatter in the markdown files themselves, because itā€™s a more widely adopted standard and serves to decouple the source repos from the Gatsby repo. Iā€™ve actually been sitting on a PR until now, which Iā€™ve just opened so you can see the kind of changes I have in mind:

One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that donā€™t have pages, for instance, ā€œDevelopmentā€. Instead, what Iā€™m doing for now, is parsing the pathname for the of those headingsā€™ child paths, eg '/development/api/', to extract and capitalize the string 'Development' and use that for the heading. This could have some obvious shortcomings, which will probably eventually require some kind of additional configuration, either in the source repo or the Gatsby config, but I figured to punt on that for now.

Yea, Iā€™m still putting the final touches on the navigation, replacing the stub file, parsing pathnames, etc, as I mentioned above. Coincidentally, this has been the most difficult part of the project to date, and I lost the better part of this weekend bogged down in recursive parsing algorithms for how to do this most effectively, yeesh! :woozy_face: However, Iā€™ve almost cracked it, and I believe it will make for much simpler logic in determining which page is currently being viewed and ensuring that the menu is open to that section of the menu by default.

Cool! I wondered what you thought that the back/next buttons. It is nice. And I think can be pretty easy to replicate. As will the copyright. Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo. There are different ways to handle either scenario, but curious your thoughts. Actually, come to think of it, the nicest, cleanest way might be to just have a separate markdown file for it, something like /docs/COPYRIGHT.md. That way you can format it and add links however you like, itā€™s easy for people to find and read in the source repo, and not closely coupled to any particular implementation.

Basically, Iā€™m thinking about the ultimate configuration, when this becomes farmos.org, how do we breakdown the paths to the different projectsā€™ documentation?

I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.

But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while. Again, probably best to just set aside some time to talk.

Gotcha - makes sense. Will frontmatter in farmOS/farmOS remove the need for this?

Along those lines, one thing weā€™ll need to do is outline what the update process will need to be - and what will be manual and what will be automatic. Weā€™ve talked a bit about this in chat.

Specifically: when a change is made to the docs in farmOS/farmOS, what (if anything) is needed to trigger the update to be pulled into farmOS.org?

Hopefully it can be fully automated? Perhaps a Git webhook that fires when a commit is pushed to farmOS/farmOS, which triggers a GitHub Action in farmOS.orgā€™s repo to rebuild via Gatsby?

If we need to make changes manually in both repos whenever edits are made that will become quite cumbersome. I love that right now I can merge a PR for farmOS.org via the GitHub UI and trust that it will be rebuilt and published automatically. Of course, pulling content from multiple other sources adds another layer of complexity to that.

One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that donā€™t have pages

If it makes it any easier, we can certainly add stub index.md pages for those, and just make that a requirement moving forward. Does that make sense? We would just expect each subdirectory to have an index.md which is used for the top-level parent (as well as perhaps being added as a link itself underneath, as in the ā€œDevelopment > Module > Getting startedā€ page, for example (which is an index.md file).

Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo.

Each repo should have their own COPYRIGHT.txt (here is farmOSā€™s, which will be added to the 2.x branch via #3162606), which covers all code inside the repo, and that also covers the docs content. Perhaps we also need to include a creative commons notice somewhere in there - I hadnā€™t thought of that.

I think we should also have a dedicated footer on farmOS.org. And maybe we just make it a policy that any docs from remote repos being aggregated into farmOS.org must agree to making it available as creative commons as well?

Either way, it needs the standard trademark text (per trademark policy): farmOS Trademark Guidelines | farmOS

I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.
ā€¦
But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while.

Cool cool! Yea agreed we can discuss - I think it makes sense to take it slow and steady.

Yes.

I wholeheartedly agree we really need a solution that automates new builds whenever thereā€™s a change to the source repo.

I havenā€™t researched this fully yet, but Iā€™m inclined to think we can set this up with a Netlify build hook, perhaps in conjunction with a simple GitHub Action in each source repo.

Yea, I thought of something similar too. I would probably like to support both the automatic generation of menu titles, plus a configuration option if you want to change it. That way, we donā€™t have to require an index.md in every source directory, but you can drop one in if you want to change the default title thatā€™s generated from the pathname. Or something like that.

Right on. So to clarify, the copyright property in your current mkdocs.yml file covers two things: the CC license is for the docs, and TM is for farmOS itself. Iā€™m thinking we split that up, and include the TM in the Gatsby repo, and CC in the docs directory somewhere.

As for the COPYRIGHT.txt files, it would be nice to generate a page for each project that displays its license, perhaps with a little additional explainer, but that might be stretch goal, since I think having it in the source repository is probably sufficient, without including it independently in each projectā€™s documentation website. But if you think otherwise, let me know.

And just to clarify my own point now, I think perhaps our immediate target should be to target docs.farmos.org as the transitional home for all 2.x docs, for all projects (farmOS.js, farmOS.py, etc), so we can really test out the aggregation of the source repos and how theyā€™ll all fit together, before deciding whether to eventually lump everything together with the rebranded farmos.org.

1 Like

Finally got back to working on this over the weekend and knocked out what I think were the only remaining hard requirements, at least in terms of site features:

  • Added the CC license and farmOS trademark info to the footer. Might be nice to separate these two at some point in the future, so separate documentation sources could have separate licenses, while the trademark info always remains the same, but I think thatā€™s a later consideration.
  • The main site nav on the left automatically opens to the page your are currently viewing, and highlights it. Because Gatsby uses SSR by default, the drawer itself will close after you click, and Iā€™d like to smooth out that transition a little more nicely, but I think thatā€™s a stretch goal.

I also spent way too long updating npm packages and the like since last time. I suppose itā€™s worth bearing in mind that this is a drawback of using Gatsby or really any JS framework(s) for a site like this.

So the only hard and fast requirement I have on my docket is to work out the build process, so whenever thereā€™s an update to a documentation source repo, the site will rebuild automatically to reflect those changes. As I said in my last comment, I think a Netlify build hook will probably suffice, although it might be necessary to add a small GitHub Action in each source repo that will fire that hook.

And a quick note on this:

I wound up using the mkdocs.yml file after all, while still retaining the parsing logic that generates the navigation tree based on the file structure alone if there is not a mkdocs.yml. Basically I realized there was more information encoded into one of the nav YAML property than I first ascertained, and it wasnā€™t worth trying to encode all that into some combination of Frontmatter and index.yml files to make that work. At least for now. I think itā€™s still a good idea worth revisiting once weā€™ve tried out using this with other source repos for a while.

For now, Iā€™m gonna just close that PR though. Maybe we wonā€™t need the favicon in the source repo any more, and weā€™ll probably replace the GitHub Pages action with something to trigger Netlify builds, but it seems worthwhile just starting fresh for those changes.

Anyways, I hope to have time in the next few nights to test out those build hooks and hopefully have this ready to deploy (finally!) sometime this week or next.

:crossed_fingers:

1 Like

Oh, and Iā€™ve got these stretch goals on my list as well:

  • Add BACK/NEXT buttons in the footer
  • Search
  • Persist the nav drawerā€™s open status across page transitions
1 Like

I set up a GitHub Action in my fork of farmOS 2.x to trigger a Netlify build hook to rebuild the main site whenever thereā€™s a push to that branch (currently 2.x-netlify-build-docs, but it should eventually be changed to the 2.x branch):

And it worked!

It should also trigger a branch deploy at https://preview-farmos--gracious-brattain-bdd606.netlify.app/ if thereā€™s a PR to that branch. I havenā€™t tried that out yet, but if someone wants to give it a shot by opening a PR on that branch of my fork, be my guest!

I think another stretch goal might be to add a conditional for the entire job so it only runs when there are changes to the docs/ directory. Once we get this workflow working just how we like, it would also be nice to set it up as a reusable workflow that all repos in the farmOS GitHub org can use for their documentation.

@mstenta, I might open a PR to farmOS/farmOS with just this workflow commit so we can play around with it a little more. It shouldnā€™t really hurt anything at this point, while it only runs on events for the 2.x-netlify-build-docs, and even then only just fires off a curl command to Netlify that shouldnā€™t effect any other workflows. We should also think about how we want to set up a more permanent Netlify deployment that can belong to farmOS, rather then my own personal Netlify account, and make sure the secret token and all are set up for the farmOS GitHub org.

A few updatesā€¦

As I said I might in the previous comment, I opened a pull request on the farmOS repository to add the GHA workflow that triggers the Netlify builds:

I did a little poking around to figure out what the matter was with those failed tests. At first it seemed to be whitespace issues, which I amended, but seems like thereā€™s more going on in the sqlite tests. More details there.

From there I resumed work trying to figure out how to make the navigation drawer stay open, or at least close more gracefully, on page transitions. This opened up a whole can of worms regarding how the navigation is structured, such as how the navigation of a particular projectā€™s documentation relates to the overall navigation for the site in general. Right now, itā€™s hard to to really anticipate what this should look like, since it wonā€™t really be an issue until weā€™re using more than one source repository. But there are unavoidable knock-on effects which seem important to consider now, namely how do we aggregate and structure the data from the source repositories and parcel out portions of that data to the pages that need it. Iā€™ve got a WIP branch started, but Iā€™m probably going to shelve that for now, so I donā€™t have to deal with that can of worms just yet.

Iā€™ve been hesitant to write up issues for this kind of stuff in my own GH repo because I hope we can move this all into a farmOS org repo, but looking further into it now, it seems GitHub has a straightforward procedure for transferring repositories, so from here on Iā€™ll open more detailed issues there.

Suffice to say, thereā€™s still some low-hanging fruit I could chip away at, and those failing tests to address, but it would be good to do a quick review and discuss next steps, if thereā€™s a good time to do so in the following week.

1 Like

Bumping this topic againā€¦

They seem to work now, soā€¦ no problem I guess?

Iā€™ve consolidated these down to a ā€œMust-havesā€ milestone, for easier reference. Thereā€™s one more issue that might go in there, but as I explain there, it involves a lot of decisions we might not be ready to make just yet.

I was going to add some more concrete next steps, too, but Iā€™ll have to come back and update this later tonight with those.

Ok, so as I promised @mstenta yesterday, here are the next steps Iā€™d like to take to fulfill the scope of this proposal:

  1. Knock out the ā€œMust-havesā€ (and add commits to PR #456 where applicable).
  2. Rename my repo from jgaehring/farmos.org-2.x to jgaehring/docs.farmOS.org.
  3. Transfer my repo to the farmOS GitHub organization.
  4. Set up a farmOS Netlify acct.
  5. Add Netlify webhook tokens to the farmOS organization secrets.
  6. Merge PR #456, which I believe should trigger the first full build. :crossed_fingers:

The only thing after that is to set up the domain registry for docs.farmos.org to use the Netlify host, but we can hold off on that for as long as we wish. The build can just live at the Netlify domain while we test it out, with the Mkdocs build still working as usual on docs.farmos.org. Then, when weā€™re happy with how everything is working, we can flip the switch.

With regards to step #2, this is intended to distinguish it from the farmOS/farmOS.org repo, at least for the time being. Iā€™m very open to other suggestions, but this seemed to make sense given the domain this will first be deployed to is docs.farmos.org. @mstenta has suggested we merge the two repos and just use separate branches, which would obviate the need for this, in order to preserve the git history while also maintaining continuity. I still think that would be great, but Iā€™d like to wait a little longer to do that. At the very least, I donā€™t think I can merge them prior to transferring my repo to the farmOS organization and preserve the issues Iā€™ve opened up there. I believe that can only be done with issues in two projects belonging to the same organization. So renaming it for the meantime makes the most sense to me, and then maybe hold off on merging the repos until we know weā€™re going to start deploying this to farmos.org.

But I think thatā€™s it! Obviously, thereā€™s lots more we can and will do, particularly when we get to the rebranding stage, but I think for the basic scaffolding phase, we should be set. I should have the ā€œmust-havesā€ done by the weekend if not sooner, so perhaps we can get started on transferring the repo sometime next week.

Iā€™ve managed to bang out all the ā€œmust-havesā€ over the last couple weekends, and also dug into some of the deeper issues with navigation, which Iā€™m finally pleased with.

Iā€™m going to start an agenda for the Monthly Call this Wednesday, and put this on the docket. Iā€™d really love to move the repo over to the farmOS org though as soon as we can.

1 Like

So, after our monthly call, I wanted pull out and revise the next steps from above:

  1. Rename my repo from jgaehring/farmos.org-2.x to jgaehring/farmOS.org-EXPERIMENTAL.
  2. Transfer my repo to the farmOS GitHub organization.
  3. Set up a farmOS Netlify acct.
  4. Add Netlify webhook tokens to the farmOS organization secrets.
  5. Merge PR #456, which I believe should trigger the first full build. :crossed_fingers:

@mstenta, Iā€™ll reach out to you next week to find a time to walk through steps 2 - 5.

1 Like