Work Proposal: Scaffold a new Gatsby site for farmOS.org & 2.x docs

Work Proposal

Title: Scaffold a new Gatsby site for farmOS.org & 2.x docs
Submitted by: Jamie Gaehring
OpenCollective account: https://opencollective.com/jamiegaehring
Cost: $600
Completion Date: 30 Apr 2021

Description

The goal of this project is to scaffold a new website for farmos.org, which can be used to host documentation from multiple GitHub repositories. Eventually the hope is that this site could also house more diverse content in the form of a blog, tutorials and other community-generated materials, as well as a richer, more marketable landing page to attract new users. Although those larger goals are not within the scope of this immediate proposal, part of the aim of this scaffolding project is to enable the gradual development of those features as a series of progressive enhancements, while we approach the launch of farmOS 2.0. I will build the new site using the Gatsby framework, as a more powerful and flexible static-site generator (SSG), along with Netlify for hosting, which combined should be able to achieve these short-term and long-term goals. Such a progression would be difficult or impossible to achieve with our current tool-chain of MkDocs and GitHub Pages.

For more background, see farmOS.org Redesign.

Deliverables & Acceptance Criteria

  • A public repository with the source code for the new site
  • The ability to use the farmOS 2.x branch as a content source for documentation
  • A theme for the documentation pages that provides basic navigation, table of contents and search, on par with the current docs.farmos.org
  • A preview of the site hosted on Netlify

Poll

Do you support funding this proposal?
  • Yes
  • No

0 voters

1 Like

I’ve got a first draft available for preview:
https://gracious-brattain-bdd606.netlify.app/

Source repository:

I’m pretty pleased with it. I’ve tried to keep parity with what we have currently at docs.farmos.org. However, I’ve tweaked the colors to be a little more on-brand, and shrunk the width of the body text to be a little easier to read.

Some features still lacking:

  • Search
  • Proper slugs and/or a different home page
  • SEO
  • Copy button code blocks
  • Center text when there’s no ToC
  • farmOS icon

Of course, the big one is search, and I’m hoping to spend tomorrow evening working on that. The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because I’m not sure how we want to structure things once we start pulling docs from multiple source repos. Right now I’ve got a pretty sloppy redirect in place, which is painfully obvious the first time you load the page and get a flash of the Gatsby starter page, but that’s just a temporary fix. SEO should be made pretty easy with this plugin, I’ll try to bang that out before tackling search tomorrow. The rest of the features struck me as relatively minor and/or not worth pursuing seriously until we have a full rebranding effort in place.

Looking forward to hearing feedback!

3 Likes

A couple little issues I’ve noticed…

Some inline code blocks overflow the main content container, as can be seen here:
https://gracious-brattain-bdd606.netlify.app/farmos/docs/development/module/fields/#structure-type

I also realized while getting the above link that I need to adjust the scroll on those heading links to compensate for the app bar.

Edit: One more thing is the main menu on mobile, you have to click on the links, it doesn’t work if you click elsewhere in the <li>'s box.

1 Like

I walked back on this a little, widening the main container from 75 chars to 90 chars, to allow for longer text strings like the one I mentioned here:

I also added word breaks for inline code blocks just to be safe.

Fixed.

Added some really basic meta tags, nothing fancy. But the structure’s in place. I borrowed pretty liberally from this tutorial and put all the details in the gatsby-config.js file in root, for anyone interested. We can revisit this to tweak the description and keywords etc at a later date. Testing out the link, preview now:

Looks like the preview image is still broken, but we can tweak that.

I hoped to get to search tonight, but don’t think that’ll happen. I did find a nice guide, though, while looking for SEO stuff. Maybe I’ll give it another go later tonight or tomorrow.

1 Like

Just fixed a major issue where all absolute links in markdown files were being prefixed with the site domain. Oops!

I’ll note that the check I wrote could probably be more robust, if we want to support more protocols than http and https; I’m just checking if the string starts with 'http'. I can’t think of any other protocols we would want to link to in our markdown files, but perhaps I’m overlooking something?

1 Like

Great progress @jgaehring! Exciting to see this start to take shape!

Quick thoughts/observations/questions:

  • First, I love how you were able to keep things so similar to the current docs.farmos.org! That’s impressive in itself! :smiley:
  • Search would be great - but I could also see that being its own follow-up with a separate budget, if it seems like it might eat up too much this round. The link navigation is enough for now.
  • I noticed that “Data model” section is omitted? Is this intentional (do you have future plans in mind - perhaps in a more top-level section)?
  • The left nav items don’t stay “open” when you click through to one of them, and it doesn’t highlight the one you’re currently looking at. Not sure how easy this would be, but the navigation is more cumbersome without it.
  • I like the “Next” and “Back” footer features that MkDocs+Material provided, but obviously not critical for this first pass.
  • Needs footer with Creative Commons and trademark text.

The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because I’m not sure how we want to structure things once we start pulling docs from multiple source repos.

Happy to help think this though. Could you elaborate a bit on what the considerations are?

Yea, I’d at least like to take a stab at the core functionality, b/c I think we’ll need to do that sooner or later, although I may skimp a bit on the styling. I actually really like the way Material for MkDocs styles and handles UX for search, but it’s a lot of CSS complexity that might be hard to emulate or cleanly copy & paste. Probably not worth the trouble right now.

Ah, yes, I forgot to mention this. Up until now, I’ve been using a stub to reproduce the data from the mkdocs.yml file, without being entirely dependent on it. Because I haven’t updated this stub since early on, it is still missing the Data Model and other sections. As I think I’ve mentioned, I’d like to get to the point where instead of the mkdocs.yml file, we use Frontmatter in the markdown files themselves, because it’s a more widely adopted standard and serves to decouple the source repos from the Gatsby repo. I’ve actually been sitting on a PR until now, which I’ve just opened so you can see the kind of changes I have in mind:

One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that don’t have pages, for instance, “Development”. Instead, what I’m doing for now, is parsing the pathname for the of those headings’ child paths, eg '/development/api/', to extract and capitalize the string 'Development' and use that for the heading. This could have some obvious shortcomings, which will probably eventually require some kind of additional configuration, either in the source repo or the Gatsby config, but I figured to punt on that for now.

Yea, I’m still putting the final touches on the navigation, replacing the stub file, parsing pathnames, etc, as I mentioned above. Coincidentally, this has been the most difficult part of the project to date, and I lost the better part of this weekend bogged down in recursive parsing algorithms for how to do this most effectively, yeesh! :woozy_face: However, I’ve almost cracked it, and I believe it will make for much simpler logic in determining which page is currently being viewed and ensuring that the menu is open to that section of the menu by default.

Cool! I wondered what you thought that the back/next buttons. It is nice. And I think can be pretty easy to replicate. As will the copyright. Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo. There are different ways to handle either scenario, but curious your thoughts. Actually, come to think of it, the nicest, cleanest way might be to just have a separate markdown file for it, something like /docs/COPYRIGHT.md. That way you can format it and add links however you like, it’s easy for people to find and read in the source repo, and not closely coupled to any particular implementation.

Basically, I’m thinking about the ultimate configuration, when this becomes farmos.org, how do we breakdown the paths to the different projects’ documentation?

I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.

But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while. Again, probably best to just set aside some time to talk.

Gotcha - makes sense. Will frontmatter in farmOS/farmOS remove the need for this?

Along those lines, one thing we’ll need to do is outline what the update process will need to be - and what will be manual and what will be automatic. We’ve talked a bit about this in chat.

Specifically: when a change is made to the docs in farmOS/farmOS, what (if anything) is needed to trigger the update to be pulled into farmOS.org?

Hopefully it can be fully automated? Perhaps a Git webhook that fires when a commit is pushed to farmOS/farmOS, which triggers a GitHub Action in farmOS.org’s repo to rebuild via Gatsby?

If we need to make changes manually in both repos whenever edits are made that will become quite cumbersome. I love that right now I can merge a PR for farmOS.org via the GitHub UI and trust that it will be rebuilt and published automatically. Of course, pulling content from multiple other sources adds another layer of complexity to that.

One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that don’t have pages

If it makes it any easier, we can certainly add stub index.md pages for those, and just make that a requirement moving forward. Does that make sense? We would just expect each subdirectory to have an index.md which is used for the top-level parent (as well as perhaps being added as a link itself underneath, as in the “Development > Module > Getting started” page, for example (which is an index.md file).

Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo.

Each repo should have their own COPYRIGHT.txt (here is farmOS’s, which will be added to the 2.x branch via #3162606), which covers all code inside the repo, and that also covers the docs content. Perhaps we also need to include a creative commons notice somewhere in there - I hadn’t thought of that.

I think we should also have a dedicated footer on farmOS.org. And maybe we just make it a policy that any docs from remote repos being aggregated into farmOS.org must agree to making it available as creative commons as well?

Either way, it needs the standard trademark text (per trademark policy): Trademark - farmOS.org

I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.

But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while.

Cool cool! Yea agreed we can discuss - I think it makes sense to take it slow and steady.

Yes.

I wholeheartedly agree we really need a solution that automates new builds whenever there’s a change to the source repo.

I haven’t researched this fully yet, but I’m inclined to think we can set this up with a Netlify build hook, perhaps in conjunction with a simple GitHub Action in each source repo.

Yea, I thought of something similar too. I would probably like to support both the automatic generation of menu titles, plus a configuration option if you want to change it. That way, we don’t have to require an index.md in every source directory, but you can drop one in if you want to change the default title that’s generated from the pathname. Or something like that.

Right on. So to clarify, the copyright property in your current mkdocs.yml file covers two things: the CC license is for the docs, and TM is for farmOS itself. I’m thinking we split that up, and include the TM in the Gatsby repo, and CC in the docs directory somewhere.

As for the COPYRIGHT.txt files, it would be nice to generate a page for each project that displays its license, perhaps with a little additional explainer, but that might be stretch goal, since I think having it in the source repository is probably sufficient, without including it independently in each project’s documentation website. But if you think otherwise, let me know.

And just to clarify my own point now, I think perhaps our immediate target should be to target docs.farmos.org as the transitional home for all 2.x docs, for all projects (farmOS.js, farmOS.py, etc), so we can really test out the aggregation of the source repos and how they’ll all fit together, before deciding whether to eventually lump everything together with the rebranded farmos.org.

1 Like