Title: Scaffold a new Gatsby site for farmOS.org & 2.x docs Submitted by: Jamie Gaehring OpenCollective account:Jamie Gaehring - Open Collective Cost: $600 Completion Date: 30 Apr 2021
Description
The goal of this project is to scaffold a new website for farmos.org, which can be used to host documentation from multiple GitHub repositories. Eventually the hope is that this site could also house more diverse content in the form of a blog, tutorials and other community-generated materials, as well as a richer, more marketable landing page to attract new users. Although those larger goals are not within the scope of this immediate proposal, part of the aim of this scaffolding project is to enable the gradual development of those features as a series of progressive enhancements, while we approach the launch of farmOS 2.0. I will build the new site using the Gatsby framework, as a more powerful and flexible static-site generator (SSG), along with Netlify for hosting, which combined should be able to achieve these short-term and long-term goals. Such a progression would be difficult or impossible to achieve with our current tool-chain of MkDocs and GitHub Pages.
Iām pretty pleased with it. Iāve tried to keep parity with what we have currently at docs.farmos.org. However, Iāve tweaked the colors to be a little more on-brand, and shrunk the width of the body text to be a little easier to read.
Some features still lacking:
Search
Proper slugs and/or a different home page
SEO
Copy button code blocks
Center text when thereās no ToC
farmOS icon
Of course, the big one is search, and Iām hoping to spend tomorrow evening working on that. The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because Iām not sure how we want to structure things once we start pulling docs from multiple source repos. Right now Iāve got a pretty sloppy redirect in place, which is painfully obvious the first time you load the page and get a flash of the Gatsby starter page, but thatās just a temporary fix. SEO should be made pretty easy with this plugin, Iāll try to bang that out before tackling search tomorrow. The rest of the features struck me as relatively minor and/or not worth pursuing seriously until we have a full rebranding effort in place.
I walked back on this a little, widening the main container from 75 chars to 90 chars, to allow for longer text strings like the one I mentioned here:
I also added word breaks for inline code blocks just to be safe.
Fixed.
Added some really basic meta tags, nothing fancy. But the structureās in place. I borrowed pretty liberally from this tutorial and put all the details in the gatsby-config.js file in root, for anyone interested. We can revisit this to tweak the description and keywords etc at a later date. Testing out the link, preview now:
Looks like the preview image is still broken, but we can tweak that.
I hoped to get to search tonight, but donāt think thatāll happen. I did find a nice guide, though, while looking for SEO stuff. Maybe Iāll give it another go later tonight or tomorrow.
Just fixed a major issue where all absolute links in markdown files were being prefixed with the site domain. Oops!
Iāll note that the check I wrote could probably be more robust, if we want to support more protocols than http and https; Iām just checking if the string starts with 'http'. I canāt think of any other protocols we would want to link to in our markdown files, but perhaps Iām overlooking something?
Great progress @jgaehring! Exciting to see this start to take shape!
Quick thoughts/observations/questions:
First, I love how you were able to keep things so similar to the current docs.farmos.org! Thatās impressive in itself!
Search would be great - but I could also see that being its own follow-up with a separate budget, if it seems like it might eat up too much this round. The link navigation is enough for now.
I noticed that āData modelā section is omitted? Is this intentional (do you have future plans in mind - perhaps in a more top-level section)?
The left nav items donāt stay āopenā when you click through to one of them, and it doesnāt highlight the one youāre currently looking at. Not sure how easy this would be, but the navigation is more cumbersome without it.
I like the āNextā and āBackā footer features that MkDocs+Material provided, but obviously not critical for this first pass.
Needs footer with Creative Commons and trademark text.
The other one that needs consideration is the slugs for the docs pages and/or home page, but I wanted to have a conversation on how best to proceed with that, because Iām not sure how we want to structure things once we start pulling docs from multiple source repos.
Happy to help think this though. Could you elaborate a bit on what the considerations are?
Yea, Iād at least like to take a stab at the core functionality, b/c I think weāll need to do that sooner or later, although I may skimp a bit on the styling. I actually really like the way Material for MkDocs styles and handles UX for search, but itās a lot of CSS complexity that might be hard to emulate or cleanly copy & paste. Probably not worth the trouble right now.
Ah, yes, I forgot to mention this. Up until now, Iāve been using a stub to reproduce the data from the mkdocs.yml file, without being entirely dependent on it. Because I havenāt updated this stub since early on, it is still missing the Data Model and other sections. As I think Iāve mentioned, Iād like to get to the point where instead of the mkdocs.yml file, we use Frontmatter in the markdown files themselves, because itās a more widely adopted standard and serves to decouple the source repos from the Gatsby repo. Iāve actually been sitting on a PR until now, which Iāve just opened so you can see the kind of changes I have in mind:
One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that donāt have pages, for instance, āDevelopmentā. Instead, what Iām doing for now, is parsing the pathname for the of those headingsā child paths, eg '/development/api/', to extract and capitalize the string 'Development' and use that for the heading. This could have some obvious shortcomings, which will probably eventually require some kind of additional configuration, either in the source repo or the Gatsby config, but I figured to punt on that for now.
Yea, Iām still putting the final touches on the navigation, replacing the stub file, parsing pathnames, etc, as I mentioned above. Coincidentally, this has been the most difficult part of the project to date, and I lost the better part of this weekend bogged down in recursive parsing algorithms for how to do this most effectively, yeesh! However, Iāve almost cracked it, and I believe it will make for much simpler logic in determining which page is currently being viewed and ensuring that the menu is open to that section of the menu by default.
Cool! I wondered what you thought that the back/next buttons. It is nice. And I think can be pretty easy to replicate. As will the copyright. Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo. There are different ways to handle either scenario, but curious your thoughts. Actually, come to think of it, the nicest, cleanest way might be to just have a separate markdown file for it, something like /docs/COPYRIGHT.md. That way you can format it and add links however you like, itās easy for people to find and read in the source repo, and not closely coupled to any particular implementation.
Basically, Iām thinking about the ultimate configuration, when this becomes farmos.org, how do we breakdown the paths to the different projectsā documentation?
I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.
But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while. Again, probably best to just set aside some time to talk.
Gotcha - makes sense. Will frontmatter in farmOS/farmOS remove the need for this?
Along those lines, one thing weāll need to do is outline what the update process will need to be - and what will be manual and what will be automatic. Weāve talked a bit about this in chat.
Specifically: when a change is made to the docs in farmOS/farmOS, what (if anything) is needed to trigger the update to be pulled into farmOS.org?
Hopefully it can be fully automated? Perhaps a Git webhook that fires when a commit is pushed to farmOS/farmOS, which triggers a GitHub Action in farmOS.orgās repo to rebuild via Gatsby?
If we need to make changes manually in both repos whenever edits are made that will become quite cumbersome. I love that right now I can merge a PR for farmOS.org via the GitHub UI and trust that it will be rebuilt and published automatically. Of course, pulling content from multiple other sources adds another layer of complexity to that.
One bit of data I realized could not be ported over from the mkdocs.yml to Frontmatter are the titles for the menu headings that donāt have pages
If it makes it any easier, we can certainly add stub index.md pages for those, and just make that a requirement moving forward. Does that make sense? We would just expect each subdirectory to have an index.md which is used for the top-level parent (as well as perhaps being added as a link itself underneath, as in the āDevelopment > Module > Getting startedā page, for example (which is an index.md file).
Although I was meaning to ask whether you think the copyright should reside in the source repo (ie, farmOS/farmOS) or in the Gatsby repo.
Each repo should have their own COPYRIGHT.txt (here is farmOSās, which will be added to the 2.x branch via #3162606), which covers all code inside the repo, and that also covers the docs content. Perhaps we also need to include a creative commons notice somewhere in there - I hadnāt thought of that.
I think we should also have a dedicated footer on farmOS.org. And maybe we just make it a policy that any docs from remote repos being aggregated into farmOS.org must agree to making it available as creative commons as well?
I could go on ad nauseam. Lots of options, probably necessitating a larger discussion, and some passage of time to see how the rest of the redesign shakes out.
ā¦
But we should also probably think in the nearer term, where farmos.org remains as it is, for 1.x docs, and we continue using docs.farmos.org for the 2.x docs, and the two sites live side-by-side for while.
Cool cool! Yea agreed we can discuss - I think it makes sense to take it slow and steady.
I wholeheartedly agree we really need a solution that automates new builds whenever thereās a change to the source repo.
I havenāt researched this fully yet, but Iām inclined to think we can set this up with a Netlify build hook, perhaps in conjunction with a simple GitHub Action in each source repo.
Yea, I thought of something similar too. I would probably like to support both the automatic generation of menu titles, plus a configuration option if you want to change it. That way, we donāt have to require an index.md in every source directory, but you can drop one in if you want to change the default title thatās generated from the pathname. Or something like that.
Right on. So to clarify, the copyright property in your current mkdocs.yml file covers two things: the CC license is for the docs, and TM is for farmOS itself. Iām thinking we split that up, and include the TM in the Gatsby repo, and CC in the docs directory somewhere.
As for the COPYRIGHT.txt files, it would be nice to generate a page for each project that displays its license, perhaps with a little additional explainer, but that might be stretch goal, since I think having it in the source repository is probably sufficient, without including it independently in each projectās documentation website. But if you think otherwise, let me know.
And just to clarify my own point now, I think perhaps our immediate target should be to target docs.farmos.org as the transitional home for all 2.x docs, for all projects (farmOS.js, farmOS.py, etc), so we can really test out the aggregation of the source repos and how theyāll all fit together, before deciding whether to eventually lump everything together with the rebranded farmos.org.
Finally got back to working on this over the weekend and knocked out what I think were the only remaining hard requirements, at least in terms of site features:
Added the CC license and farmOS trademark info to the footer. Might be nice to separate these two at some point in the future, so separate documentation sources could have separate licenses, while the trademark info always remains the same, but I think thatās a later consideration.
The main site nav on the left automatically opens to the page your are currently viewing, and highlights it. Because Gatsby uses SSR by default, the drawer itself will close after you click, and Iād like to smooth out that transition a little more nicely, but I think thatās a stretch goal.
I also spent way too long updating npm packages and the like since last time. I suppose itās worth bearing in mind that this is a drawback of using Gatsby or really any JS framework(s) for a site like this.
So the only hard and fast requirement I have on my docket is to work out the build process, so whenever thereās an update to a documentation source repo, the site will rebuild automatically to reflect those changes. As I said in my last comment, I think a Netlify build hook will probably suffice, although it might be necessary to add a small GitHub Action in each source repo that will fire that hook.
And a quick note on this:
I wound up using the mkdocs.yml file after all, while still retaining the parsing logic that generates the navigation tree based on the file structure alone if there is not a mkdocs.yml. Basically I realized there was more information encoded into one of the nav YAML property than I first ascertained, and it wasnāt worth trying to encode all that into some combination of Frontmatter and index.yml files to make that work. At least for now. I think itās still a good idea worth revisiting once weāve tried out using this with other source repos for a while.
For now, Iām gonna just close that PR though. Maybe we wonāt need the favicon in the source repo any more, and weāll probably replace the GitHub Pages action with something to trigger Netlify builds, but it seems worthwhile just starting fresh for those changes.
Anyways, I hope to have time in the next few nights to test out those build hooks and hopefully have this ready to deploy (finally!) sometime this week or next.
I set up a GitHub Action in my fork of farmOS 2.x to trigger a Netlify build hook to rebuild the main site whenever thereās a push to that branch (currently 2.x-netlify-build-docs, but it should eventually be changed to the 2.x branch):
It should also trigger a branch deploy at https://preview-farmos--gracious-brattain-bdd606.netlify.app/ if thereās a PR to that branch. I havenāt tried that out yet, but if someone wants to give it a shot by opening a PR on that branch of my fork, be my guest!
I think another stretch goal might be to add a conditional for the entire job so it only runs when there are changes to the docs/ directory. Once we get this workflow working just how we like, it would also be nice to set it up as a reusable workflow that all repos in the farmOS GitHub org can use for their documentation.
@mstenta, I might open a PR to farmOS/farmOS with just this workflow commit so we can play around with it a little more. It shouldnāt really hurt anything at this point, while it only runs on events for the 2.x-netlify-build-docs, and even then only just fires off a curl command to Netlify that shouldnāt effect any other workflows. We should also think about how we want to set up a more permanent Netlify deployment that can belong to farmOS, rather then my own personal Netlify account, and make sure the secret token and all are set up for the farmOS GitHub org.
As I said I might in the previous comment, I opened a pull request on the farmOS repository to add the GHA workflow that triggers the Netlify builds:
I did a little poking around to figure out what the matter was with those failed tests. At first it seemed to be whitespace issues, which I amended, but seems like thereās more going on in the sqlite tests. More details there.
From there I resumed work trying to figure out how to make the navigation drawer stay open, or at least close more gracefully, on page transitions. This opened up a whole can of worms regarding how the navigation is structured, such as how the navigation of a particular projectās documentation relates to the overall navigation for the site in general. Right now, itās hard to to really anticipate what this should look like, since it wonāt really be an issue until weāre using more than one source repository. But there are unavoidable knock-on effects which seem important to consider now, namely how do we aggregate and structure the data from the source repositories and parcel out portions of that data to the pages that need it. Iāve got a WIP branch started, but Iām probably going to shelve that for now, so I donāt have to deal with that can of worms just yet.
Iāve been hesitant to write up issues for this kind of stuff in my own GH repo because I hope we can move this all into a farmOS org repo, but looking further into it now, it seems GitHub has a straightforward procedure for transferring repositories, so from here on Iāll open more detailed issues there.
Suffice to say, thereās still some low-hanging fruit I could chip away at, and those failing tests to address, but it would be good to do a quick review and discuss next steps, if thereās a good time to do so in the following week.
Iāve consolidated these down to a āMust-havesā milestone, for easier reference. Thereās one more issue that might go in there, but as I explain there, it involves a lot of decisions we might not be ready to make just yet.
I was going to add some more concrete next steps, too, but Iāll have to come back and update this later tonight with those.
Rename my repo from jgaehring/farmos.org-2.x to jgaehring/docs.farmOS.org.
Transfer my repo to the farmOS GitHub organization.
Set up a farmOS Netlify acct.
Add Netlify webhook tokens to the farmOS organization secrets.
Merge PR #456, which I believe should trigger the first full build.
The only thing after that is to set up the domain registry for docs.farmos.org to use the Netlify host, but we can hold off on that for as long as we wish. The build can just live at the Netlify domain while we test it out, with the Mkdocs build still working as usual on docs.farmos.org. Then, when weāre happy with how everything is working, we can flip the switch.
With regards to step #2, this is intended to distinguish it from the farmOS/farmOS.org repo, at least for the time being. Iām very open to other suggestions, but this seemed to make sense given the domain this will first be deployed to is docs.farmos.org. @mstenta has suggested we merge the two repos and just use separate branches, which would obviate the need for this, in order to preserve the git history while also maintaining continuity. I still think that would be great, but Iād like to wait a little longer to do that. At the very least, I donāt think I can merge them prior to transferring my repo to the farmOS organization and preserve the issues Iāve opened up there. I believe that can only be done with issues in two projects belonging to the same organization. So renaming it for the meantime makes the most sense to me, and then maybe hold off on merging the repos until we know weāre going to start deploying this to farmos.org.
But I think thatās it! Obviously, thereās lots more we can and will do, particularly when we get to the rebranding stage, but I think for the basic scaffolding phase, we should be set. I should have the āmust-havesā done by the weekend if not sooner, so perhaps we can get started on transferring the repo sometime next week.
Iāve managed to bang out all the āmust-havesā over the last couple weekends, and also dug into some of the deeper issues with navigation, which Iām finally pleased with.
Iām going to start an agenda for the Monthly Call this Wednesday, and put this on the docket. Iād really love to move the repo over to the farmOS org though as soon as we can.
However, Iāve almost cracked it, and I believe it will make for much simpler logic in determining which page is currently being viewed and ensuring that the menu is open to that section of the menu by default.
