A custom farmos agents.md / claude.md file for FarmOS

Development
1

So I’m getting my environment set up to try to vibe code in FarmOS. I did some work already on the firefox extension I already shared, but also want to be able to work in FarmOS directly.

In reviewing other large projects that use AI across the workflow, it seems important to have a core agents.md (or for anthropic claude.md) file that gives AI the necessary background learning so it doesn’t have to relearn past lessons. It’s like a global memory file that anyone can use to get started, hopefully (?) making AI assisted coding from non-experts easier.

So I worked on an initial AGENTS.md with claude 4.6 thinking mode and reviewed it. I made some minor changes but felt it did pretty good. I requested it focus on FarmOS 4, review the forum and farmos.org. It includes contribution guidelines, best practices, business logic, api info and gotchas, etc. etc.

I know this is a lot, but any review or updates would be helpful. If this is used a starting point for “How AI should behave and understand FarmOS” it’s potentially an important document.

My Prompt

Hi claude, I would like to create an agents.md file and agentic setup specifically for contributing code, issues and discussions to FarmOS (farmos.org, https://farmos.discourse.group).  It should follow the community norms and processes for contribution, discussion, issue creation, as well as Drupal's norms for core, contrib and community modules.  Mike Stenta is a good role model to follow, as he's the primary contributor.  My goal is to be able to have a strong agents.md file (and other history or reference data) that allows me, as a non-developer but active FarmOS user, to most easily contribute to existing FarmOS work and develop new ideas I can try myself and share.  This should thoroughly review the existing data structure, business logic, API and documentation, and historical coding norms so that any new contributions fit within what has been done before.  

This will require a thorough review of farmos.org and farmos.discourse.group, and following up with key links (especially in the last 1 - 2 years) on active threads and discussions.   In addition, I actively develop conventions.farm (https://gitlab.com/our-sci/conventions/common_farm_conventions) and related tools which utilize farmOS's data model.  That would also be good to review, though it's not farmOS directly.  I would also search and review other public projects which are creating best practices for agent-based interaction with large open sourced databases... what are the best practices generally, and what can we implement here that follows those?

If there seem like some very different paths forward which are supported by farmos or community norms, then please summarize and present them.  If the answer seems fairly obvious, please just continue until you have a completed product and summarize for me and others to review.  I plan to share the finished concept and files with the FarmOS community as a whole to get feedback as well.

The AGENTS.md File Claude created. Please review, comment and change!!!

  • (Nextcloud) You can download the file, modify and share back here
  • Sorry… It was too long to post here so this was the easiest way to share. I can also put this in a git repo to track changes that way if it’s most comfortable.
1 Like
2

Only got as far as the end of the “Assets” section, but I have some concerns so far…

Key 4.x changes (breaking from 3.x)

Misses the important schema changes and the changes to default installed modules.

Assets — Things being tracked (persistent records)

Misses compost and sensor.

Universal asset fields:

Includes “status” even though that’s been removed in 4.x.

Fixed vs. mobile assets

Misrepresents (IMHO) the nature of the “fixed” asset attribute such that a reader might reasonably infer it is only relevant to land assets.

3

I just learned about agents.md this week (https://agents.md/), so this post is timely @gbathree.

I will say, as I was reading about it earlier, and evaluating whether or not we should consider adding one to the farmOS repo, I ultimately decided that it probably wasn’t worth it for farmOS… (yet?)

My thinking was twofold:

  1. We’ve spent a lot of time crafting the documentation on farmOS.org, which is maintained in the farmOS repo’s docs directory. All of that is written in markdown, so I would assume it as readable as anything we would put in a dedicated agents.md. I would much rather direct agents to all of the work we’ve done already, which specifically tries to lay out how to develop farmOS modules (and where the overlap and divergences are from standard Drupal modules). And if that isn’t enough, IMO we should invest time in improving the docs for humans, knowing that LLMs will benefit from that too.

  2. This is the same consideration we have for all of our documentation: who is going to maintain it?

    • How do we know when updates to farmOS code require updates to agents.md?
    • Do we rely on an LLM to maintain it? Can we trust that it will be accurate?
    • What additional expectations (and problems) does introducing an agents.md imply? Will I need to debug and fix issues if someone tells me that it caused problems with their agent?

In summary, I think I struggle to see why projects would invest in maintaining an agents.md file, instead of (or in addition to) investing in quality documentation for humans. LLMs work by reading human language, after all. :slight_smile:

1 Like
4

I should say: even if we don’t add something to the farmOS repo, I think it’s a great idea to experiment with something like this and let others iterate on it! So +1 to putting it in a public Git repo! I would assume that it’s pretty easy to point your agent at that repo for its context…

5

We few notes:

@Symbioquine a human would also make mistakes, and it would be natural to note them and improve. That’s normal, so errors are expected - ai is definitely not magic or perfect. The difference between junk and something useful is review and improvement regardless.

@mstenta This references and pulls material from the core documentation, as well as the forum. It’s my experience that the core documentation has most, but certainly not all, the information needed to effectively, efficiently and usefully contribute (which is what this file is all about). It’s like saying as a person entering a company is provided the operating agreement, mission and vision and expected to understand what to do - of course not! There’s norms, culture, historical conversations, etc. are also really important. The goal of agents.md is to cover the key documentation and source information but also provide a wider scope. The reason it rewrites things (which I think is your concern) is that it has to summarize (shorten) things so that it can keep these rules in context (just like humans) and know when it needs to go check something out later. So 100% improving the human documentation is the priority - this is a secondary product really for AI that emerges from that and other key content.

It would be interesting to run a project with and without this (or another) agents.md file and see how much it improves (or doesn’t!) a development outcome.

Thanks for the feedback!

6

That makes sense. My concern is more about maintaining this file. Who updates it, when and why? What are the standard operating procedures around maintaining this, and who does that burden fall on?

Yea I’m curious about this too!

7

Here is a related curiosity/experiment:

What if you run the same prompt used to generate this file 10 different times over the course of 10 separate days? How will the outputs differ?

As I understand it, one of the challenges (and powers!?) of LLMs is that they are inherently non-deterministic. The same prompt never returns the same result.

So how do we judge what is the “right” content for this file?

8

I think the focus of my concern isn’t so much about “making mistakes” as the nature of the data transformation that’s occurring. As I understand it, there’s no actual “reasoning” (IMHO) going on in a classical sense. So the “intent” of the output is more or less just an artifact of “people tend to put these words near each other in response to this kind of input”.

In this case - w.r.t. the 4.x changes summary part - it “sees” that the bulk of the changes pertain to the code cleanup stuff (and some can certainly be ignored for the sake of a summary), but doesn’t have any way to reason that the parts most users care about are produced by a relatively small number of seemingly mundane changes among those. Specifically, users actually probably don’t care about most of the things it did list in the 4.x changes and probably do care that the data model schema changed and that they now have an extra step/wizard upon installing farmOS in order to have any modules available.

Similarly, the cost of trusting the other output I’ve looked over seems pretty high since you effectively already need somebody with near expert knowledge to look over it to spot the errors and misleading bits. Once you have that person spending that time, they could be spending it on improving the source material. e.g. Maybe there needs to be a condensed summary built-into the docs?

Moreover, that cost is paid in order to make it easier to spend even more in the future fixing that kind of mundane probabilistic output errors. Is that the future we want to sign up for?

As you can tell I’m very skeptical, but I’m also curious. It’s very impressive technology, no doubt. I just haven’t seen anything that tells me the output is an improvement on a human bringing their raw knowledge, skills, intention, and creativity to an endeavor. It definitely seems like it lowers the bar for creating mediocre derivative content though.

9

I would not necessarily maintain this document solely through AI. AI initiated it, and we need to find the missing pieces an improve it (as @Symbioquine began doing) - just like if a person initiated it and posted to the forum.

The way to think about it is more - imagine if an eager, capable but junior dev wants to contribute to FarmOS. They don’t have infinite memory, though they’ve read through the documentation and the forum and can reference that. They need their own personal notes that they can actually keep in their memory without having to look it up every time. That’s this, for AI.

I’ll respond to @Symbioquine below. re. the other stuff.

2 Likes
10

I just haven’t seen anything that tells me the output is an improvement on a human bringing their raw knowledge, skills, intention, and creativity to an endeavor. It definitely seems like it lowers the bar for creating mediocre derivative content though.

On this point, I couldn’t disagree more. FarmOS has an amazing core contributor community… but it’s focus is on items that matter to you all… which is not necessarily what matters to other farmers (or the majority of farmers).

IMO from the last 7 years, the greatest limitation to FarmOS is the inability for people to take advantage of the tremendous flexibility to do what they actually need to do. They often know what they need, and are very capable and smart of explaining what it is, but there is a moat of development time and effort between here and there. And thus, even with a powerful tool, engagement relative to potential is low (IMHO of course).

I think this AI stuff can allow people with domain knowledge to more actively contribute solutions they need, and thus make FarmOS more appropriately designed for them. I don’t think it’s duplicative or derivative content, it’s customized content driven by the person who needs it. It’s what open source really should be best for, but is limited by the current moat of development (and the broader development process).

Spreadsheets are sticky because it allowed everyone to be a developer (sounds silly, but in 1985 a spreadsheet was as powerful as a developer in many cases). Airtable is doing the same thing (start talking to farmers and you’ll find tons of them are moving to Airtable). We lost that empowerment in the age of SaaS and highly customized software. I’d love to see some of that come back.

All that I say in the box of ‘we should still be developing technology’… I’m questioning the role of technology writ large, and our ability to handle it without destroying ourselves. So don’t peg me here as an AI techno-utopian here… But in the paradigm and reality we have, these are my thoughts.

2 Likes
11

This describes the value of the agents.md file very succinctly - thanks @gbathree! I still haven’t played around with “agents” myself, but I see the power they have to help potential contributors across the “moat of development” as you described it.

That was my assumption as well - this would need to be a living document that is manually maintained alongside farmOS.

That’s why I hesitate adding it directly to the core farmOS repo. But I don’t think that it needs to be added to the core repo in order to still be useful, right? It can be maintained by those who are using it separately, just like any other “contrib” project.

This feels very similar to the discussions that led to the creation of the community blog. Maintaining detailed documentation is hard, so our solution as a community was to create a separation between “official docs” and “community blog posts”. We promise to keep the official docs updated alongside farmOS (and even that is probably not perfect), but we make it clear that blog posts (which may be more detailed/specific) might not be updated/relevant over time (or they might be, if the original authors decide to keep them updated). But if they don’t get updated, it’s OK - because they have a date and we can communicate that they are not maintained. High value proposition in the short term with a low maintenance cost in the long term. :slight_smile:

Maybe the best way to start would be to create a dedicated farmOS-agents.md Git repository, and a companion blog post that describes how to use it (and how to help maintain it). That gives us something we can easily point people to who are interested in experimenting.

12

Yes, I think you nailed it. I agree it’s not part of core FarmOS, but it’s a related and important piece of information which should be reviewed and validated by core devs and the community for people to use.

If you’d like to make a separate repo, I’m happy to contribute this as a starting point. Or I can provide the prompt that generated it.

Also worth doing testing it to see if it’s actually helping (things from ‘make a quickform…’ to ‘build a module which…’). That would help build confidence that it’s helping, and what types of things it can do well or not.

1 Like
13

I think it works best when repos are maintained by the people who are using them.

Therefore @gbathree I nominate you to start a repo and a blog post with guidance for how to use it and how to contribute to its ongoing usefulness. :slight_smile:

2 Likes
14

I totally agree that we need to be working more directly on solving first-order problems for real-world farming use-cases.

I see this (“it’s easy to just vibe-code X”) as another layer of the “everyone should learn to code” fantasy. At some level I’m bought into that too since I think working with code/software can be an excellent way to stretch ones brain - just like I’d recommend that everyone learn to juggle (physical objects in the air) at least a little.

I also don’t have a problem with having the power to make good software more distributed. I just worry about the sustainability of jumping to LLMS as an approach to that - in many ways…

We need to make sure that the lowered barriers for creating extra custom stuff - that I suspect will have more subtle and confounding problems than conventional human-developed software - won’t further isolate and fragment the technology-curious members of the farming community that would ideally otherwise be our ambassadors.

We need to make sure the message of KISS is strong. i.e. the arguments I was making in Organizing Farm Data with farmOS | farmOS (tl;dr; Figure out the minimum data you need to solve your real world problems and store it in the simplest, cheapest, and most conventional manner you can.)

And finally, I worry about the larger technology landscape. I see a massive dystopian intellectual land-grab occurring where the power to work with data and build abstract things with computers will be concentrated and commodified by proprietary technology (which is itself built on very shaky or outright unsound intellectual property theory). This is another step in the global concentration of wealth that has been playing out for centuries.

1 Like
15

Hi folks :waving_hand: - appreciating this discussion - @gbathree would love to chat about setting up a custom agents.md / claude.md file for OFN - I have to confirm with a few folks, but I think we’d be down to be guinea pigs on this, and could then report back here - Greg, if you want to explore this further ping me and I can work on connecting everyone

We are already starting to see agent contributors cropping up in the GOAT ecosystem already: From what I understand, LiteFarm have started working with this agent https://nedforgood.com/getting-started/ and are trying to figure out some solid processes, guardrails, and best practices etc

Not much to report yet, but it’s forked their code base, and I think setup on their slack channel