Ecowitt weather station integration

[paul121]

Last year for Christmas my parents asked for a simple weather station/sensor they could use to monitor a few indoor and outdoor temperatures and potentially expand in the future. After doing some research I ultimately decided to try the Ecowitt suite of sensors, consoles and displays. These are cool because you can purchase only the components you want, have many available sensors (rain, wind, solar rad, soil, air quality, etc…), and displays and consoles that provide some “smart IoT/API” capabilities.

I ordered a few outdoor/indoor temp + humidity sensors and the WN1980 display. Everything was easy to setup and continued to work flawlessly after many months. So a few months ago I decided to get the same setup for myself, also getting a rain gauge to try out. As expected, these were also easy to setup and have been working well.

Now for the fun part… to start playing with the API! And hopefully get things connected to farmOS My ultimate goal is to have a local-first integration that doesn’t require sending data to the Ecowitt, Weather underground or other “cloud” weather services. Ideally my local network can collect + process data from the console before sending directly to farmOS. I’m also quite interested in storing this data inside a self-hosted instance of the Influxdb time-series database that I’m already quite familiar with.

I started to look at the IoT/API capabilities while doing my initial research but didn’t dig in too much. It turns out these Ecowitt devices (and many other brands like Ambient) are re-branded from a Chinese manufacturer called “Fine Offset”. A great WX forum post explains this in great detail. The latest consoles are stated to be compatible with WeeWx, weather services like weather underground, and also provide a locally accessible REST API with an option to send data to remote “custom” endpoints.

WeeWx

(TLDR; I actually never tried to run WeeWx, sorry folks.)

To start, I began looking at WeeWx:

WeeWX is a free, open source, software program, written in Python, which interacts with your weather station to produce graphs, reports, and HTML pages. It can optionally publish to weather sites or web servers. It uses modern software concepts, making it simple, robust, and easy to extend. It includes extensive documentation.

This seemed promising - it already supports Ecowitt stations, provides access to data, and could potentially relay/send data to farmOS or Influxdb. I saw there is a WeeWx docker image and an additional WeeWx plugin for Influxdb. This was looking good, but I knew the docker image wouldn’t include the plugins I wanted and would need yet more work… And WeeWx has a daunting configuration file I was not excited to figure out… It was about this time when I started looking more into the Influxdb integration and realized that I really wanted this much more than I wanted WeeWx… and wondered if there was an easier way…

Telegraf + Influxdb

Then I found this slocan/weatherDashboard repo for Ecowitt + Telegraf + Influxdb + Grafana. It’s a really elegant solution.

1. Ecowitt is configured to post data (in a form-urlencoded format) directly to a Telegraf endpoint
2. Telegraf processes this data & sends to influxdb
3. Grafana is configured with dashboards that retrieve data from influxdb

I knew there were docker images for influx + telegraf. I assembled a docker compose file and was up an running. It’s really quite nice because you can build the Telegraf configuration file from inside the Influxdb UI.

Docker compose

version: '3'

services:
  influxdb:
    image: influxdb:2.7-alpine
    volumes:
      # Mount for influxdb data directory and configuration
      - influxdbv2:/var/lib/influxdb2:rw
    ports:
      - "8086:8086"
  telegraf:
    image: telegraf:1.25-alpine
    depends_on:
      - influxdb
    # Replace the command to use config provided by influxdb. Follow steps in Influxdb UI to create the config.
    command: telegraf --config http://influxdb:8086/api/v2/telegrafs/{telegraf ID}
    environment:
      INFLUX_TOKEN: {create access token with read access to telegraf config, write access to influx database}
    ports:
      - "7080:7080"

volumes:
  influxdbv2:

Telegraf config

# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true
[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]

  ## Token for authentication.
  token = "$INFLUX_TOKEN"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "default"

  ## Destination bucket to write into.
  bucket = "default"

  ## The value of this tag will be used to determine the bucket.  If this
  ## tag is not set the 'bucket' option is used as the default.
  # bucket_tag = ""

  ## If true, the bucket tag will not be added to the metric.
  # exclude_bucket_tag = false

  ## Timeout for HTTP messages.
  # timeout = "5s"
[[inputs.http_listener_v2]]
  ## Address and port to host HTTP listener on
  service_address = ":7080"

  ## Path to listen to.
  paths = ["/telegraf"]

  ## HTTP methods to accept.
  methods = ["POST", "PUT"]

  ## maximum duration before timing out read of the request
  # read_timeout = "10s"
  ## maximum duration before timing out write of the response
  # write_timeout = "10s"

  ## Maximum allowed http request body size in bytes.
  ## 0 means to use the default of 524,288,000 bytes (500 mebibytes)
  max_body_size = "1MB"

  ## Data format to consume.
  ## Part of the request to consume.  Available options are "body" and
  ## "query".
  data_source = "body"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "form_urlencoded"
  form_urlencoded_tag_keys = ["stationtype", "model", "PASSKEY"]

Configure Ecowitt to POST to the telegraf endpoint, and data gets piped to influx!

Telegraf + farmOS

So this got me wondering… could I use Telegraf to post directly to farmOS? Turns out this config was quite easy with the HTTP output plugin + JSON formatting tweaks. Telegraf streams all data from the Ecowitt station to a single sensor asset URL in farmOS, creating separate data streams for each variable/sensor that the Ecowitt reads. And telegraf is able to do this while simultaneously sending to influxdb, too!

Telegraf farmOS config

# Generic HTTP write listener
[[outputs.http]]
  ## Sensor asset URL to send to
  url = "https://farmos/asset/d672697d-9e1f-4650-b2b8-69503b6feb21/data/basic?private_key=1234"

  ## Timeout for HTTP message
  timeout = "20s"

  ## HTTP method, one of: "POST" or "PUT" or "PATCH"
  method = "POST"

  ## Data format to output.
  ## Each data format has it's own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  data_format = "json"

  ## The resolution to use for the metric timestamp.  Must be a duration string
  ## such as "1ns", "1us", "1ms", "10ms", "1s".  Durations are truncated to
  ## the power of 10 less than the specified units.
  json_timestamp_units = "1s"

  ## A [JSONata](https://jsonata.org/) transformation of the JSON in
  ## [standard-form](#examples). Please note that only version 1.5.4 of the
  ## JSONata is supported due to the underlying library used.
  ## This allows to generate an arbitrary output form based on the metric(s). Please use
  ## multiline strings (starting and ending with three single-quotes) if needed.
  json_transformation = '''
  $merge([{"timestamp": timestamp}, fields])
  '''

  ## Use batch serialization format (default) instead of line based format.
  ## Batch format is more efficient and should be used unless line based
  ## format is really needed.
  use_batch_format = false

Ecowitt + farmOS

But then this got me wondering… could I just configure Ecowitt to send this data directly to farmOS? I realized that Ecowitt is sending a data payload as application/x-www-form-urlencoded which is pretty standard. It’s not quite the same JSON structure we currently expect but a simple change in the endpoint that processes this request could easily handle both JSON or form-urlencoded data payloads and likely be a general improvement as well. I gave it a go… and it works**!

**The only issue is that the Ecowitt station seems to have a limit on how long the custom URL “path” can be configured to… In a path like /asset/{uuid}/data/basic?private_key={key} the sensor asset UUID + default private key are too long. Even shortening the UUID to just a simple asset ID requires a shorter private key :-/. But this makes me think, maybe it would be nice to allow either an ID or UUID in the path when POSTing data to sensors

With this approach a few additional “empty” data streams are created because the Ecowitt includes non-numeric data fields like stationtype, model, passkey in this data payload. But the empty data streams aren’t a huge deal. It would also be quite easy to make a module specifically for ecowitt sensors that provides a separate endpoint like /asset/{uuid}/data/ecowitt?private_key={key} to process these specific requests more accurately. Then this would really mean ecowitt stations work “out of the box” with farmOS, no intermediary processing or other infrastructure to manage… that is quite nice.

Conclusion…

I’m still pretty interested in building out the ecowitt + telegraf + influxdb + farmOS integration further… the additional flexibility influxdb provides for querying and aggregation from many different types of sensors/data sets is appealing. And with the addition of Telegraf to help facilitate this data ingestion to influxdb (or directly to farmOS) I think these are some cool pieces for a more generalized approach. But still, hard to beat just configuring your sensor out of the box to send to farmOS!

Curious if anyone by chance has experience using Ecowitt or other Fine Offset sensors? Does this seem interesting to others? I could be motivated to build such a module if there was more general interest, otherwise might just focus on Influxdb pieces.

Ecowitt has also recently announced an automated irrigation solution that integrates (garden/smaller scale) with many of these pieces I mentioned above. I’m not sure how exactly it works and to what degree the API might be integrated with this… but it’s another exciting thing to think about! WittFlow smart water timer – Ecowitt