MQTT history without exposing your broker

Why a collector rather than a cloud connection

A Mosquitto on a home network or a plant LAN sits behind NAT — nothing on the internet can dial into it. And the people most likely to want MQTT history are exactly the people whose broker is unreachable from outside.

So the collector connects outward instead: outward to your broker on the local network, outward over HTTPS to the API. No port forwarding, no inbound firewall rule, no broker exposed to the internet.

It also means a dropped uplink is not automatically a lost reading. The collector buffers to disk and replays when the connection returns — within the bound described below.

1. Get an API key

Create an account and an API key first. The free plan is €0 and takes an email address — no card — and gives you 5 tags and 50,000 readings a day. Then create the key on the Settings page in Tag Historian Explorer — and the scope you give it is not a detail.

The key is shown once. Copy it straight into the container’s environment — if you lose it, generate a new one rather than hunting for the old one.

2. Write a config file

The collector reads its configuration and keeps its offline buffer in one directory, /config, so a single mounted volume is the whole install. This one subscribes to a Mosquitto on a home network:

{
  "TagHistorian": {
    "ApiUrl": "https://api.taghistorian.com",
    "ApiKey": ""
  },
  "MqttSources": [
    {
      "Name": "home",
      "Host": "192.168.1.10",
      "Topics": [ "zigbee2mqtt/+" ]
    }
  ],
  "OpcServers": []
}

Leave ApiKey empty in the file and pass it as an environment variable instead. The collector refuses to start without a key — on purpose: a field install with a forgotten key would otherwise get a 401 on every POST and grow its retry buffer until the box ran out of memory.

3. Run it

docker run -d --name tag-collector \
  -v mydata:/config \
  -e TagHistorian__ApiKey="<your-api-key>" \
  --restart unless-stopped \
  ghcr.io/softi-dev/tag-historian-clients/collector:latest

Or with compose:

services:
  collector:
    image: ghcr.io/softi-dev/tag-historian-clients/collector:latest
    restart: unless-stopped
    volumes:
      - mydata:/config
    environment:
      TagHistorian__ApiKey: "<your-api-key>"

volumes:
  mydata:

The same volume that carries the config also holds the store-and-forward queue (/config/queue), so buffered readings survive a container restart or an image upgrade for free. It is the same image that speaks OPC UA — one collector, both protocols.

Topics become tag names

Slashes become dots, and StripTopicPrefix removes a routing prefix you do not want in your tag list forever. A prefix only matches a whole segment, so stripping tele leaves television/... alone.

Payloads

Four shapes are understood, because those are the four real devices send.

Booleans are stored as 1 and 0 rather than as a state, so they aggregate into something useful: an hour in which a pump ran fifteen minutes averages to 0.25, which is its duty cycle.

A payload with nothing numeric in it stores nothing at all. Availability topics, status strings and JSON with only text in it are counted and reported in the log, not turned into zeroes. A zero is a reading; “I could not read this” is not, and a chart full of invented zeroes is worse than a chart with a gap.

Arrays are not expanded. An array index is not a stable identity — today's element 0 is tomorrow's element 1, and the tag would quietly start describing a different thing.

Timestamps

MQTT carries no timestamp of its own, so arrival time is used by default. If your device puts one in the payload, name the field with "TimestampField": "ts". Both ISO-8601 strings and Unix seconds are understood. If the field is missing from a message, that message falls back to arrival time rather than being backdated to something invented. The field itself never becomes a tag.

What happens when the internet drops

Duplicates from QoS 1

QoS 1 is at least once: a broker that does not see your acknowledgement sends the message again. There is no dedup step — nothing compares an arriving reading against one already on disk.

What absorbs a redelivery in practice is deadband compression, which is on by default. A tag stores a reading when it differs from the last stored value by more than CompressionDeadband — default 0.001, a fraction of the last stored value, not a distance in engineering units — or when MaxCompressionTimeSpanSeconds has elapsed, default 1 hour. A redelivered message carries the identical value, so the change is zero and it is compressed away.

But note what that depends on:

• Turn compression off on a tag and duplicates are stored, as two points a few milliseconds apart with the same value. EnableCompression = false means store every sample, and a redelivery is a sample.
• The same rule swallows a genuine unchanged reading, which is intended — that is what MaxCompressionTimeSpanSeconds exists to bound, so an unchanging tag still gets a periodic sample.
• Discrete values (integers, booleans) are never deadbanded — they store on any change — but an exact repeat is still a non-change, so a redelivery is absorbed there too.

Aggregates are unaffected either way: an hour's average, minimum, maximum and count are computed from what was stored.

Because the deadband is fractional, do not size it against your unit. 0.5 is not “half a bar” — it means store only on a move larger than 50%, which for most process signals discards nearly everything. Useful values are small: 0.001 stores on a move over 0.1%, 0.01 on a move over 1%. The range is 0 to 1; values above 1 are accepted without an error today and suppress almost every sample, so a number picked as if it were engineering units fails quietly rather than loudly.

Settings

Tag count is what you pay for

Every distinct tag counts against your plan, and a wildcard subscription can create a great many very quickly — # on a busy broker will happily discover hundreds. Cost tracks tag count and not message volume, so subscribe to the topics you want rather than everything and filter later.

Once you are at your plan's limit, existing tags keep accepting readings — going over your plan does not silence instruments that are already running. But the refusal is coarser than one tag, and it is worth knowing the shape of it.

The collector uploads in batches, and the batch endpoint is all-or-nothing: a batch containing one over-quota new tag is rejected whole, with 402 naming that tag. Nothing in it is stored. What happens next is the queue doing its job:

1. The rejected batch goes back on the disk queue — nothing is thrown away.
2. The sender drops to one record per request, which isolates the offending tag and lets every other reading through individually.
3. After RejectedBatchRetries attempts (default 3) the offending record is written to <QueuePath>/poison/rejected-*.jsonl and the queue moves on. It is set aside as recoverable JSON lines, not discarded.

So the honest version: existing tags keep flowing, at reduced throughput, while over-quota readings accumulate in poison/ rather than vanishing. This does not self-heal — every fresh reading of the over-quota tag repeats the cycle — so either raise the plan or narrow the subscription. In the log, store-and-forward: API rejected a batch ... with 402 is the line to grep for.

The line protocol endpoint is the deliberate exception: it drops only the over-quota points and stores the rest with a 422, because Home Assistant and Telegraf treat a whole-request failure as a reason to retry everything forever.

What this is not

• We do not host a broker. You keep yours. There is no mqtt.taghistorian.com to point a device at.
• We do not publish back. Ingestion only; the collector never writes to your broker.
• Bridging is normal. If you would rather not point devices at a second broker, the collector simply subscribes to the one you have. Your local automation is untouched — it is one more subscriber.