MQTT history without exposing your broker

Why a collector rather than a cloud connection

A Mosquitto on a home network or a plant LAN sits behind NAT — nothing on the internet can dial into it. And the people most likely to want MQTT history are exactly the people whose broker is unreachable from outside.

So the collector connects outward instead: outward to your broker on the local network, outward over HTTPS to the API. No port forwarding, no inbound firewall rule, no broker exposed to the internet.

It also means a dropped uplink is not automatically a lost reading. The collector buffers to disk and replays when the connection returns — within the bound described below.

1. Get an API key

Create an account and an API key first. The free plan is €0 and takes an email address — no card — and gives you 5 tags and 50,000 readings a day. Then create the key on the Settings page in Tag Historian Explorer — and the scope you give it is not a detail.

The key is shown once. Copy it straight into the container’s environment — if you lose it, generate a new one rather than hunting for the old one.

2. Start it once to get a config file

The collector reads its configuration and keeps its offline buffer in one directory, /config, on a Docker volume. Start it once on a new, empty volume and it writes a complete, commented example configuration there, says what to edit, and stops:

docker run --name tag-collector -v tag-collector:/config ghcr.io/softi-dev/tag-historian-clients/collector:latest

It ends with exit code 78, “configuration required”. That is the expected result here, not a failure: nothing has connected anywhere yet. The stopped container stays behind so the next step can reach the volume through it.

3. Edit the config

Copy the file out of the volume, so you can edit it with any editor:

docker cp tag-collector:/config/appsettings.json appsettings.json

In appsettings.json, fill in your broker under MqttSources: its Host, and the Topics to subscribe to. Every topic and JSON field becomes a tag against your plan, so subscribe narrowly (tag count is what you pay for). Then delete the example entry in OpcServers if you are not using it. Every value still in <angle brackets> is a placeholder; everything else is a default, with a comment saying what it does. Trimmed to what matters, the edited file comes down to this:

{
  "TagHistorian": {
    "ApiUrl": "https://api.taghistorian.com",
    "ApiKey": ""
  },
  "MqttSources": [
    {
      "Name": "broker",
      "Host": "192.168.1.10",
      "Topics": [ "zigbee2mqtt/+" ]
    }
  ],
  "OpcServers": []
}

Leave ApiKey empty. The key goes into the container's environment in the next step, so it is never stored on the volume. Then copy the file back and remove the first-start container:

docker cp appsettings.json tag-collector:/config/appsettings.json
docker rm tag-collector

If you run docker with sudo on Linux, use it for all of these, and edit the copied file with sudo as well: a file docker copied out as root belongs to root.

4. Run it

docker run -d --name tag-collector --restart unless-stopped -v tag-collector:/config -e TagHistorian__ApiKey="<your-api-key>" ghcr.io/softi-dev/tag-historian-clients/collector:latest

Follow it with docker logs -f tag-collector. Without a key the collector collects nothing and says so in the log, and a key still reading <your-api-key> stops it with exit code 78 and the name of the variable. Both on purpose: a missing key is a mistake to fix now, not an outage to wait out. Left to run, the collector would have every upload refused with a 401, which it treats like a revoked key, holding the readings on disk and retrying, so a forgotten key would look like a running collector until the buffer reached its 7-day / 512 MiB bound and began dropping the oldest readings.

A placeholder left in the file also stops it with exit code 78 and the name of the setting, and so does a file that no longer parses after the edit, with the line and position where the parser gave up; your file is never overwritten. Under --restart unless-stopped such a container shows as Restarting (78) in docker ps, and the log says why. To change the configuration later, copy it out and back in the same way and run docker restart tag-collector.

Or with compose, once steps 2 and 3 have created the volume and put the config in it:

services:
  collector:
    image: ghcr.io/softi-dev/tag-historian-clients/collector:latest
    container_name: tag-collector
    restart: unless-stopped
    volumes:
      - tag-collector:/config
    environment:
      TagHistorian__ApiKey: "<your-api-key>"

volumes:
  tag-collector:
    external: true

Every command here is one line, so it pastes the same into bash, zsh or PowerShell. The image is built for linux/amd64 and linux/arm64: the same commands run on a PC, on Docker Desktop for Windows or macOS, and on an ARM box with a 64-bit OS such as a Raspberry Pi or an industrial gateway.

Updating the collector

A container keeps the image it was created from, so docker restart tag-collector starts the same collector again. To move to the newest, pull it and create the container again on the same volume; the configuration and the readings not yet sent stay where they are:

docker pull ghcr.io/softi-dev/tag-historian-clients/collector:latest
docker stop tag-collector
docker rm tag-collector
docker run -d --name tag-collector --restart unless-stopped -v tag-collector:/config -e TagHistorian__ApiKey="<your-api-key>" ghcr.io/softi-dev/tag-historian-clients/collector:latest

with your key in place of <your-api-key>, as before. With compose, docker compose pull and then docker compose up -d do the same. Tag files, browse and Modbus devices need collector 1.0.47 or later: an older one ignores TagsFile and ModbusDevices without a word, so update a collector you started before then.

On Windows, without Docker

Where IT does not allow Docker, on the SCADA PC or a small plant's Windows server, the same collector runs as a Windows service: one self-contained program for 64-bit Windows 10, 11 or Server 2016 and later, with nothing else to install. It reads the same appsettings.json, from C:\ProgramData\Tag Historian\Collector instead of a volume, and keeps its offline queue and certificates beside it.

Download the Windows zip and check it against the SHA-256 checksum published beside it.

In an elevated PowerShell, in the folder the zip was extracted to:

powershell -ExecutionPolicy Bypass -File .\install.ps1

install.ps1 copies the program to C:\Program Files\Tag Historian\Collector and registers the service TagHistorianCollector: automatic start, restarted a minute after a crash, running as its own virtual account NT SERVICE\TagHistorianCollector, which is granted write access to its data folder and nothing else. The data folder is readable by administrators and the service only, because your API key goes in it. Then it starts the service once, which writes the example configuration and stops, as the container's first start does.

Edit the file from the same elevated prompt, and put your Write key in ApiKey: on Windows the key goes in the file, because a service does not see a system environment variable set after Windows started. Then start the service and follow its log:

notepad "C:\ProgramData\Tag Historian\Collector\appsettings.json"
Start-Service TagHistorianCollector
Get-Content -Wait -Tail 20 "C:\ProgramData\Tag Historian\Collector\logs\collector.log"

logs\collector.log in the data folder carries the lines docker logs would show, and warnings and errors also go to the Application event log under Tag Historian Collector. An empty ApiKey, a placeholder left in, or a file that no longer parses stops the service with the reason in both places, and it stays stopped instead of restarting every minute.

To update, run install.ps1 from a newer zip's folder the same way: it stops the service, replaces the program and starts it again, and keeps the data folder. The zip's name carries the collector's version; tag files, browse and Modbus devices need 1.0.47 or later.

The program and scripts are not signed yet. Started from Explorer, the program gets “Windows protected your PC”: choose More info, then Run anyway. The -ExecutionPolicy Bypass above is what lets the unsigned install script run, for that one run. On Windows 11 with Smart App Control turned on, unsigned programs are blocked outright; use the container there until the collector is signed.

The same volume that carries the config also holds the store-and-forward queue (/config/queue), so buffered readings survive a container restart or an image upgrade for free. It is the same image that speaks OPC UA — one collector, both protocols.

Topics become tag names

Slashes become dots, and StripTopicPrefix removes a routing prefix you do not want in your tag list forever. A prefix only matches a whole segment, so stripping tele leaves television/... alone.

From collector 1.0.48, a tag name is collected from one source only. If an OPC UA server, a Modbus device or another MQTT or Sparkplug B source already collects the name a topic makes, that topic's readings are not sent and the log says so once, naming both places; TagPrefix keeps two sources apart.

Payloads

Four shapes are understood, because those are the four real devices send.

Booleans are stored as 1 and 0 rather than as a state, so they aggregate into something useful: an hour in which a pump ran fifteen minutes averages to 0.25, which is its duty cycle.

A payload with nothing numeric in it stores nothing at all. Availability topics, status strings and JSON with only text in it are counted and reported in the log, not turned into zeroes. A zero is a reading; “I could not read this” is not, and a chart full of invented zeroes is worse than a chart with a gap.

Arrays are not expanded. An array index is not a stable identity — today's element 0 is tomorrow's element 1, and the tag would quietly start describing a different thing.

Timestamps

MQTT carries no timestamp of its own, so arrival time is used by default. If your device puts one in the payload, name the field with "TimestampField": "ts". Both ISO-8601 strings and Unix seconds are understood. If the field is missing from a message, that message falls back to arrival time rather than being backdated to something invented. The field itself never becomes a tag.

What happens when the internet drops

When the uplink returns, the buffer drains oldest first, in uploads of up to BatchSize readings (default 5000, the most the API takes in one request). The API answers an upload once it is in its write-ahead log, which commits about every two seconds, so a backlog drains at up to about 2 400 readings a second: a full day of a Business plan's allowance in about 34 minutes, of a Pro plan's in under four. New readings queue behind the backlog and arrive in order once it has gone. Replayed readings count towards the daily allowance of the day they arrive, like any others; if it runs out on the way, the collector holds the rest and carries on after UTC midnight.

Duplicates from QoS 1

QoS 1 is at least once: a broker that does not see your acknowledgement sends the message again. There is no dedup step — nothing compares an arriving reading against one already on disk.

What absorbs a redelivery in practice is deadband compression, which is on by default. A tag stores a reading when it differs from the last stored value by more than CompressionDeadband — default 0.001, a fraction of the last stored value, not a distance in engineering units — or when MaxCompressionTimeSpanSeconds has elapsed, default 1 hour. A redelivered message carries the identical value, so the change is zero and it is compressed away.

But note what that depends on:

• Turn compression off on a tag and duplicates are stored, as two points a few milliseconds apart with the same value. EnableCompression = false means store every sample, and a redelivery is a sample.
• The same rule swallows a genuine unchanged reading, which is intended — that is what MaxCompressionTimeSpanSeconds exists to bound, so an unchanging tag still gets a periodic sample.
• Discrete values (integers, booleans) are never deadbanded — they store on any change — but an exact repeat is still a non-change, so a redelivery is absorbed there too.

Aggregates are unaffected either way: an hour's average, minimum, maximum and count are computed from what was stored.

Because the deadband is fractional, do not size it against your unit. 0.5 is not “half a bar” — it means store only on a move larger than 50%, which for most process signals discards nearly everything. Useful values are small: 0.001 stores on a move over 0.1%, 0.01 on a move over 1%. The range is 0 to 1, and a number outside it is refused with an error that says what the number means, so one picked as if it were engineering units fails loudly instead of quietly suppressing almost every sample.

The deadband is the tag's, not the source's: there is no deadband in the MQTT settings below, and every tag starts at 0.001. The Explorer cannot change it yet; the tags API can, with an Admin key, as the OPC UA guide shows.

Settings

Tag count is what you pay for

Every distinct tag counts against your plan, and a wildcard subscription can create a great many very quickly — # on a busy broker will happily discover hundreds. Cost tracks tag count and not message volume, so subscribe to the topics you want rather than everything and filter later.

Once you are at your plan's limit, existing tags keep accepting readings — going over your plan does not silence instruments that are already running. But the refusal is coarser than one tag, and it is worth knowing the shape of it.

The collector uploads in batches, and the batch endpoint is all-or-nothing: a batch containing one over-quota new tag is rejected whole, with 402 naming that tag. Nothing in it is stored. What happens next is the queue doing its job:

1. The rejected batch goes back on the disk queue — nothing is thrown away.
2. The sender halves the batch until the offending record is alone; the halves without it are accepted on the way, so every other reading gets through. A record in a batch of 5000 is found in a few dozen requests.
3. Sent on its own, it is tried RejectedBatchRetries times (default 3) and then written to <QueuePath>/poison/rejected-*.jsonl, and the queue moves on. It is set aside as recoverable JSON lines, not discarded.

So the honest version: existing tags keep flowing, at reduced throughput, while over-quota readings accumulate in poison/ rather than vanishing. This does not self-heal — every fresh reading of the over-quota tag repeats the cycle — so either raise the plan or narrow the subscription. In the log, store-and-forward: API rejected a batch ... with 402 is the line to grep for.

The line protocol endpoint is the deliberate exception: it drops only the over-quota points and stores the rest with a 422, because Home Assistant and Telegraf treat a whole-request failure as a reason to retry everything forever.

What this is not

• We do not host a broker. You keep yours. There is no mqtt.taghistorian.com to point a device at.
• We do not publish back. Ingestion only; the collector never writes to your broker.
• Bridging is normal. If you would rather not point devices at a second broker, the collector simply subscribes to the one you have. Your local automation is untouched — it is one more subscriber.