Sparkplug B history from your own broker

What it does, and what it does not

• Subscribes, on your broker. To spBv1.0/<group>/# for each group you name, or to all of spBv1.0/#, plus spBv1.0/STATE/#. Your edge nodes and every other consumer on the broker — Ignition, a SCADA package — are untouched; it is one more subscriber. There is no Tag Historian broker to point anything at.
• Connects outward only. To the broker on your network and to the API over HTTPS. No port forwarding, no inbound firewall rule.
• Reads the birth certificates. NBIRTH and DBIRTH carry each metric's name, alias and datatype; data messages after them carry an alias and a value. Births are not retained, so a collector that starts while the plant is running asks each edge node to publish its birth again — a rebirth request — rather than waiting for one that may be months away.
• Stores numbers and booleans. Integers, floats, doubles, booleans and date-times. Strings, datasets, arrays, files and templates are counted and listed, never stored, and never cost a tag.
• Never writes to your equipment. The rebirth request is the only message it publishes to an edge node, and host STATE only if you ask for it (below).

1. Get an API key

Create an account and an API key first. The free plan is €0 and takes an email address — no card — and gives you 5 tags and 50,000 readings a day. Then create the key on the Settings page in Tag Historian Explorer — and the scope you give it is not a detail.

The key is shown once. Copy it straight into the container’s environment — if you lose it, generate a new one rather than hunting for the old one.

2. Start it once to get a config file

The collector reads its configuration and keeps its offline buffer in one directory, /config, on a Docker volume. Start it once on a new, empty volume and it writes a complete, commented example configuration there, says what to edit, and stops:

docker run --name tag-collector -v tag-collector:/config ghcr.io/softi-dev/tag-historian-clients/collector:latest

It ends with exit code 78, “configuration required”. That is the expected result here, not a failure: nothing has connected anywhere yet. The stopped container stays behind so the next step can reach the volume through it.

3. Edit the config

Copy the file out of the volume, so you can edit it with any editor:

docker cp tag-collector:/config/appsettings.json appsettings.json

In appsettings.json, the SparkplugSources entry ships commented out, because one edge node can announce hundreds of metrics and every one you collect is a tag. Remove the leading // from the lines between its brackets and fill in your broker's Host, your Groups, and MaxTags: your plan's tag count, or what is left of it. For the first run, set MetricInclude to a pattern that matches nothing, as below, so the collector shows you the plant before it collects any of it (step 5). Then delete the example entries in MqttSources and OpcServers if you are not using them. Every value still in <angle brackets> is a placeholder; everything else is a default, with a comment saying what it does. Trimmed to what matters, the edited file comes down to this:

{
  "TagHistorian": {
    "ApiUrl": "https://api.taghistorian.com",
    "ApiKey": ""
  },
  "MqttSources": [],
  "OpcServers": [],
  "SparkplugSources": [
    {
      "Name": "plant",
      "Host": "192.168.10.20",
      "Groups": [ "Sweden-Boras" ],
      "MetricInclude": [ "look-first" ],
      "MaxTags": 50
    }
  ]
}

Leave ApiKey empty. The key goes into the container's environment in the next step, so it is never stored on the volume. Then copy the file back and remove the first-start container:

docker cp appsettings.json tag-collector:/config/appsettings.json
docker rm tag-collector

If you run docker with sudo on Linux, use it for all of these, and edit the copied file with sudo as well: a file docker copied out as root belongs to root.

4. Run it

docker run -d --name tag-collector --restart unless-stopped -v tag-collector:/config -e TagHistorian__ApiKey="<your-api-key>" ghcr.io/softi-dev/tag-historian-clients/collector:latest

Follow it with docker logs -f tag-collector. Without a key the collector collects nothing and says so in the log, and a key still reading <your-api-key> stops it with exit code 78 and the name of the variable. Both on purpose: a missing key is a mistake to fix now, not an outage to wait out. Left to run, the collector would have every upload refused with a 401, which it treats like a revoked key, holding the readings on disk and retrying, so a forgotten key would look like a running collector until the buffer reached its 7-day / 512 MiB bound and began dropping the oldest readings.

A placeholder left in the file also stops it with exit code 78 and the name of the setting, and so does a file that no longer parses after the edit, with the line and position where the parser gave up; your file is never overwritten. Under --restart unless-stopped such a container shows as Restarting (78) in docker ps, and the log says why. To change the configuration later, copy it out and back in the same way and run docker restart tag-collector.

Or with compose, once steps 2 and 3 have created the volume and put the config in it:

services:
  collector:
    image: ghcr.io/softi-dev/tag-historian-clients/collector:latest
    container_name: tag-collector
    restart: unless-stopped
    volumes:
      - tag-collector:/config
    environment:
      TagHistorian__ApiKey: "<your-api-key>"

volumes:
  tag-collector:
    external: true

Every command here is one line, so it pastes the same into bash, zsh or PowerShell. The image is built for linux/amd64 and linux/arm64: the same commands run on a PC, on Docker Desktop for Windows or macOS, and on an ARM box with a 64-bit OS such as a Raspberry Pi or an industrial gateway.

Updating the collector

A container keeps the image it was created from, so docker restart tag-collector starts the same collector again. To move to the newest, pull it and create the container again on the same volume; the configuration and the readings not yet sent stay where they are:

docker pull ghcr.io/softi-dev/tag-historian-clients/collector:latest
docker stop tag-collector
docker rm tag-collector
docker run -d --name tag-collector --restart unless-stopped -v tag-collector:/config -e TagHistorian__ApiKey="<your-api-key>" ghcr.io/softi-dev/tag-historian-clients/collector:latest

with your key in place of <your-api-key>, as before. With compose, docker compose pull and then docker compose up -d do the same. Tag files, browse and Modbus devices need collector 1.0.47 or later: an older one ignores TagsFile and ModbusDevices without a word, so update a collector you started before then.

On Windows, without Docker

Where IT does not allow Docker, on the SCADA PC or a small plant's Windows server, the same collector runs as a Windows service: one self-contained program for 64-bit Windows 10, 11 or Server 2016 and later, with nothing else to install. It reads the same appsettings.json, from C:\ProgramData\Tag Historian\Collector instead of a volume, and keeps its offline queue and certificates beside it.

Download the Windows zip and check it against the SHA-256 checksum published beside it.

In an elevated PowerShell, in the folder the zip was extracted to:

powershell -ExecutionPolicy Bypass -File .\install.ps1

install.ps1 copies the program to C:\Program Files\Tag Historian\Collector and registers the service TagHistorianCollector: automatic start, restarted a minute after a crash, running as its own virtual account NT SERVICE\TagHistorianCollector, which is granted write access to its data folder and nothing else. The data folder is readable by administrators and the service only, because your API key goes in it. Then it starts the service once, which writes the example configuration and stops, as the container's first start does.

Edit the file from the same elevated prompt, and put your Write key in ApiKey: on Windows the key goes in the file, because a service does not see a system environment variable set after Windows started. Then start the service and follow its log:

notepad "C:\ProgramData\Tag Historian\Collector\appsettings.json"
Start-Service TagHistorianCollector
Get-Content -Wait -Tail 20 "C:\ProgramData\Tag Historian\Collector\logs\collector.log"

logs\collector.log in the data folder carries the lines docker logs would show, and warnings and errors also go to the Application event log under Tag Historian Collector. An empty ApiKey, a placeholder left in, or a file that no longer parses stops the service with the reason in both places, and it stays stopped instead of restarting every minute.

To update, run install.ps1 from a newer zip's folder the same way: it stops the service, replaces the program and starts it again, and keeps the data folder. The zip's name carries the collector's version; tag files, browse and Modbus devices need 1.0.47 or later.

The program and scripts are not signed yet. Started from Explorer, the program gets “Windows protected your PC”: choose More info, then Run anyway. The -ExecutionPolicy Bypass above is what lets the unsigned install script run, for that one run. On Windows 11 with Smart App Control turned on, unsigned programs are blocked outright; use the container there until the collector is signed.

5. Look before you collect

An example: one edge node, EdgeGW-Line2 in group Sweden-Boras, with four filler lines under it as Sparkplug devices, Filler-01 to Filler-04, on a Pro plan with 50 tags. With the config above, docker logs -f tag-collector shows the subscription, then — on the first data message, because the collector has no birth on record yet — a rebirth request, then the births: one line for the edge node's own metrics, and one for each device:

sparkplug source 'plant': subscribed to 'spBv1.0/Sweden-Boras/#' at QoS 1. ...
sparkplug source 'plant': Sweden-Boras/EdgeGW-Line2: received data with no birth certificate
  on record. ... asking the edge node to rebirth.
sparkplug source 'plant': Sweden-Boras/EdgeGW-Line2 announced 16 metric(s): 0 being collected,
  16 not. Skipped: 5 of a type this product cannot store (string, dataset, array, file),
  5 excluded by MetricExclude, 6 not matched by MetricInclude.
sparkplug source 'plant': Sweden-Boras/EdgeGW-Line2/Filler-01 announced 74 metric(s): 0 being collected,
  74 not. Skipped: 3 of a type this product cannot store (string, dataset, array, file),
  2 excluded by MetricExclude, 69 not matched by MetricInclude.
... and the same line for Filler-02, Filler-03 and Filler-04

Nothing was stored: look-first matched no metric. But the whole namespace is in the discovery file, one per edge node and one per device, rewritten at every birth — every metric with its Sparkplug identity, the tag it would become, its datatype and why it is not collected. Copy them out of the volume — on Windows they are in the sparkplug-discovery folder of the data folder, and the log lines are in logs\collector.log beside it:

docker cp tag-collector:/config/sparkplug-discovery sparkplug-discovery

{
  "Identity": "Sweden-Boras/EdgeGW-Line2/Filler-03/Speed",
  "TagName": "Sweden-Boras.EdgeGW-Line2.Filler-03.Speed",
  "DataType": "Float",
  "SkipReason": "NotIncluded",
  "SkipDetail": "MetricInclude is set and this metric matches none of it"
}

6. Choose, and collect

Name what you want to chart in MetricInclude, in place of look-first. Patterns are matched against the identity — group/node[/device]/metric — so the filler speeds and every temperature on the line are:

"MetricInclude": [
  "Sweden-Boras/EdgeGW-Line2/Filler-*/Speed",
  "Sweden-Boras/EdgeGW-Line2/**/Temperature"
]

Copy the file back in and docker restart tag-collector (on Windows, Restart-Service TagHistorianCollector). The next births read:

sparkplug source 'plant': Sweden-Boras/EdgeGW-Line2 announced 16 metric(s): 0 being collected,
  16 not. Skipped: 5 of a type this product cannot store (string, dataset, array, file),
  5 excluded by MetricExclude, 6 not matched by MetricInclude.
sparkplug source 'plant': Sweden-Boras/EdgeGW-Line2/Filler-01 announced 74 metric(s): 6 being collected,
  68 not. Skipped: 3 of a type this product cannot store (string, dataset, array, file),
  2 excluded by MetricExclude, 63 not matched by MetricInclude.
... and the same line for Filler-02, Filler-03 and Filler-04

Six on each filler — its speed and five zone temperatures — so 24 are tags now; the edge node's own metrics match neither pattern. They appear on the Tags page in Tag Historian Explorer as their readings arrive, with a unit wherever the edge node sent one, and every one counts against your plan from its first reading. Choosing first matters because a tag, once created, stays on your plan until you delete it — and deleting a tag deletes its history.

Tag names

[TagPrefix.] group_id . edge_node_id [. device_id] . metric_name

• Every part is in the name. Leave out the device and identical pumps under one node would share a tag; leave out the node and every line's Boiler/Temp would. Two series in one tag interleave, and the deadband then throws away real values.
• Slashes in a metric name become dots, so a nested name like Process/Zone 3/Temperature keeps its hierarchy. Anything outside A-Z a-z 0-9 . _ - becomes _, which is normal: Sparkplug ids often contain spaces.
• Over 200 characters, a name keeps its first 191 and gains a short hash, so two long paths with the same start stay two tags.
• Aliases are never in a name. An edge node renumbers them when it restarts; a tag named after one would rename itself.
• One source per tag name. From collector 1.0.48, a metric whose name an OPC UA server, a Modbus device or another MQTT or Sparkplug source already collects is not collected here: the birth's inventory lists it as skipped, the log names both places once, and it does not use up MaxTags.

Filters, and the tag budget

Every metric goes through the same checks, in this order, and the first that applies decides. The ones that cost nothing come before the one that spends your budget, so a string metric never takes a tag a reading could have had:

1. The edge node marked it is_transient: not stored, as it asks.
2. Its datatype cannot be stored (see the table below).
3. It matches MetricExclude. The default is not empty: **/bdSeq, **/Node Control/**, **/Device Control/** and **/Properties/** — session counters, command points and firmware strings every edge node publishes. Set it to [] if you want them.
4. MetricInclude is set and it matches none of it.
5. MaxTags is set and full.

Patterns are globs, case-insensitive, matched against the whole identity: * is anything within one segment, ** crosses segments, ? is one character, and exclude always beats include. DeviceFilter and EdgeNodeFilter cut at a coarser grain — the two filler lines rather than the whole site.

MaxTags

A ceiling on the tags this source will create. It defaults to 0, no ceiling, because the collector does not know which plan your key belongs to — and then it says so in the log at every start. With a ceiling, the collector chooses which metrics fit in a fixed order (by identity), so the same plant always gives the same selection, and a metric it took keeps its place when a node rebirths. The ceiling is for the whole source, not per node: with several nodes, the first to publish its birth fills it first. If that order matters to you, name your metrics in MetricInclude.

What is stored

A machine state you want in history has to be a number: Machine/StateCode = 3 is stored and charts; Machine/State = "Running" is not. The datatype comes from the birth certificate, because the specification asks data messages not to repeat it. Change a metric's type at the edge, and the new type takes effect at the node's next birth.

A metric's engUnit property, from its birth certificate, becomes its tag's unit. The specification does not require one, so a metric without it is normal. The timestamp is the metric's own, else the message's, else the time it arrived.

Rebirths

A birth certificate is the only message that ties a metric's name to its alias, so the collector asks an edge node for one when:

• data arrives from a node with no birth on record — a collector that started mid-stream;
• an alias arrives that no birth announced;
• the sequence numbers show a gap, which means messages were lost;
• its own connection to the broker came back.

At most one request per node every RebirthCooldownSeconds (30) and MaxRebirthsPerMinute (10) for the whole source, because a rebirth makes a node publish its whole namespace again. After a reconnect, readings that arrive before the new birth are counted and dropped rather than decoded against aliases that may have changed: a second or two of data, against right values filed under the wrong tag. With RebirthEnabled: false the collector asks for no rebirth, and collects from a node only once that node publishes a birth on its own.

Primary host: opt in, and know what it does

Leave HostId empty and the collector is a subscriber, not a Sparkplug host application. Set it and it publishes spBv1.0/STATE/<HostId>, retained, with an offline will.

Settings

The health line

Once a minute by default, one line per source, at Warning while the broker is down or a problem counter is growing, and from the first droppedTypeConflict or dropsUnaccounted until the collector restarts:

sparkplug source 'plant' health: connected=True connects=1 messages=4210 measurements=3977
tags=24 excludedMetrics=288 unknownAliases=0 seqGaps=0 dataBeforeBirth=1 awaitingRebirth=0
rebirthsSent=1 droppedNotCollected=11820 droppedTypeConflict=0 droppedTransient=0
droppedNoValue=0 droppedNoLastGood=0 unnamedBirthMetrics=0 dropsUnaccounted=0 restamped=0
undecodable=0

• A reading that reaches the collector and is not stored moves one of these, which is what tells “filtering as configured” from “storing nothing”. The exception is a node or device that EdgeNodeFilter or DeviceFilter leaves out: it is dropped by its topic before it is read, and moves nothing.
• droppedNotCollected grows whenever MetricInclude, MetricExclude, MaxTags or an unstorable type leaves a metric out: the edge node still publishes it. Watch it against measurements.
• unknownAliases climbing means readings arrive that no birth can name, and rebirths are not fixing it.
• droppedTypeConflict and dropsUnaccounted should be 0. The first is a sender contradicting its own birth; the second is a bug in the collector — please tell us.

Limits

• Beta. Not yet run against a production gateway; see the top of this page.
• Tags are what you pay for. Free holds 5, Pro 50, Business 500. An unfiltered edge node announcing hundreds of metrics fits none of them, which is why the filters come first.
• Not “Sparkplug Compatible”. That is an Eclipse Foundation certification this has not been through. It implements the Sparkplug B specification.
• Not a host application unless HostId is set.
• No Sparkplug A, no alarm limits from engLow/engHigh (alerts are set in Tag Historian), and no reordering: a sequence gap is counted, the message in hand is stored at its own timestamp, and a rebirth is asked for.
• One broker per source. A redundant broker pair needs two sources, and a reading that arrives through both is sent twice and counts twice against the daily readings.
• Alias 0 is never used to name a reading, because Sparkplug's protobuf cannot tell alias 0 from no alias. A node that numbers aliases from 1 is unaffected; one that gives a real metric alias 0 loses that metric's alias-only readings, loudly, in unknownAliases. Readings that carry the name still arrive.
• The buffer is bounded at 7 days or 512 MiB, as for MQTT.