Fathym
Menu
fAIOpenXThe LoopHolonicAboutDocsBlogLog in to OpenX
Docs menu

Firmware Reference Flow

The flow a unit runs from its first boot for as long as it is in the field, written out as states, transitions and pseudo-code, so you can check your firmware against it step by step. Device Makers gives the short version and the reasons; Pilot Devices shows how to get each unit's key and how to make a pass from it.

Everything above Later: with a provisioning service works with OpenX today. Firmware updates run on your own update channel: OpenX does not deliver firmware.

What the unit starts with

Written at flashingNever on the unit
Its own key-based connection string (Device ConnStr (primary) from the connection's Inspector)A pass or token, written to storage
The TLS trust roots for the hub (DigiCert Global Root G2 and Microsoft RSA Root CA 2017)A key shared with another unit
Your update channel's public verification key (the signing key stays with you)An OpenX API key, or your signing key

The key string has this shape (placeholders):

HostName=<hub>.azure-devices.net;DeviceId=<device id>;SharedAccessKey=<device key>

The states

StateWhat the unit does
BOOTReads the key string written at flashing.
NO_KEYNo usable key string in storage. It sends nothing and waits for one to be written (by hand, or through your update channel).
SYNC_CLOCKSets its clock from the network (NTP, or the cellular network's time). No pass is made before this.
CONNECTMakes a 24-hour pass from its key, in memory, and connects over MQTT with TLS on port 8883.
RUNSends telemetry. Watches the pass's age and the connection.
BACKOFFWaits a growing, jittered delay after a transient failure, then tries again. It still takes an update offered meanwhile.
WAIT_FOR_RECOVERYThe hub keeps refusing the key. It retries about once an hour and stays reachable on your update channel.
UPDATEChecks, installs and proves a new firmware image (Firmware updates).

Transitions

FromEventTo
BOOTa device key string is in storageSYNC_CLOCK
BOOTthe key string is missing or is not a device keyNO_KEY
NO_KEYa key string is written to the unitSYNC_CLOCK
NO_KEYyour update channel offers an imageUPDATE
SYNC_CLOCKthe clock is setCONNECT
SYNC_CLOCKthe clock cannot be setBACKOFF
CONNECTthe hub accepts the pass (CONNACK 0)RUN
CONNECTthe hub refuses (CONNACK 2, 4 or 5) for the first timeSYNC_CLOCK
CONNECTthe hub refuses againWAIT_FOR_RECOVERY
CONNECTthe protocol version is refused (CONNACK 1)WAIT_FOR_RECOVERY
CONNECTCONNACK 3, closed with no CONNACK, TLS fails, no networkBACKOFF
RUNthe pass reaches 80% of its lifeCONNECT
RUNthe hub or the network closes the connectionBACKOFF
RUNyour update channel offers an imageUPDATE
BACKOFFthe delay has passed, after a failed clock syncSYNC_CLOCK
BACKOFFthe delay has passed, after any other failureCONNECT
BACKOFFyour update channel offers an imageUPDATE
WAIT_FOR_RECOVERYa new key string is written to the unitSYNC_CLOCK
WAIT_FOR_RECOVERYthe hourly retry is dueSYNC_CLOCK
WAIT_FOR_RECOVERYyour update channel offers an imageUPDATE
UPDATEthe unit restarted, on the new image or the old oneBOOT
UPDATEnothing was installed, and the unit has a keySYNC_CLOCK
UPDATEnothing was installed, and the unit has no keyNO_KEY

Provision

Today a unit is provisioned when you write its key string at flashing: one Connection per unit, then Deploy, and the hub gives that unit its own random key (One key per unit). At boot the unit reads the key string. A string that is not a device key is not used: a pass or token (it would expire in the field), or a string with SharedAccessKeyName (that is a hub policy, which opens every device on the hub, never one unit).

Cache

The hub's host name is in the key string, so the unit already knows its hub and never looks it up anywhere else. Nothing else is cached: a pass is made for each connect and never written to storage.

Renew

  • Each pass lasts 24 hours. The unit makes the next one when the current one reaches 80% of its life (about 19 hours after it connected), so it is never connected on an expired pass.
  • On MQTT a renewal is a disconnect and a reconnect with the new pass: about one reconnect a day. On AMQP the pass is refreshed on the open connection.
  • When the hub or the network closes the connection, the unit waits a backoff delay (a few seconds the first time), then makes a fresh pass and reconnects. A hub that accepts the unit and then drops it straight away (for example a second unit connected with the same device ID, as the hub keeps one MQTT connection per device) gets growing delays, not a tight reconnect loop.
  • A unit that sleeps and wakes goes through SYNC_CLOCK and CONNECT at each wake, with a fresh pass.

The unit keeps its key for years. Renewing the pass never needs anyone to touch the unit.

Re-provision

Today there is no provisioning service, so "re-provision" means getting the unit working again with the key it has, or with its new key string.

  1. The hub refuses the pass once. The commonest cause is a wrong clock, so the unit sets its clock again, makes a fresh pass and tries once more.
  2. The hub refuses it again (MQTT CONNACK 2, 4 or 5): the stored key no longer opens the hub. The unit goes to WAIT_FOR_RECOVERY. Over MQTT a changed key, a switched-off device and a deleted device all look the same to the unit (refused), so it does not give up.
  3. In WAIT_FOR_RECOVERY it tries the same key about once an hour. A Connection that was switched off and is switched on and deployed again accepts the same key, so the unit heals by itself.
  4. When the key has really changed, the unit needs its new key string: open the unit's Inspector -> Connection Info, copy Device ConnStr (primary) and write it to the unit, by hand or through your update channel. As soon as a new key string is in storage, the unit starts again at SYNC_CLOCK. A unit in NO_KEY (its storage was wiped) gets its key back the same way: the hub still holds it.

What changes a unit's key in OpenX: deleting its Connection and adding it back, and a hub that is re-created under its old name (both say so before they run), and moving a Connection from the free shared environment to a hub of your own. Deploying again leaves a device's keys alone: it only re-tags the device and sets it on or off. A device missing from the hub is added back, with new keys.

Backoff

After a transient failure the unit waits, then tries again, and each wait is longer than the last:

delay = min(maximum, base * 2^attempt) * (0.5 + 0.5 * random())

A base of 5 seconds and a maximum of 1 hour suit a cellular unit. The jitter keeps a fleet that lost the network together from reconnecting all at once. A connection that stays up for 10 minutes resets the attempt count; a connect alone does not, so a connection the hub drops at once keeps the growing delay. While it waits, the unit still checks your update channel: if its trust roots no longer match the hub, every TLS handshake fails, and an update is the way out.

Pseudo-code

on boot:
  state = BOOT

loop forever:
  switch state:
    BOOT:
      keyString = readKeyStringFromStorage()        # written once at flashing
      if isDeviceKeyString(keyString): key = parse(keyString); state = SYNC_CLOCK
      else: state = NO_KEY

    NO_KEY:                                          # sends nothing to the hub
      if a device key string has been written: key = parse(it); state = SYNC_CLOCK
      else if updateOffered(): state = UPDATE         # the update channel has its own credentials
      else: sleep(1 minute)

    SYNC_CLOCK:
      if setClockFromNetwork(): state = CONNECT
      else: backoff(then = SYNC_CLOCK)

    CONNECT:
      pass = makePass(key, lifetime = 24 hours)      # in memory only
      result = connectMqtt(key.host, key.deviceId, pass)
      forget(pass)
      if result is accepted:
        connectedAt = now; renewAt = now + 0.8 * 24 hours; refusals = 0; state = RUN
      else if result is refused (CONNACK 2, 4 or 5):
        refusals += 1
        state = (refusals == 1) ? SYNC_CLOCK : WAIT_FOR_RECOVERY
      else if result is protocol refused (CONNACK 1):  # a firmware fault
        state = WAIT_FOR_RECOVERY
      else:                                          # CONNACK 3, no CONNACK, TLS, no network
        backoff(then = CONNECT)

    RUN:
      if now >= connectedAt + 10 minutes: attempt = 0  # the connection held
      if updateOffered(): state = UPDATE
      else if connectionClosed(): backoff(then = CONNECT)   # never a tight reconnect loop
      else if now >= renewAt: disconnect(); state = CONNECT
      else: sendTelemetry()

    BACKOFF:
      if updateOffered(): state = UPDATE              # e.g. new trust roots when TLS keeps failing
      else if now >= retryAt: state = then            # SYNC_CLOCK or CONNECT, where it failed

    WAIT_FOR_RECOVERY:
      if updateOffered(): state = UPDATE
      else if a different key string is in storage: key = parse(it); refusals = 0; state = SYNC_CLOCK
      else if an hour has passed since the last try: state = SYNC_CLOCK

    UPDATE:
      installed = runUpdate()                         # see Firmware updates
      if installed: restart()                         # comes back at BOOT, on the new or the old image
      else if key: state = SYNC_CLOCK                 # offer refused or download stopped
      else: state = NO_KEY

backoff(then):
  retryAt = now + min(1 hour, 5 seconds * 2^attempt) * (0.5 + 0.5 * random())
  attempt += 1
  state = BACKOFF

Never log the key or a pass. Log the state, the device ID, times and counts.

Firmware updates

OpenX does not deliver firmware today; the update runs on your board's update path or your own. These are the steps for an ESP32-S3 built with ESP-IDF; the same shape fits other microcontrollers.

  1. Check the offer. Accept an image only if its signature verifies with your public key (ESP-IDF checks signed apps with Secure Boot v2, or with CONFIG_SECURE_SIGNED_ON_UPDATE_NO_SECURE_BOOT). Refuse an image that is not newer than the running one, unless you sent it as a downgrade for safety, and log every downgrade.
  2. Refuse anything below the security floor, even for safety. With CONFIG_BOOTLOADER_APP_ANTI_ROLLBACK, each image carries a security version (CONFIG_BOOTLOADER_APP_SECURE_VERSION), and the bootloader refuses an image whose version is lower than the one recorded in the chip's eFuse.
  3. Write to the spare slot. With ESP-IDF's OTA calls, begin the write to the spare OTA partition, write each block, then end it, which validates the written image. A download that stops part way leaves the unit on its old image; try again later.
  4. Boot it on trial. Set the spare partition as the boot partition, then restart. With CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE, the new image starts in the pending-verify state (ESP_OTA_IMG_PENDING_VERIFY).
  5. Prove it. The self-test checks what the image is responsible for: it starts, sets its clock, reaches your update channel, completes the TLS handshake with the hub and gets an answer to its connect. A refusal of the key (CONNACK 2, 4 or 5) is an answer, not the image's fault (the old image would be refused too), so it passes; a refused protocol version (CONNACK 1) is the image's fault, so it fails; no answer at all keeps the image on trial, and it tests again later. When it passes, mark the app valid (this cancels the rollback and raises the security version recorded in the chip). When it fails, mark it invalid and reboot, which rolls back. If the unit resets before either, the bootloader goes back to the previous image.
  6. Log every rollback, with both versions and the reason, and report it on your update channel when the unit next reaches it.
  7. Update the modem's firmware too when the board's MQTT runs on a cellular modem; see Firmware updates on Device Makers.

Keep the update channel working on its own credentials, apart from the hub: it is the only way to reach a unit in WAIT_FOR_RECOVERY, and the way to replace its trust roots if a root ever changes in an emergency.

Later: with a provisioning service

Coming, not available yet. When OpenX runs a device provisioning service, the flow changes in four places:

  • Provision: on first boot the unit registers with the provisioning service using an identity held in hardware, and sends its printed ID and firmware version.
  • Cache: it keeps the hub it was assigned and does not register again on every boot.
  • Re-provision: after repeated connect failures that are not load, it registers again, which is how a move to another hub heals; it also accepts a request to register again, as a direct method, and as a setting it reads when it wakes if it sleeps.
  • Renew: an identity held in hardware replaces the key string, so the key never sits in ordinary storage.

Sources

Read on 2026-10-20.

On this page