Data Connection

A data connection is where data enters your workspace. In OpenX it is one thing: a device in an Azure IoT Hub, registered for you when you deploy. Something sends messages as that device (a simulator, an API source, or a real machine), the messages land in your workspace's analytics table, and warm queries read them.

On the canvas it is the Connection node. In a proposal its kind is DataConnections.

What a data connection is not

There is no source type to pick. A data connection is not an Event Hubs, Kafka or MQTT broker setting, and it is not a database. Every data connection that deploys today is an Azure IoT Hub device: its Type is AzureIoTHub (Deploy fills that in when it is missing). Other Type values can be stored, but nothing deploys or removes them.

You do not bring the hub either. OpenX decides where the device goes:

Until your workspace has a private landing zone and its own IoT fabric, the device registers in Fathym's shared IoT hub.
Once both exist (Data & IoT Fabric), it registers in your own hub, in your own Azure subscription.

So nobody needs to supply a hub hostname (<hub>.azure-devices.net), a connection string, a subscription, a resource group or a hub name to create a connection. An agent is not allowed to set the subscription, resource group or hub name on a proposal at all; a proposal that does is refused.

What you need to create one

A name. That is the whole proposal:

{ "Details": { "Type": "AzureIoTHub", "Name": "boiler-4" } }

Optional: a description, and Is IoT Edge Device (off unless the device really is an IoT Edge device).

Accepting the proposal writes the connection to the workspace. Deploy registers the device in the hub. After that, the connection's Connection Info tab shows its host name, device id and credentials for a real device to use (Live Devices).

What feeds a data connection

A connection on its own just waits for messages. Three things can send them. For a simulator or an API source, the link is a field on the connection, not on the feeder:

Feeder	How it is wired	Line on the canvas
Simulator	The connection's `SimulatorLookup` names the simulator	"simulates"
API Source	The connection's `APISourceLookup` names the API source	"polls"
A real device	The device uses the credentials from the connection's Connection Info	none

A simulator sends as the connection's device: on Deploy it starts a container that sends your message template at your interval (15 seconds at the least). An API source polls a URL and sends what it gets back as the connection's device too. Either way the data arrives through the same hub device, so you can start on a simulator and switch to a real source later without changing your queries.

How warm queries read it

Messages a connection receives land in your workspace's analytics table in Azure Data Explorer: Data, or Devices in workspaces whose IoT fabric was set up earlier. The warm-query guide in guides/ names the table for your workspace. Each row has EnqueuedTime, DeviceID and the message itself as JSON in RawData.

DeviceID is an id OpenX derives for the connection's device, not the connection's name, so sample before you filter on it:

Data | take 5

A warm query names the connections it is about in its own DataConnectionLookups, and its steps are KQL you write over that table. See Warm query and the Query Patterns Library.

How a surface groups them

A surface holds the pieces of one job. It binds connections with DataConnectionLookups and warm queries with WarmQueries. The connection and the query do not point back at the surface. A warm query that no surface binds still exists, but it shows on no canvas, so a warm query is always proposed together with the surface that binds it.

One connection can sit on many surfaces, and one surface can hold many connections.

Worked example: a temperature simulator on a surface

Ask: "Simulate a boiler's temperature, 18 to 30 degrees every 15 seconds, and give me the hourly average on a Boiler Room surface."

That is four records, proposed together in one bundle:

Kind	Lookup	What it holds
`Simulators`	`boiler-temp-sim`	The message template and its variables
`DataConnections`	`boiler-4`	A name, and `SimulatorLookup: "boiler-temp-sim"`
`WarmQueries`	`boiler-avg-temp-1h`	One KQL step, and `DataConnectionLookups: ["boiler-4"]`
`Surfaces`	`boiler-room`	`DataConnectionLookups: ["boiler-4"]` and the warm query binding

The Proposed part of each:

{
  "boiler-temp-sim": {
    "Details": {
      "Name": "Boiler Temperature Simulator",
      "Type": "AzureDocker",
      "MessageTemplate": "{\"temperature\": $.Temp}",
      "Variables": "[{\"name\":\"Temp\",\"random\":true,\"min\":18,\"max\":30}]",
      "MessageIntervalMS": 15000,
      "MessageCountPerDevice": 0
    }
  },
  "boiler-4": {
    "Details": { "Type": "AzureIoTHub", "Name": "boiler-4" },
    "SimulatorLookup": "boiler-temp-sim"
  },
  "boiler-avg-temp-1h": {
    "Details": {
      "Name": "Boiler Average Temperature (1h)",
      "ApiPath": "boiler-avg-temp-1h",
      "Steps": {
        "main": {
          "Kind": "Query",
          "Engine": "kql",
          "Source": { "Kind": "analytics" },
          "Text": "Data | where EnqueuedTime > ago(24h) | extend Payload = todynamic(RawData) | summarize avg(todouble(Payload.temperature)) by bin(EnqueuedTime, 1h)"
        }
      },
      "Outputs": { "result": { "Step": "main" } }
    },
    "DataConnectionLookups": ["boiler-4"]
  },
  "boiler-room": {
    "Details": { "Name": "Boiler Room" },
    "DataConnectionLookups": ["boiler-4"],
    "WarmQueries": { "boiler-avg-temp-1h": {} }
  }
}

The query's Data is the table name; use the one your workspace's warm-query guide names. The KQL was not run against a workspace: once the simulator is sending, sample the table, then check and run the draft before proposing it.

Then a person accepts the proposals and presses Deploy. The device is registered, the simulator starts, and its first message arrives after the first interval. get_stats shows how the connection is doing.

What an agent should not need to ask

Do not ask	Why
"Which source type: IoT Hub, Event Hubs, Kafka?"	There is one kind of data connection, an IoT Hub device
"What is your IoT Hub hostname?" (`*.azure-devices.net`)	OpenX picks the hub: Fathym's shared hub, or the workspace's own once it has one
"What is the connection string, subscription or resource group?"	OpenX resolves them at Deploy; an agent may not set them on a proposal
"MQTT, AMQP or HTTPS?"	The device chooses when it connects; Connection Info lists all three after Deploy

What is worth asking, when the request does not say: what the connection should be called, whether the data should come from a simulator (and which readings, ranges and interval), an API source, or a real device, and which surface it belongs on.

How-to -> Connecting Data, Simulator, Live Devices, API Source
Where it is read -> Warm query, Surface

On this page

Data Connection
- ╰─▶What a data connection is not
- ╰─▶What you need to create one
- ╰─▶What feeds a data connection
- ╰─▶How warm queries read it
- ╰─▶How a surface groups them
- ╰─▶Worked example: a temperature simulator on a surface
- ╰─▶What an agent should not need to ask
- ╰─▶Related