Consuming bulk data

Patterns for an integration that consumes everything a workgroup collects.

This page is for building an integration that keeps its own complete copy of the data an Ardexa workgroup collects — a data warehouse, an analytics pipeline, a customer system of record. It covers the patterns that get you a complete, de-duplicated feed, and the mistakes that quietly lose data.

It assumes you can already run a search, and that you have read the vocabulary and transport summary in API Overview. The API Quick Start Guide and Search cover the basics.

The two jobs

Every integration of this kind does two separate things, and they want different tools:

Keeping up — taking delivery of records as they arrive from here on.
Catching up — filling in a period you do not have yet, whether that is the initial load or a gap after an outage.

Most integrations need both. Build them as two paths that meet, rather than trying to make one mechanism serve both.

Keeping up: follow the stream

Streaming is the preferred way to take ongoing delivery of live data, and the one to design for unless you have a reason not to. Every message carries an offset — its position in the stream — and resuming from a stored offset is what makes reconnection safe. Stream covers both forms; the Websocket Reference has the event-by-event detail.

There are two ways to be fed:

Pushed (stream:latest over a websocket) hands you each message as it arrives. Best for alerting and live displays, and for pipelines that can hold a connection open.
Polled (an HTTP request against the stream) returns a batch per request, up to 10,000 messages. Best when you would rather run on a timer than hold a connection open — it slots into existing batch code with far less change.

Both read the same stream and both resume from an offset, so the choice is about how your consumer is built, not what it receives.

Persist the offset durably, and treat "my offset is too old" as an expected branch rather than an error. The stream holds about an hour. A consumer that is down for longer — a deploy, an overnight outage — cannot recover the gap from the stream at all, and has to fall back to catching up. This will happen; design the fallback before you need it.

Keeping up without a stream: poll on `store_time`

If a long-lived connection or a tight timer does not suit, you can keep up with searches alone. Two fields, present on every table, are what make this reliable:

store_time — the time a record was written to the Ardexa Cloud's data store.
msg_id — a per-device message counter.

Before these existed, an integration had to track every device's online status so it could go back and collect events when a device returned after hours, days or weeks offline. Polling on store_time and de-duplicating on msg_id removes that bookkeeping entirely, and needs no state on our side and no long-lived connection on yours.

The procedure:

every 5 or 15 mins:
  for each table of interest:
    search all records >= the last known table.store_time
    for all records fetched:
      if table.device.msg_id <= last known table.device.msg_id:
        continue    # discard the record
      log(record)

A device that was offline for three weeks uploads its backlog when it returns; those records carry old event_time values and a current store_time, so a store_time query picks them up on the next poll. The trade against streaming is latency — minutes rather than seconds.

The mistake this pattern exists to prevent is paging on event_time. event_time is when the Ardexa agent captured the event, so a backlog arrives with timestamps behind the window you last polled — a consumer that tracks event_time never sees it, and the gap is silent. Track store_time, and use event_time only for the arithmetic once the records are yours.

Two consequences worth designing for:

Records are stored in batches, so allow up to 40 seconds before a reading is searchable. Polling for "the last 10 seconds" every 10 seconds returns nothing and convinces you the data is missing.
Duplicates are normal, which is what msg_id is for. Keep a high-water mark per device, per table.

Catching up: backfill with Search

For an initial load, or a gap the stream can no longer reach, search the period you are missing, ordered oldest-first.

For an extract of more than 10,000 records, Search scrolling is the HTTP answer: page through with a scroll_id until the records run out. It is deliberately documented as being for irregular use — it holds resources open on our side, and each scroll_id expires quickly, so a consumer that pauses between pages has to start the search again.

If your consumer cannot keep up with that, run the same search over the websocket with scroll in the query. The cloud pages through the whole result set and pushes each page as you read it, with no scroll id to manage and no expiry to race. That is the reason to prefer it for bulk work; the trade is that you hold a connection for the duration.

Either way, ask for the columns you need with fields. On a wide table the columns you discard are still columns we read and sent.

Putting it together

The hour-long stream window is why the two jobs are usually wired together, and this is the shape to reach for whenever completeness and latency both matter:

Search the period you are missing, ordered oldest-first, and record what you consumed.
Subscribe to the stream and follow it from there, persisting the offset as you go.
Whenever your offset is too old to resume, fall back to step 1.

Overlap the two by a comfortable margin rather than trying to hand over at an exact instant, and de-duplicate on msg_id — a record reaches the stored record and the stream by separate paths, and not necessarily at the same moment.

Do not pull what you can have computed

If what you actually want is a total, an average or a series of buckets, ask for it as an aggregation rather than pulling the records and computing it yourself. A day of one-second data is 86,400 records per device; the average of it is one number. API Search Functions covers the available functions, and it is the single biggest saving available on most integrations.

Aggregations cannot be combined with scrolling, which is rarely a real constraint: if a result is small enough to be a summary, it is small enough to return in one response.

What to design for

A token restricted to a device group sees only that group's devices, on every path, with no error to say so — the data is simply narrower. Regions, permissions and device-group scoping are covered in API Overview.
The audit_logs table is not available to integrations. Reading it needs the Read audit logs permission, which cannot currently be granted to an API token. On the stream's - wildcard, audit records are quietly left out rather than the request being refused.
Time expressions resolve in UTC unless you pass a timezone, and an unrecognised timezone name falls back to UTC rather than failing. See Timeframe.
Records arrive late and more than once. Neither is a fault; store_time and msg_id exist to make both survivable.