Skip to content

Binary Media and S3 Storage

Telemetry is not limited to numbers: a device can upload binary media — camera images, inspection photos, scanned documents — together with its measurements. The binary bytes never enter the database or the message bus. NereidLink streams them into S3-compatible object storage (MinIO, SeaweedFS, AWS S3, …) and the telemetry point keeps only a lightweight blob:// reference.

This works on both MQTT and HTTP, for directly connected devices and for sub-devices behind a gateway.

Developer documentation

This page covers the device-facing contract and the operator-side configuration. The storage implementation lives in the NereidLink repository under internal/blob/ and internal/ingesthttp/media.go — see NereidAI.

How it works

device / gateway                         NereidLink                         S3-compatible store
      |                                     |                                       |
      |  telemetry frame (scalars + media)  |                                       |
      |-----------------------------------> |                                       |
      |                                     |  validate size / content type        |
      |                                     |  stream bytes (PUT object)           |
      |                                     |-------------------------------------->|
      |                                     |            object key                 |
      |                                     |<--------------------------------------|
      |                                     |  store point v_string = blob://<key> |
      |             202 accepted            |  (PostgreSQL / time-series only)     |
      |<----------------------------------- |                                       |

The stored object key is always generated by the platform and is scoped to the owner:

media/<tenant-id>/<device-id>/<uuid>

The time-series row and every Envelope carry only the string blob://media/<tenant>/<device>/<uuid>, never the bytes. Read-side consumers detect the blob:// scheme and resolve it to a short-lived signed download URL on demand.

Option 1 — wrap media inside a telemetry frame (MQTT or HTTP)

One structured field shape is accepted inside an otherwise flat frame — the binary media marker. The file payload is base64-encoded (standard or URL-safe alphabet, with or without padding):

{
  "temperature": 21.5,
  "online": true,
  "snapshot": {
    "blob": true,
    "content_type": "image/jpeg",
    "filename": "inspection-01.jpg",
    "data": "/9j/4AAQSkZJRgABAQ..."
  }
}

Scalar measurements and media fields can be mixed freely in one frame. Each media field becomes one string-valued point (blob://…); every other field is processed exactly as usual.

Marker field specification

This is the only structured object accepted inside a telemetry frame — null, arrays and any other nested objects are rejected. The wire shape is stable and versioned as a device-facing contract:

Field Required Specification
blob yes Must be the boolean true. false or any other value rejects the field, so an unrelated object is never mistaken for media.
data yes The file bytes encoded as base64 — standard or URL-safe alphabet, with or without padding. Raw plain text is not accepted: "HELLO WORLD." must be sent as "SEVMTE8gV09STGQu". Empty data is rejected.
content_type no MIME type of the bytes. Omit it or send "" to let the platform sniff the type from content (the platform never guesses units, but it does store/detect a media type).
filename no Original file name, display only. It never influences the stored object key.

Accepted encodings of the same payload: SEVMTE8gV09STGQu, SEVMTE8gV09STGQ, SEVMTE8gV09STGQ_ (URL-safe).

Two hard rules

  1. data is base64 only — never put plain text or a URL there. To reference an external file, upload the bytes; the platform owns the object key.
  2. Bytes are never inlined into the time-series row or the Envelope. Without a configured blob backend, the marker is rejected rather than stored inline.

Over MQTT

Publish the frame to the device telemetry topic — the same path used for scalar telemetry:

PUBLISH v1/me/telemetry   (QoS 0 or 1)

A gateway relays sub-device media the same way, inside the batched v1/gateway/telemetry frame: the marker goes into the per-device values map alongside the sub-device's scalars.

With the Go Device SDK, Frame.Set accepts scalars only; build the marker as a plain map and publish it with PublishRaw:

marker := map[string]any{
    "blob": true, "content_type": "image/jpeg",
    "filename": "inspection-01.jpg", "data": base64Img,
}
frame, _ := json.Marshal(map[string]any{
    "temperature": 21.5,
    "snapshot":    marker,
})
_ = client.PublishRaw(ctx, device.TopicTelemetry, frame, 1)

MQTT packet size

Base64 inflates payloads by roughly a third. Keep MQTT frames within your broker's maximum packet size and the platform's per-object limit (10 MiB by default). For large files use the raw HTTP upload below, which streams bytes without base64.

Over HTTP

The same JSON document is accepted by POST /v1/me/telemetry with the device credential.

Option 2 — raw binary HTTP upload

For larger files, or to avoid base64 overhead, stream the raw bytes directly:

POST /v1/me/blob?key=snapshot HTTP/1.1
Authorization: <device credential>
Content-Type: image/jpeg

<binary body>

The body is streamed straight into object storage (it is never fully buffered), and the platform then publishes one telemetry point referencing it. The key query parameter is the measurement key (default blob; letters, digits, _, ., - only); the object key is generated server side.

202 Accepted:

{
  "published": 1,
  "key": "media/tenant-a/dev-1/8f3c2d0e-...",
  "uri": "blob://media/tenant-a/dev-1/8f3c2d0e-...",
  "size": 48213
}

If the telemetry point cannot be published after the object was stored, the object is deleted again — no orphan files.

Accepted file types and size

Policy Default Configuration
Max size per media object 10 MiB blob.media_max_bytes
Accepted content types image/* blob.media_allowed_types
Download URL lifetime 15 minutes blob.presign_ttl

An allowlist entry ending with / matches a type prefix (image/); other entries match exactly (for example application/pdf). To accept documents, extend the allowlist in the platform configuration:

blob:
  media_max_bytes: 20971520          # 20 MiB
  media_allowed_types:
    - image/                         # all images
    - application/pdf                # PDF documents
    - application/msword
    - application/vnd.openxmlformats-officedocument.wordprocessingml.document

An empty Content-Type is accepted and sniffed on the read side instead of being guessed at upload time. Firmware packages use the same object store but a separate, much larger upload limit.

Storage backends

  • S3-compatible (default for production): MinIO, SeaweedFS, AWS S3 — standard SigV4 path-style access, one region and bucket. Uploads use the internal cluster endpoint; signed download URLs embed the externally reachable public_endpoint.
  • Local filesystem: for small or air-gapped deployments without an object service; the platform serves files itself (HTTP Range included) and signs HMAC download links.

When no blob backend is configured, media endpoints return 503 BLOB_UNAVAILABLE and media-marker fields are rejected.

Error codes

Response When
413 BLOB_TOO_LARGE Object exceeds media_max_bytes (the HTTP stream is cut off while uploading).
415 BLOB_TYPE_REJECTED Content-Type is not in media_allowed_types (HTTP upload refused before the body is read).
503 BLOB_UNAVAILABLE No blob backend configured.
502 BLOB_STORE_FAILED / BLOB_PUBLISH_FAILED Object store or telemetry publish failed; on publish failure the stored object is deleted again.

Reading media back

  • The owning device downloads its own objects with GET /v1/me/blob?key=<key> (HTTP Range, ETag and conditional requests supported). Only objects whose key encodes the authenticated device are served — a device cannot read another device's object by guessing its key.
  • Console and API users resolve a blob:// telemetry value through the tenant REST endpoint:
GET /v1/tenants/{tenantId}/blob/sign?key=media/<tenant>/<device>/<uuid>

It returns a short-lived presigned URL (url, size, content_type) — either a presigned S3 URL or an HMAC-signed /v1/blobs/get URL, valid for the configured presign_ttl (15 minutes by default). The key must belong to the tenant in the path, and the caller needs the device-read permission — a valid key for another tenant is rejected, so a reference cannot be turned into cross-tenant access.

The console renders the resolved media by type: images as thumbnails, video in a player, text/JSON/XML in a text viewer (the first 256 KiB inline), and everything else as a download.