IoT Scalability: Millions of Devices, Message Brokers

Scaling an IoT deployment from a 50-unit pilot to a 5-million-unit fleet is not a matter of buying bigger servers. Connection handling, message routing, and state storage all hit different bottlenecks as device count grows, and the architecture that worked at 50 devices (one MQTT broker, one database, synchronous REST calls) rarely survives the trip to 500,000: synchronous call chains and the single database usually saturate first, and a single broker is both a capacity ceiling and a single point of failure. Understanding where the limits are — and how message brokers, and the systems behind them, are designed to push past them — is foundational to any IoT system architecture.

Why Scale Breaks Naive Architectures

A single device polling a REST endpoint every 30 s is trivial. The problem is multiplicative:

Total messages/sec = N_devices × (1 / reporting_interval)

At 1 million devices reporting every 30 s: 1,000,000 / 30 ≈ 33,333 msg/s sustained, before counting command/config traffic or retries. A relational database committing one transaction per message typically tops out somewhere between a few thousand and a few tens of thousands of transactions per second on one node, and a single broker node, even one that can handle the rate, is still a single point of failure. Three resources hit limits independently:

• Connection count: each device typically holds a persistent session — a TCP connection with TLS for MQTT, or DTLS session state over UDP for CoAP. One broker process has file-descriptor and memory ceilings: roughly 4–10 KB of kernel and broker memory per idle MQTT connection, more once TLS buffers are counted, so 1M devices need on the order of 10 GB of RAM just to hold sessions open. Mass reconnect events (for example, after a regional outage) add TLS-handshake CPU on top.
• Message throughput: the broker must authenticate, route, and filter every message, and store some of them (retained messages, and QoS 1/2 messages queued for offline persistent sessions). Topic-matching cost and fan-out (one telemetry message delivered to several subscribers: storage, rules engine, dashboard) multiply the effective load.
• State storage: device shadows/twins, last-known values, and time-series telemetry all need writes at device scale. A general-purpose relational database doing one insert per message runs into per-commit log flushes and index maintenance long before it runs out of disk bandwidth. Batched inserts, a time-series database for telemetry, and a key-value store for shadows and last-known values are the usual fixes.

The Thundering Herd Problem

A specific failure mode worth naming: mass reconnect. If a broker node restarts or a mobile network segment recovers from an outage, every affected device tries to reconnect within seconds. This "thundering herd" spikes connection and TLS-handshake rates, plus session-restore traffic (resubscriptions, queued messages), far above steady state — and a full TLS handshake costs far more CPU than routing a message.

Mitigations:

• Jittered exponential backoff: randomize the first reconnect attempt too, and spread retries across the whole window: delay = random(0, min(cap, base × 2^attempt)) ("full jitter"). A narrow jitter such as base × (1 + random(0,1)) still leaves the whole fleet retrying within the same few seconds. Choose the cap so that N_devices / window stays below the connection rate the cluster can admit.
• Connection rate limiting at the load balancer or broker cluster ingress, so excess clients are turned away cheaply instead of piling up half-finished handshakes.
• Cheaper reconnects: TLS session resumption (session tickets, or PSK in TLS 1.3) and ECDSA server certificates make each reconnect far cheaper than a full RSA handshake. Persistent sessions (Clean Start = 0 in MQTT 5, CleanSession = 0 in 3.1.1) keep subscriptions on the broker so devices needn't resubscribe; the price is session state held for offline devices, so set a session expiry.
• Staggered schedules: group devices by ID hash and spread scheduled actions — firmware checks, certificate renewal, daily reports — across time windows instead of a fixed clock tick (see IoT device management: registration, monitoring, OTA).

Horizontal Scaling: Broker Clustering

The core answer to "more devices than one machine should hold" is partitioning connections and topics across many broker nodes rather than one instance.

A common production pattern combines sharding by geography or tenant with a clustered broker layer inside each shard, and a lightweight bridge or log replication for global aggregation (fleet-wide alarms, cross-region analytics).

Check the load balancer too. A proxying load balancer — one that terminates the device's TCP or TLS connection — opens its own connection to a broker node for each device, and each needs a unique source port. With one source IP and one backend IP:port, Linux's default ephemeral port range allows about 28,000 concurrent connections, and the port space itself caps out near 64,000. Give the proxy several source IPs or the brokers several listener ports, or use a pass-through layer-4 load balancer.

Decoupling Ingestion from Processing

The single biggest architectural lever at scale is decoupling ingestion from processing. Instead of a device's message triggering synchronous downstream work (write to DB, run rules, update dashboard — all before acknowledging the device), the ingestion tier only accepts the message, and consumers process it at their own pace.

The MQTT broker handles the device side: terminating connections, authenticating, routing on topics. It isn't a durable log, though — it pushes messages to subscribers and buffers for a slow or offline subscriber only up to configured queue limits. Large deployments therefore bridge the broker into a durable log or queue (Apache Kafka, Apache Pulsar, Amazon Kinesis, or a managed service's rules engine feeding queues), where each consumer group reads at its own pace and tracks its own offset (see IoT ecosystem: devices, gateways, brokers, apps):

Device → MQTT broker cluster → durable log (e.g., Kafka) → [consumer group: time-series DB writer]
                                                         → [consumer group: rules engine]
                                                         → [consumer group: alarm service]

This gives three properties needed at scale:

• Backpressure absorption: a burst of 50,000 msg/s can be absorbed even if the database writer sustains only 10,000 writes/s. A 60-second burst leaves a backlog of 2.4 million messages, which drains once input falls back below writer capacity. The log smooths transient mismatches; it can't fix a sustained one, so steady-state ingest must still fit within consumer capacity.
• Independent consumer scaling: add instances of the slow consumer (for example, the database writer) without touching ingestion or device firmware. In Kafka, parallelism within a consumer group is capped by the topic's partition count, so plan partitions for the largest consumer group you'll need.
• Fault isolation: if the alarm service crashes, telemetry storage keeps running, and the alarm consumer resumes from its last committed offset on recovery — as long as the outage is shorter than the log's retention period.

Within MQTT itself, MQTT 5 shared subscriptions ($share/group/topic) spread one subscription's messages across several consumer instances. That helps throughput, but it doesn't turn the broker into a buffer for long backlogs.

QoS and Delivery Guarantees at Scale

Delivery semantics matter more as fleet size grows, because retries and acknowledgments multiply load:

No MQTT QoS level gives end-to-end exactly-once processing on its own: once a message leaves the broker for a log or a database, consumer retries can duplicate it again. Production systems usually combine at-least-once delivery with idempotent processing — upserts keyed by message ID, or a dedupe store — which gives effectively-once results at QoS 1 cost. That's also the right pattern for billing-relevant events.

A common design pattern is telemetry at QoS 0 or 1 and commands at QoS 1 with idempotent handlers. Make commands idempotent by design: send absolute state ("set valve = open", "setpoint = 21 °C") with a command ID the device remembers, never relative actions ("toggle", "+1 °C", "dispense 10 ml") whose outcome changes if they're delivered twice. Give commands an expiry (the MQTT 5 Message Expiry Interval) so one queued during an outage doesn't execute hours late, or carry them as desired state in a device shadow that the device reconciles when it reconnects. For firmware updates, MQTT should carry the job notification; the image itself is downloaded over HTTPS.

Worked Example: Sizing a Broker Tier

Assume a fleet of 2,000,000 devices, each sending one 200-byte telemetry message every 60 s at QoS 1 over TLS, plus a fan-out of 3 (storage, rules engine, dashboard cache) per message.

1. Ingestion rate: 2,000,000 / 60 ≈ 33,333 msg/s inbound.

2. Fan-out rate: 33,333 × 3 ≈ 100,000 msg/s delivered from the broker (or log) to consumers.

3. Bandwidth: the payload alone is 33,333 × 200 B ≈ 6.67 MB/s ≈ 53.3 Mb/s. Per-message overhead is fixed, though, so for small messages it is large. Assuming a 40-byte topic, TLS 1.3, and IPv4:

Payload                                    200 B
MQTT PUBLISH header + 40-byte topic         48 B
TLS 1.3 record (header + AEAD tag)          22 B
TCP/IPv4 headers (with timestamps)          52 B
Total on the wire                        ≈ 322 B   (≈ +61%)

That's about 86 Mb/s inbound, plus about 21 Mb/s of PUBACKs going back to the devices. MQTT 5 topic aliases replace the 40-byte topic with a 2-byte alias after the first message (≈ +43%), and batching ten readings per message brings the overhead down to about 6%.

4. Connections: 2,000,000 persistent sessions at ~6 KB each (broker + kernel) is 2,000,000 × 6 KB ≈ 12 GB just for session state. That fits in one large server, so memory alone doesn't force a cluster; failure domains and reconnect capacity do. If one node holds every session, losing it disconnects the entire fleet at once, with no spare node to absorb the reconnects. A cluster of 20 nodes at ~100,000 connections each — a comfortable per-node target for common MQTT broker implementations — rises to about 105,000 per node if one node fails, so leave that headroom.

5. Reconnect storm: assume ~1 ms of CPU per full mutual-TLS handshake (RSA-2048 server key; ECDSA keys and session resumption cost less). If all 2,000,000 devices reconnect within 10 s, that's 200,000 handshakes/s, or about 200 CPU cores. With full jitter spreading the reconnects over 10 minutes, it's about 3,300 handshakes/s, or 3–4 cores across the cluster.

6. Storage writers: if one writer instance sustains 15,000 writes/s, the 33,333 msg/s ingest rate needs 33,333 / 15,000 ≈ 2.2, so at least 3 instances to keep up in steady state. Headroom decides recovery time: after a 5-minute writer outage the backlog is 10 million messages, which 4 writers clear in about 6 minutes while keeping up with live traffic, and 3 writers in about 14. The writer tier, not broker throughput, is the binding constraint for storage — and the log needs at least as many partitions as writer instances.

Data Path Implications

• Batch or downsample at the edge/gateway (see IoT device classification: constrained, gateway, cloud). Batching several readings into one message keeps every sample while amortizing the fixed per-message overhead; aggregating to statistics over a window (min/max/avg) also cuts data volume, but discards detail, so keep the min and max, not just the average. Either can cut message count by 10–100× before traffic reaches the broker.
• Use compact encodings (CBOR or Protobuf instead of verbose JSON) to cut per-message bytes, which lowers bandwidth and the parsing cost in the rules engine and consumers. The broker itself routes on the topic and doesn't parse payloads.
• Design the topic hierarchy for filtering, such as region/site/device-id/telemetry, rather than one global topic, so subscriptions and access rules can be scoped and each consumer receives only the traffic it needs. Spread fleet-wide streams across consumer instances with shared subscriptions or log partitions, not a single subscriber on #.

Key Takeaways

• Naive single-broker, single-database architectures hit connection, throughput, storage, or availability limits long before 5-million-device scale; size them quantitatively (msg/s = N / interval) instead of assuming.
• Thundering-herd reconnect storms after outages need full-jitter backoff, rate limiting, and cheap reconnects (TLS session resumption, persistent sessions), or a brief node restart can spike load far above steady state.
• Broker clustering, sharding, and geographic federation are the standard ways to scale connection count and throughput horizontally; a cluster's main job is to limit failure domains and absorb reconnects, not just to add memory.
• The MQTT broker handles connections and routing; a durable log or queue behind it provides backpressure absorption, independent consumer scaling, and fault isolation.
• QoS choice trades reliability for cost per traffic type, but no QoS level gives end-to-end exactly-once processing on its own; combine at-least-once delivery with idempotent processing and application-level message IDs.
• For small messages, protocol overhead rivals the payload; edge batching, topic aliases, and compact encodings cut load before it reaches the broker, often more cheaply than adding broker capacity.