MQTT实战 – MQTT vs HTTP：为什么你的物联网项目上线就崩？（二） – IT规划师 – 博客园

Many IoT teams are in the prototype stage and simply choose HTTP for device access. In the demo environment, three or five devices ran very smoothly, the interfaces were adjusted, and data was reported normally.

Once put into mass production and connected to hundreds or thousands of 4G/NB-IoT devices, various disasters will follow: the server CPU is full, a large number of interface timeouts, device batteries are quickly exhausted, request storms occur immediately after network failure recovery, and command issuance is seriously delayed.

I have seen more than one project: the prototype was quickly completed using HTTP, but after it went online, it could not support the scale of the equipment, and the access layer had to be completely reconstructed, resulting in huge rework costs.

But I don’t want to jump to the conclusion: HTTP is garbage, and all projects must use MQTT.

Eight years of practical experience in the Internet of Things told me:There is no absolute best protocol, only protocols that adapt to business scenarios. HTTP still has a lot of suitable scenarios in the Internet of Things. What’s really annoying is that HTTP is used in scenarios where it is not good at long-term device connections, high-frequency reporting, and two-way command issuance.

From the perspective of the .NET IoT backend, this article compares the real production performance of MQTT and HTTP, dismantles the underlying reasons why many projects fail after they go online, and provides selection decision-making criteria that can be directly applied.

Many people doing IoT development come from web backend backgrounds, and are so familiar with HTTP that they will subconsciously apply it to all scenarios.

You may have probably made these four misconceptions in a testing environment.

‌Illusion 1: “The device will wait for me to get the data” – This is the original sin of polling. ‌

HTTP is a ‌request-response‌ model. The device cannot actively push. The device can only actively ask, or you can poll the device. Think about it: you don’t know when the water meter has water, so you ask “Is there any data?” every 30 seconds. After asking 99 times, it is empty, and it is only the 100th time. These 99 times were all in vain. When there are thousands of devices, these “free” requests will exhaust the server to death.

‌Illusion 2: “A few hundred more bytes of HTTP header are nothing” – the cost is magnified a hundred times on a narrowband network. ‌

An HTTP request can easily contain several hundred bytes of headers (Host, User-Agent, Content-Type, Cookie…). The minimum message header of MQTT is only 2 bytes. It seems that they all “send a temperature value”. One is indistinguishable from a broadband perspective, and the other is the straw that breaks the equipment’s back on a link such as 4G/NB-IoT/LoRa that is billed by traffic and metered by milliamps.

‌Illusion 3: “The network is always stable” – Once HTTP is interrupted, the data is gone. ‌

The network in the test environment is the network of your company’s computer room, which is very stable. What about the site? The optical fiber at the construction site will be cut over, there will be no signal in the basement, and the base station in the mountainous area will be floating. Once a short connection like HTTP is disconnected, the request will fail directly, and the data will either be lost or wait for you to try again. MQTT uses ‌long connections‌, and it also comes with QoS and offline message temporary storage, so it can be repaired if it is interrupted.

‌Illusion 4: “The device is plugged in” – battery-powered devices cannot withstand the torment of HTTP. ‌

It doesn’t matter how the device is plugged into the outlet. But there are a large number of battery-powered sensors in the Internet of Things. Every time HTTP is sent, a connection must be established, handshaked, and disconnected after sending. The power consumption of the radio frequency module is several times that of continuously maintaining a long MQTT connection. Many teams only found out after going online that the battery with a nominal one-year rating ran out of power in three months, and it was too late to go back and change the agreement.

HTTP is the greatest protocol on the Internet, but it was not designed for “continuous communication between massive devices.”

I have personally answered each of the five questions below. This is not a theoretical deduction, but a real fault review in a production environment.

Breakpoint 1: Connection overhead – each request is a “re-handshake”

HTTP is a request/response model. Every time data is reported, the device must go through the complete process:

Let’s do the math:

A device reports once every minute, which is one day1440 complete handshakes。

200 devices × 1440 times = 288,000 TCP connections established + destroyed per day。

The server is not defeated by the data, but by the connection overhead.

I have done actual measurement and comparison (same configuration of cloud server, 2-core 4G, 2000 devices concurrent scenario):

Note: In the HTTP short connection mode, periodic reporting by a large number of devices will generate massive TCP handshakes and releases, and CPU and memory resources will be consumed by connection management. MQTT relies on resident long connections, and only establishes connections when the device comes online for the first time. Subsequent heartbeat keep-alives are relied on, and server resource consumption is much lower than that of HTTP.

The CPU usage of HTTP is that of MQTT5–10xand as the number of devices increases, HTTP grows linearly, while MQTT remains almost unchanged.

How to solve MQTT? A long TCP connection lasts for hours or even days. After the device is connected to the Broker, the connection is always open. If there is data, it will be sent, and if there is no data, the heartbeat will keep it alive.

Breaking point 2: The server proactively delivers – HTTP cannot “push”

In the Internet of Things scenario, the backend often needs to actively send instructions to the device:

• If the temperature in the greenhouse exceeds 35 degrees, the backend must issue an “open ventilation” command.

• When the smart door lock receives a visitor request, the background will issue a “temporary unlocking password”

• Industrial equipment is abnormal, and a “shutdown protection” command must be issued in the background.

How to do it using HTTP? Both options are dead ends:

Option A: Device polling

2000 devices, polling every 10 seconds = 120,000 HTTP requests per minute. The server is exhausted just processing these empty polls, and the command delay is up to 10 seconds.

99% of polls return “no command”, and network bandwidth and server CPU are all wasted on empty polls.

Option B: Long Polling

The device sends a request and the server will return only if there is an instruction. But the problem is:

• The connection is hung and not released, 2000 units = 2000 long connections

• The instruction delay depends on the polling interval and cannot be real-time

• Implementation complexity is higher than MQTT

How does MQTT do it? Publish/subscribe model. The device subscribes to a Topic, such as device/001/command. When the backend wants to send an instruction, send a message to this Topic, and the Broker will automatically push it to the device.

There is no need for polling, no need for the device to expose interfaces, and no need to know the device IP. a message,Arrival in milliseconds。

Breaking point 3: Reconnection storm in a weak network environment

The network environment of IoT devices is 100 times worse than that of Internet applications:

• 4G signal is spotty

• WiFi is severely attenuated after penetrating the wall

• The equipment enters the elevator or basement and is directly disconnected from the network.

• Electromagnetic interference at industrial sites and network interruptions

• Underground parking lot (poor signal), disconnection frequency is every 5-10 minutes

Real case: A shared equipment project uses HTTP. The equipment is installed in an underground parking lot (the signal is poor), and the disconnection frequency is every 5-10 minutes. The TLS handshake is superimposed on the reconnection storm, and the server CPU is completely full.

Even more troublesome is the NAT penetration problem. Many IoT devices run on 4G/NB-IoT, and the operator’s NAT timeout is usually only 30–60 seconds. HTTP short connections require a three-way handshake every time, and the handshake failure rate is extremely high in weak network environments.

MQTT reconnection mechanism:

• Existing session (Clean Session = false): After reconnection, continue from where you last disconnected. No need to resubscribe or re-authentication.

• Keep-Alive heartbeat: It is recommended to set it to 30 seconds, which is less than the NAT timeout and can maintain the connection stably without interruption.

• QoS 1/2: The message is delivered at least once and will be automatically reissued after the network is restored.

• Session persistence: When the device is disconnected from the Internet, the Broker stores the message and pushes it once after the device is reconnected.

• Will message (LWT): When the device is disconnected abnormally, the Broker automatically notifies the backend that “this device is offline”

Breaking point 4: HTTP’s “retry storm” can crash the server

The HTTP request failed, what should I do? Try again. The device side usually writes like this:

Looks okay? But if it is 2000 devices retry at the same time Woolen cloth?

The server returned a 500 error due to a bug. 2,000 devices received the error at the same time, waited for 1 second, and initiated the request again…This is the prototype of a DDOS attack.

MQTT’s reconnection mechanism is built-in, and the Broker will control the reconnection rate (such as exponential backoff), so that the service will not be destroyed because the devices collectively reconnect.

Breaking Point 5: Stateless = Lost connection when offline

The HTTP server does not know when the device went offline. The device is powered off, the SIM card is in arrears, and the signal disappears – the server has no awareness until the next polling times out, which may have been several minutes.

In an alarm scenario, this delay is fatal:

Suppose you want to implement “immediate alarm when the device temperature exceeds 40°C”.

• HTTP scheme: The device POSTs the temperature every 30 seconds. In the worst case, the alarm delay is close to 30 seconds. If you want to reduce the delay to 5 seconds, the POST frequency there will have to be increased 6 times. 500 devices will make 6,000 requests per minute – the server and the operator’s traffic charges will explode together.

• MQTT solution: The device subscribes to the device/{id}/command topic, and when the temperature exceeds the threshold, it immediately publishes a message to device/{id}/alarm, and the server receives it in milliseconds. Zero polling, zero waste.

MQTT’s Last Will mechanism Solved this problem: Pre-register a “I’m offline” message when the device is connected. Once the device is disconnected abnormally, the Broker will automatically publish it for it. The server is aware in seconds, and alarms, work orders, and operation and maintenance notifications are all automatically triggered.

Breaking Point 6: HTTP packets are too large, wasting traffic and battery

How big are HTTP request headers? For any POST request, the header has several hundred bytes:

Header occupies 200+ bytes, and the actual data is only 45 bytes.The payload ratio is less than 20%.

For sensors powered by 4G traffic cards and battery, this means:

• Data charges doubled

• Battery consumption doubled

• Communication costs double

What about MQTT messages? Minimum only 2 byte header. A temperature reporting message is less than 50 bytes in total, which is smaller than HTTP Save over 80%。

Five crash points quick check

IoT devices access common fault points. The prototype is normal, but problems are exposed when the scale increases.

One sentence summary: HTTP is a post office model of “ask once and answer once”, while MQTT is a telephone model of “call and keep chatting”. In an IoT scenario with a large number of devices, high frequency, and poor network, every design assumption of HTTP is becoming a burden.

After the scale of HTTP short connections increases, the five major problems of connection, push, reconnection, retry, and messaging have exploded; the MQTT long connection model avoids the above problems at the protocol level.

To understand the difference between MQTT and HTTP, you can’t just look at “which one is faster”, but look at each of themWhy is it designed like this?。

HTTP was born in the 1990s and solved the problem of “how the browser obtains web pages from the server”. Its core model isRequest/Response——The client actively initiates a request, the server responds passively, and then the connection is closed. This model is perfect for scenarios where the user clicks and the server returns a page.

MQTT was born in 1999 and was originally designed for oil pipeline telemetry. Its core model ispublish/subscribe——After the device establishes a long connection with the Broker, the message is routed through the Topic. The publisher does not need to know who is receiving, and the subscriber does not need to know who is sending.

The difference between these two models in the Internet of Things scenario can be understood with an analogy:

HTTP is like sending a letter. You have to go to the post office (establish the connection), fill out the envelope (carry the header information), send the letter (send the request), and then go home and wait for a reply (wait for the response). Repeat this process every time you send a letter. If you want to receive letters frequently, you can only go to the post office every once in a while and ask “Do you have my letter?” (polling).

MQTT is like making a phone call. You dial once (establish the connection) and then both parties can talk at any time. You don’t need to redial every time you talk. The other party can also take the initiative to tell you the news without you having to ask repeatedly.

The most direct consequence of this difference is that the HTTP server cannot actively push messages to the device. In the Internet of Things scenario, “the platform issues instructions to the device” is an extremely high-frequency requirement. HTTP can only be achieved by device polling, and the delay and waste of resources in polling are fatal.

This comparison table can be used for program review and technology selection, and comes with a description of the actual business impact.

Let’s take a look at a few sets of data from actual measurements. These data directly explain why HTTP “cannot hold up” in the Internet of Things scenario.

Protocol header overhead: 2 bytes vs 247 bytes

This is the most intuitive difference. The minimum fixed header for MQTT is only2 bytesand the HTTP request header, even for the simplest GET request, is at leastseveral hundred bytes。

A study on microservice architecture for intelligent transportation systems measured the header sizes of two protocols:MQTT is 2 bytes, HTTP is 247 bytes. This means that each time data is transmitted, HTTP consumes moreMore than 100 timeshead overhead.

For a scenario where the sensor reports temperature every 5 seconds, a JSON data packet itself may only be 50 bytes. Using HTTP transmission, the actual network traffic is 50+247=297 bytes, and the efficiency is only 17%. Using MQTT to transmit, the actual traffic is 50+2=52 bytes, and the efficiency exceeds 96%.

Reliability: 91.88% vs 81.06%

A comparative study was conducted in a real agricultural IoT environment, and HTTP and MQTT transmission data were collected simultaneously for 54 hours. result:

MQTT is more reliable than HTTPMore than 10 percentage pointsdelay only HTTP’squarterhigher bandwidth efficiencynearly 7 times。

Another study on intelligent transportation systems gave even more extreme figures: MQTT’s delivery reliability reached99.97%while HTTP is99.89%lower than the 99.9% SLA requirement. The study clearly states that HTTP is “barely below” industrial-grade SLA standards, so the architectural decision isRoute primary sensor traffic to MQTT, HTTP reserved only for legacy system integration。

Latency: 63ms vs 1419ms

The actual measured data on the ESP32 microcontroller is even more amazing. The researchers compared the latency performance of the three protocols under three transmission sizes of 16, 64, and 256 bytes:

The gap exceeds20 times. Even on the more powerful ESP32-S3, HTTP’s latency is 449 milliseconds, while MQTT remains lower.

Energy consumption: MQTT saves 20% than HTTP

A French doctoral thesis specifically compared the energy consumption performance of MQTT and HTTP on IoT devices. The conclusion is: in publish/subscribe mode,MQTT has about 20% lower energy consumption than HTTP。

More critically, the study also found:The publish/subscribe model saves an average of 92% of energy consumption compared to the request/response model.. This means that even without looking at the differences in the protocol itself, just changing HTTP polling to MQTT subscription can bring about an order of magnitude energy consumption optimization.

For battery-powered sensor devices, this difference directly determines whether the device can run for a month or a year.

Rules: Start from the first question and follow the arrows. If you hit any ✅ yes, you will stop judging directly; if all answers are no, you will implement the all-inclusive plan.

Summary cheat sheet (take away with one picture)

This form can be used for quick screening of project approvals: first choice✅/alternative/one-vote veto❌; “—” means no suitable alternatives/no veto items.

Additional instructions:
1. A veto means that the use of this protocol is not recommended in this demand scenario. It is not that the protocol itself has bugs, but that the cost is huge in the business scenario;
2. Engineering best practices: A hybrid architecture is recommended, with MQTT used for device telemetry commands and HTTP for files and third-party docking;
3. On the Web side, large screens give priority to WebSocket/MQTToverWebSocket, and do not allow the browser to directly connect to the Broker port 1883.

High-risk combination (areas most affected by traps, please avoid them if you see them)

This table can be used for risk review and inspection. When the following combinations occur, key interceptions are required. Various combinations are derived from real online fault cases.

Risk warning:The above table is the project review check items. The occurrence of high-risk combinations does not mean that failure will occur immediately, but failures will inevitably occur when the equipment scale increases and the network deteriorates.

Best Practices for Real-World Projects: Hybrid Architecture (Not an Either/Or)

Engineering suitability is the most important. A mature IoT platform is never “all MQTT” or “all HTTP”, but each performs its own duties.

Hybrid architecture division of labor: Different business responsibilities correspond to the most appropriate agreement, and production projects should not be dominated by a single agreement.

One sentence summary：

• There are many devices, need to push, poor network, high frequency → MQTT

• Few devices, pure reporting, good network, low frequency → HTTP

• Both are not fully compatible → Prioritize MQTT (better scalability)

Many newbies fall into black and white thinking: either all MQTT or all HTTP. In engineering, suitability is the most important thing. The real experience is: ‌Don’t think of changing the protocol only in the middle of the project, that’s called rework; figure out what data goes which way from the beginning, that’s called architecture.‌

Real project comparison: the same project, HTTP vs MQTT

I did a comparison test in 2019 on the same smart building project:

Server costs have been reduced by 60%, and latency has been reduced by an order of magnitude.

If we realize during the architecture design stage that the scenario of “2000 devices reporting continuously” is not suitable for HTTP and choose MQTT as the access protocol, the direction of the entire project will be completely different.

The cost of choosing the wrong protocol is not just technical debt, but also time, money, team morale, and customer trust.

This is why I put this article in the second part of the series. Before you start writing any MQTT code, please make sure:Is your scenario really suitable for MQTT? If it fits, please read on. If it’s not suitable, it’s fine to use HTTP, don’t force it.

The core of selection is not “who is better”, but “who is more suitable”。

I compiled a “Internet of Things Communication Protocol Selection Comparison Table” and “Internet of Things Communication Protocol Selection In-depth Comparison Table”, including:

• Complete comparison of 11 protocols (MQTT / HTTP / WebSocket / CoAP / Modbus / AMQP / QUIC / LwM2M / OPC UA / TCP Private / UDP Private)

• 20 core dimensions are dismantled item by item (communication model, message overhead, real-time performance, power consumption, number of connections, security, ecology, offline messages, will messages, etc.)

• Measured performance data (comparison of CPU, memory, and number of connections between MQTT vs HTTP under 2,000 devices, the gap is 5–10 times)

• A selection decision tree (8 steps of positioning, directly from equipment conditions to which protocol to use)

• 4 sets of hybrid architecture templates + 6 avoidance lists of misunderstandings in selection (including high-risk combination reminders)

Send a private message to the “IoT Protocol Selection Table” to get it directly, no tricks.

Friends who are doing MQTT for the Internet of Things can communicate together. I will continue to update this series in this account later, and will also intersperse some records of software architect exam preparation, technology management transformation, and embedded learning.

I am an[IT planner]. I have been working on .NET IoT backend for 8 years. I am moving from pure backend to software and hardware integration architect. The road isn’t easy, but it’s worth it.

Note: The test method code and sample source code have been uploaded to the code library. Those who are interested can take a look.