Skip to content

fix: validate encoded telemetry message size - #1241

Draft
Carter Tinney (cartertinney) wants to merge 3 commits into
mainfrom
agents/fix-message-size-validation
Draft

fix: validate encoded telemetry message size#1241
Carter Tinney (cartertinney) wants to merge 3 commits into
mainfrom
agents/fix-message-size-validation

Conversation

@cartertinney

Copy link
Copy Markdown
Member

Summary

  • replace sys.getsizeof() telemetry validation with deterministic IoT Hub message accounting: encoded body bytes, outgoing system-property values, and custom-property names and values
  • mirror Paho's payload conversion (str as UTF-8, int/float as ASCII, None as empty, and bytes/bytearray as-is) while excluding MQTT topic and packet overhead
  • share system/custom property normalization with MQTT topic encoding and include the module output system property before validation
  • replace Python-layout-dependent magic numbers and Python 3.12+ skips with exact below/equal/above 262144-byte tests for Unicode, bytes, custom properties, and system properties across sync/async device/module sends

Behavior and compatibility

IoT Hub defines message size independently of the transport as body bytes plus system-property values plus application-property names and values. The old implementation measured CPython object allocation instead, so identical messages could validate differently across Python versions.

This change intentionally affects messages near the 256-KB boundary:

  • exactly 262144 logical message bytes remain accepted; larger messages are rejected
  • UTF-8 strings and raw byte payloads are measured as actually published by Paho
  • custom-property keys now contribute to size, while Python object headers no longer do
  • outgoing system-property values, including an IoT Edge module output name, contribute to size
  • percent-encoding, the base MQTT topic, and MQTT framing do not contribute because they are protocol overhead rather than IoT Hub message content
  • payload types unsupported by Paho now fail during size validation with TypeError instead of later in the publish call; the accepted payload type set is unchanged

The MQTT topic encoding itself is unchanged; sizing and encoding now consume the same normalized property values to prevent drift.

Validation

  • focused message/topic/sync/async send tests: 178 passed on Python 3.10, 3.11, 3.12, 3.13, and 3.14
  • full unit suite on Python 3.10: 5463 passed
  • full unit suite on Python 3.13: 5463 passed
  • full unit suite on Python 3.14: 5463 passed
  • full Python 3.11 runs reached 5462/5461 passes with only pre-existing short timing-window failures in test_alarm.py and test_sync_handler_manager.py; every failure passed immediately in isolation
  • full Python 3.12 run reached 5462 passes with only a pre-existing test_alarm.py timing-window failure; it passed immediately in isolation
  • changed files pass flake8 and git diff --check

Measure IoT Hub messages from their encoded payload and logical property contribution instead of CPython object layout, and make boundary coverage deterministic across supported Python versions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cartertinney
Carter Tinney (cartertinney) marked this pull request as draft August 21, 2026 23:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Replaces Python object-size telemetry validation with deterministic IoT Hub byte accounting.

Changes:

  • Measures encoded payload and normalized property sizes.
  • Includes module output names before validation.
  • Adds exact 256-KB boundary coverage for sync and async clients.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
azure-iot-device/azure/iot/device/iothub/models/message.py Implements deterministic message sizing.
azure-iot-device/azure/iot/device/iothub/pipeline/mqtt_topic_iothub.py Shares property normalization with sizing.
azure-iot-device/azure/iot/device/iothub/sync_clients.py Includes output name in sync validation.
azure-iot-device/azure/iot/device/iothub/aio/async_clients.py Includes output name in async validation.
tests/unit/iothub/models/test_message.py Tests encoding and size boundaries.
tests/unit/iothub/test_sync_clients.py Tests synchronous boundary validation.
tests/unit/iothub/aio/test_async_clients.py Tests asynchronous boundary validation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

import urllib

from azure.iot.device.iothub.models.message import (
_get_custom_properties,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants