fix(gsm): follow-up review fixes + restore the connections/gsm package - #615
Merged
Conversation
Closes item 8 of #611. SerialCommunication was a parallel hand-rolled reimplementation of the Connection abstraction living in interfaces/, and it shared its name with connections/serial/connection.py's panel transport -- the same non-substitutable collision item 7 fixed for the protocols. - New paradox/connections/gsm/ package holding GsmSerialProtocol and GsmSerialConnection, mirroring ip/, serial/ and prt3/. - GsmSerialConnection extends Connection for lifecycle only: the protocol-aware connected property, _protocol plumbing, and close(), which the GSM transport had no equivalent of. on_message stays overridden to keep the queue and recv_callback dispatch, because AT command/response does not fit Connection's parsed-message handler registry. - The request/response method is renamed send_command(). Keeping it as write() would have collided with Connection.write(), which is a synchronous fire-and-forget and would leave send_message's coroutine un-awaited. - GSMTextInterface.stop() now closes the port instead of logging a TODO. Bugs fixed in the moved code, all of which connections/serial/connection.py already got right: - on_connection and on_connection_loss set connected_future unguarded, so a modem drop after a successful connect raised InvalidStateError. - The open timeout handle was never cancelled, and the constructor's timeout argument was accepted and then ignored in favour of a hardcoded 5. - send_command's timeout argument was ignored in favour of a hardcoded 5. - The port check was os.path.exists, which misses the far more common permission failure that os.access(R_OK|W_OK) catches. - connect() logged "Could not connect" via logger.exception outside any except block, emitting a bogus "NoneType: None" traceback.
…flow Follow-up review fixes on top of #613 and #614. - GsmSerialProtocol.send_message was declared async but awaited nothing, violating the synchronous ConnectionProtocol.send_message contract. That LSP violation is what forced the class rename and the Connection.write() NotImplementedError override; making it synchronous removes both workarounds and adds the missing check_active(). - LineFramer emitted the tail of an over-long line as if it were a whole line. It now resynchronises on the next terminator after a discard. - LineFramer's drop_blank_lines is renamed to drop_whitespace_lines: empty payloads are always skipped, whitespace-only ones are opt-in. - GsmSerialConnection.connect() bounds the port open itself with asyncio.wait_for; the existing call_later handler could not fire while create_serial_connection was the thing hanging.
Outbound SMS has never worked, before or after the framing rewrite: the code at d347e0b has all three of these defects verbatim, and the test for it was left commented out. - GSMTextInterface.send_message was the codebase's only async override of the synchronous AbstractTextInterface.send_message. handle_notify and handle_panel_event call it directly, so every notification built a coroutine that nobody awaited and silently discarded it. It is now synchronous and schedules the send as a task. - connect() ends by installing data_received as the recv callback, after which on_message routed *every* line to it and none to the queue. That starved send_command for the life of the process, so even a correctly awaited SMS could only time out. set_recv_callback is typed Callable[[bytes], bool]; that contract is now honoured, and data_received reports True only for lines it consumed (+CMT and its body, +CUSD). Command replies fall through to the queue. - AT+CMGS was sent as a single blob. Text-mode SMS is a two-stage exchange: the modem answers with a '> ' entry prompt and only then accepts the body, terminated by Ctrl-Z rather than CRLF. The prompt carries no terminator, so LineFramer can never surface it; the protocol gains a one-shot expect_prompt() that a caller arms when it knows a prompt is due, which avoids mistaking an unfinished line for one. Failure paths send ESC so the modem does not swallow the next command as message text.
Five defects found reviewing the SMS work, three verified by execution. - The command channel had no mutual exclusion. Two notifications arriving together put two AT commands on the wire before either reply came back; for AT+CMGS that means the second command is swallowed as the body text of the first SMS. Exchanges are now serialised by a lock in the interface, which owns the AT semantics -- an SMS spans send_command, write_raw and read, so the transport cannot guard it without becoming reentrant. The lock is also held across the whole init sequence. - A reply arriving after its command timed out stayed in the queue and satisfied the *next* command. send_command now drains the queue before writing: anything already there predates the command. clear() drains in place rather than replacing the queue, which used to strand a coroutine already blocked in read(). - expect_prompt() stayed armed when its command timed out, priming the next unterminated line to be read as an SMS prompt. send_command now disarms it in a finally. - run() exited once connected and nothing ever reset modem_connected, so a modem reset or unplug was fatal until PAI restarted. It now watches the transport's connected flag and rebuilds the port. connect() closes the previous port first, which also stops a failed retry leaking the last attempt. - The init helper re-sent the command in an outer loop that could never iterate twice, accumulated a `data` string it discarded, and ignored an ERROR arriving in its inner loop -- so a rejected init command cost a five second timeout instead of reporting itself. Replaced by _at_command(), sharing the final-result reader with the SMS path. In-flight send tasks are now tracked, so stop() cancels them and the loop keeps a strong reference to them.
No behaviour change; the existing tests cover all of it. - _read_final_result computed its own deadline. asyncio.wait_for around an inner loop does the same thing without the arithmetic. - Its first_line parameter existed only because _at_command went through send_command, which pops the first line and then has to hand it back, forcing a None guard on every iteration. _at_command now writes the command itself, so both the parameter and the guard go away. send_command remains for the one case that needs it: the SMS prompt. - Dropped the _OK constant, which was indirection for a two-byte literal, and folded the bare ERROR check into the error prefix tuple. - AT_COMMAND_TIMEOUT duplicated DEFAULT_COMMAND_TIMEOUT in the connection module. Import the one that already exists. - run() checked `self.port is not None`, which cannot be observed: stop() has no await between cancelling the task and clearing the port.
send_command had grown into a 25 line wrapper with two optional flags and exactly one caller. It did five things -- connection check, drain, log, arm the prompt, write, read -- none of which belong together. Deleting it leaves the transport with four orthogonal primitives: clear(), write(), write_raw() and read(). The prompt expectation moves onto read(), which is what it actually affects, and both exchanges become the same three steps: clear, write, read. Also removes an unnecessary list() around the send task set, flagged by SonarCloud. cancel() does not run done callbacks synchronously, so the set cannot mutate while it is iterated. The other seven SonarCloud findings ask for asyncio.timeout() in place of asyncio.wait_for(). That context manager is 3.11+ and this project supports 3.8, so they do not apply. The stale-reply test moves to the interface, where the drain now lives, and is split into the two cases that actually matter: a stale OK satisfying the next command, and a stale line being read as the SMS prompt. The single test it replaced passed with the fix reverted -- _read_final_result skips unrecognised lines, so stale junk was never the risk.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Follow-up to #613 and #614, from the post-merge critical review.
#614's content never reached
dev. It was stacked onfix/611-gsm-line-framingand GitHub marked it MERGED when it merged into that branch — but #613 was merged todevbeforehand, soparadox/connections/gsm/does not exist ondevtoday. The first commit here is a cherry-pick of9803043to bring it back.Fixes
1.
send_messagewas async but awaited nothing — the root cause of item 7ConnectionProtocol.send_messageis a synchronous abstract method.GsmSerialProtocoloverrode it asasync defaround a baretransport.write(). That LSP violation is why it collided withserial.protocol.SerialConnectionProtocolin the first place: #613 renamed the class to dodge the collision, and #614 then had to overrideConnection.write()to raiseNotImplementedErrorso nobody could reach the un-awaited coroutine.Making it synchronous deletes both workarounds.
send_command()now uses the inheritedConnection.write(), and the missingcheck_active()guard is in place.2.
LineFrameremitted the tail of an over-long line as a valid frameBefore, after discarding an overrun the framer would treat whatever followed up to the next terminator as a complete line:
It now resynchronises: bytes up to and including the next terminator are dropped, so only
b"OK"comes through.reset()clears the resync state.3.
drop_blank_lines→drop_whitespace_linesThe old name conflated two things. Empty payloads are now always skipped; skipping whitespace-only lines is the opt-in behaviour the name now describes. PRT3 is the only caller and keeps its existing semantics.
4.
connect()did not actually bound the port openopen_timeoutwas scheduled withcall_later, but it only resolvesconnected_future— useless ifcreate_serial_connectionis what hangs, sinceconnect()is not awaiting that future yet. The open is now wrapped inasyncio.wait_for.Tests
1633 passing. New coverage for the overflow resync,
reset()clearing it, the bounded open, andwrite()reaching the transport (replacing theNotImplementedErrortest).5. GSM outbound SMS never worked at all
Not a regression — the code at
d347e0b, before any of these PRs, has all three of these defects verbatim. The test for it was left commented out intests/interfaces/test_gsm.py:a.
send_messagewasasyncover a synchronous base.GSMTextInterfacewas the codebase's onlyasyncoverride ofAbstractTextInterface.send_message, whichhandle_notifyandhandle_panel_eventcall directly. Every notification built a coroutine that nobody awaited and dropped it — the source of theRuntimeWarning: coroutine 'GSMTextInterface.send_message' was never awaitedseen throughout the test suite. It is now synchronous and schedules the send as a task.b. The response queue was starved after init.
connect()finishes withset_recv_callback(self.data_received), after whichon_messagerouted every line to the callback and none to the queue — sosend_command()'squeue.get()could never be satisfied. Even a correctly awaited SMS could only time out.set_recv_callbackis typedCallable[[bytes], bool], but the result was discarded. Honouring it gives URC demultiplexing for free:data_receivednow returnsTrueonly for lines it consumed (+CMTand its body,+CUSD);OK,ERRORand+CMGS:fall through to the waiting command.c.
AT+CMGSwas sent as one blob. Text-mode SMS is a two-stage exchange: the modem answers with a>entry prompt and only then accepts the body, terminated by Ctrl-Z rather than CRLF. Sending both at once leaves the modem sitting in entry mode with nothing sent.The prompt is
\r\n>— unterminated, soLineFramercan never surface it.GsmSerialProtocolgains a one-shotexpect_prompt()that a caller arms when it knows a prompt is due; unarmed,>is correctly treated as a line that has not finished arriving. Failure paths send ESC so the modem does not swallow the next command as message text.Tests
1654 passing, up from 1633 on
dev. Each fix was verified to be genuinely covered by reverting it and confirming the new tests fail:test_unsolicited_lines_do_not_satisfy_a_pending_commandtest_sms_is_a_two_stage_exchangetest_sms_is_a_two_stage_exchangedrives real bytes through the real protocol and asserts the full command → prompt → body →+CMGS:/OKsequence, which is what the commented-out test was reaching for.Follow-up: rubber-duck review of the above
Reviewing the SMS work turned up five more defects. Three were confirmed by running them, not just by reading.
6. The command channel had no mutual exclusion (blocking)
Two notifications arriving together each schedule a send. Observed: both commands reach the modem before either reply comes back.
For
AT+CMGSthat is worse than mispaired replies — between the>prompt and the Ctrl-Z the modem treats everything written as message text, so the second command becomes the body of the first SMS.The lock lives in
GSMTextInterfacerather than the transport: an SMS exchange spanssend_command+write_raw+read, so guarding it inside the connection would require a reentrant lock. It is also held across the whole init sequence, so a half-initialised modem cannot see an SMS interleaved betweenATE0andAT+CMGF=1.7. A stale reply satisfied the next command (blocking)
The late answer to a timed-out command sat in the queue and was handed to whatever came next:
send_commandnow drains before writing — anything already queued predates the command.clear()drains in place instead of replacing the queue object, which previously stranded any coroutine already blocked inread().8.
expect_prompt()stayed armed after a timeoutNothing disarmed it on the error paths, leaving the next unterminated line primed to be read as an SMS prompt. Now cleared in a
finally.9. A modem drop was fatal until restart
run()loopedwhile not self.modem_connectedand returned once connected.on_connection_lossclears the transport's flag but nothing ever reset the interface's, so an unplugged or reset modem meant no more alerts until PAI restarted — on an alarm interface, silently.run()now watches the transport and rebuilds the port, andconnect()closes the previous one first (a failed retry used to leak the last attempt).10. The init helper was dead code around a real bug
The outer loop can never iterate twice (the inner one only exits when
r == expected),datais accumulated and thrown away, and anERRORarriving in the inner loop is ignored — so a rejected init command cost a 5 s timeout instead of reporting itself. Replaced by_at_command(), which shares the final-result reader with the SMS path.Also: in-flight send tasks are now tracked, so
stop()cancels them and the event loop keeps a strong reference (asyncio only holds a weak one).Tests
1664 passing, up from 1633 on
dev. Every fix was verified by reverting it and confirming the new tests fail:test_concurrent_sends_are_serialisedtest_a_stale_reply_does_not_satisfy_the_next_commandtest_a_timed_out_command_disarms_the_prompttest_run_reconnects_after_the_modem_dropstest_unsolicited_lines_do_not_satisfy_a_pending_commandtest_sms_is_a_two_stage_exchange