v4.3.1: coalesce quote/depth forwards so NEW_BAR is not starved (#143) - #145
Merged
Merged
Conversation
Keep one in-flight quote_update and market_depth task per contract and replace extras with the latest payload. An RTH-open flood no longer fills the asyncio queue ahead of market_trade tick processing. User-hub events are unchanged. PATCH 4.3.1.
Gateway unpacks handler(*[contract_id, data]), so the coalesce key must use args[0] when it is a string. Packed list/dict payloads still work. Dropped counts only increment on pending overwrite. Drain generations prevent cleanup/cancel from clearing a newer inflight bit.
Owner
Author
|
Follow-up after review: live SignalR calls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #143.
Problem
At CME RTH open, every SignalR
quote_update/market_depthmessage scheduled its ownrun_coroutine_threadsafetask (_use_batchingdefaults false). That queue grew without bound and starved data-manager tick processing, so 1-minuteNEW_BARarrived tens of seconds late (or not at all) while a 1s heartbeat still looked healthy. Halt then cancelled a pile of pending_forward_event_asynctasks.Related #137 (
get_session_datahang) was not the stall this time —on_barnever ran.Fix
Latest-wins coalescing per
(event_type, contract_id)forquote_updateandmarket_depth:enable_batching()is on (that path also used to schedule one coroutine per tick)Never dropped:
market_trade(volume / OHLC /NEW_BAR) and user-hub events (order_update,position_update,account_update,trade_execution).Dropped counts are on
suite.realtime.get_stats()ascoalesced_quote_droppedandcoalesced_depth_dropped.Quotes are snapshots, so latest-wins is correct. Depth is per price level, so a flood may skip intermediate book updates until later rows or a reset arrive. That matches the live consumer workaround that unstarved
NEW_BAR.Tests
TestCoalescedMarketEventSchedulingintests/realtime/test_event_handling.py:market_tradeand user events are never dropped during a quote floodhandle_quotepath is also boundedcleanup()clears coalesce stateLocal verification:
3253 passed, 2 skipped(uv run pytest tests/ --ignore=tests/benchmarks/). mypy clean (118 files). Pre-commit ruff/mypy/bandit passed.Release
PATCH
4.3.1. Changelog, README, docs/guide/realtime.md, User-Agent, and version strings updated.