framing: reject invalid sync buffer lengths

[mcc0nnell]

Summary

Reject invalid negative lengths in the sync buffering API and avoid signed overflow in the ogg_sync_wrote() bounds check.

Details

ogg_sync_wrote() currently accepts a negative bytes value. That can move oy->fill backwards, leaving the sync state pointing before the start of its allocated buffer. A subsequent buffer operation can then access memory before the allocation. I reproduced this as an ASan heap-buffer-overflow.

This change:

• rejects negative sizes in ogg_sync_buffer();
• rejects negative byte counts in ogg_sync_wrote();
• rewrites the upper-bound check as bytes > storage - fill to avoid signed overflow in fill + bytes;
• adds regression coverage for negative values and LONG_MAX.

Testing

• test_framing
• test_bitwise
• ASan/UBSan build and regression run

Signed-off-by: Robert McConnell robert@mcc0nnell.org