Support all whitespace characters in header values

[TimvdLippe]

This PR is part of a series of PRs across several crates, to eventually fix web-platform-test failures for Servo.

Servo currently fails /content-security-policy/generic/only-valid-whitespaces-are-allowed.html which is a test that checks whether various whitespace characters are properly parsed in the Content-Security-Policy header.

Currently, Servo times out on this test, since it attempts to send a HTTP request with Hyper. Hyper calls httparse to parse the response returned by the test server with, which includes the Content-Security-Policy with the special whitespace character. When httparse attempts to parse this, it fails since it contains characters it currently does not accept.

While RFC7230 (HTTP) does not allow such characters to be present, for web browsers section 3.5 is relevant. That section states that for historical reasons (which is the case for web browsers), both %x0B and %x0C MAY be treated as regular space characters.

Since currently httparse rejects parsing, Servo is unable to implement that requirement. To fix this issue, the following changes are made:

1. In httparse, support parsing %x0B and %x0C in header values (Support all whitespace characters in header values #225)
2. In http, consider %x0B and %x0C as opaque bytes for byte constructors (Support all whitespace characters in header values hyperium/http#862)
3. In hyper, use HeaderValue::from_bytes rather than the generic HeaderValue::from_maybe_shared_unchecked to operate on the raw bytes (Support all whitespace characters in header values hyperium/hyper#4155)
4. In content-security-policy update policy parsing to check if directive value is an ASCII string. It already handles %x0B and %x0C in is_char_ascii_whitespace, but with the HTTP changes it exposed this other missing check (Support all whitespace characters in header values rust-ammonia/rust-content-security-policy#74)
5. In Servo, replace header.to_str() with str::from_utf8(header.as_bytes()) (net, script: Support all whitespace characters in header values servo/servo#47194)

[TimvdLippe]

@seanmonstar Friendly ping, do you mind looking at this group of PRs? All in all, this fixes a whole bunch of test failures for Servo and I would love to continue making Servo more web-compatible.

[seanmonstar]

Hey! Interesting history, indeed.

I'll admit, I'm reluctant to move forward with this. It seems to weaken the parser for everyone, and the changed in the other libraries look like potential reduction in performance and guarantees of HeaderValue.

For the gain of what I assume is a very rare feature. Is simply marking that test as wont-fix an option?

[TimvdLippe]

This affects at least 18 tests across various web platform features in Servo. You can see the full list under the "Stable unexpected results" in servo/servo#47194 (comment)

I understand your reluctance. Would you be more comfortable if we put this behind a feature flag instead?

[TimvdLippe]

When discussing what the value of the change is, @jdm pointed out this breaks real websites: webcompat/web-bugs#18902 Therefore, we consider this at Servo an important quirk to fix, such that those websites load correctly.