runtime: I/O API + simulator

[kim]

Adds an io-uring-compatible (file) I/O API with simulator and tokio impls.

The DST runtime uses the simulator to drive I/O operations and inject failures according to a configuration. Failure injection is more or less for demonstration purposes, and may need to become more sophisticated for actual DST.

It is not yet clear how the application-facing API should look like: in any case, we will need a way to enter the runtime context.

[Shubham8287]

This seems really good, I just think we should have sim I/O ops to use time to simulate latencies, Which is also required for sim executor to make progress correctly.

We can merge this PR, and do it in next PR though (Before we start using uring interface)

[kim]

Putting this back into draft as I had to rework it substantially.

Namely, in order to support concurrency (like join_all or FuturesUnordered), the compiler needs to know the type of the operation futures and they need to unify. I.e. -> impl Future can't be stored in a list.

To do that, then, the B: AlignedBytes buffers also need to unify, which lead me down the road of passing pointers to heap allocations down the executor. Incidentally, this is what we'd have to do with actual io-uring. The downside is that *mut u8 is !Send, which keeps us honest but makes this machinery hard to integrate with the current runtime.

I don't think it's a big issue, because the API itself must come from somewhere, and we only really need the runtime to drive the completion futures (i.e. oneshot channels).

I'm keeping the PR in draft for now, as I think the SpacetimeIO API could potentially be changed to hide more of those details (and potentially also support linked batch operations, which could turn out useful).

[Shubham8287]

, which lead me down the road of passing pointers to heap allocations down the executor.

Do you consider ErasedBox as long term solution? asking because, Boxing requires alloc.

[kim]

Do you consider ErasedBox as long term solution? asking because, Boxing requires alloc?

I don't think there is any other solution if we want it to work with actual io-uring.

I'm imagining that the pre-allocated memory is allocated on the heap (e.g. Vec<Box<Block>>>). I'll need to probably change the API to take Box<impl AlignedBytes> everywhere.

[Shubham8287]

I'm imagining that the pre-allocated memory is allocated on the heap (e.g. Vec<Box>>). I'll need to probably change the API to take Box everywhere.

Yeah, I was expecting something in that direction and, I also think Vec could be problematic to divide and move around due to borrow checker so maybe some link-list type data structure.

Though, None of it needs to be addressed in this PR.

[Shubham8287]

Do you think it will be too sophisticated to put concurrency limit on individual code components and treating the cap as invariant in executor? rather than doing back-pressure from executor which will be unpredictable and awkward.

I see, several benefits of it doing it at components level: We will know exactly what part is taking more than required resource, and we will have more granular control on things like snapshot witting would not be able to block archival, etc.

Though i am not sure about practicality of it, so I am figuring that out, seeking your thoughts.

[kim]

Yeah this should probably just be a Vec. But note that the submission and completion queues are bounded (the ring in uring).

Typically this is not very large (1024 is considered large), so if we can't predict the number of iops (which I think we can't), we'll need to put a semaphore in front.

[kim]

Ah uhm no, it works a bit differently: the SQ size only limits the batch size basically, and then the kernel executes stuff until it runs out of resources. There is never submission backpressure if the CQ overflows, completions will either be dropped or just appear on the CQ later.

I think we should never use IORING_FEAT_NODROP, so either panicking or dropping CQEs seem reasonable modes for the simulator.

[Shubham8287]

I think we should never use IORING_FEAT_NODROP, so either panicking or dropping CQEs seem reasonable modes for the simulator.

Should not we put custom backpressure in submission queue to avoid both panicking and dropping CQEs in prod implementation. Which implies simulator should always be panicking in that case?

[kim]

I think dropping could still be a valuable fault injection mode, because it makes completion channels pending forever without dropping the sender.