Skip to content

feat(rest): support configurable page sizes - #3168

Draft
alexanderbianchi wants to merge 1 commit into
apache:mainfrom
alexanderbianchi:feat/rest-page-size
Draft

feat(rest): support configurable page sizes#3168
alexanderbianchi wants to merge 1 commit into
apache:mainfrom
alexanderbianchi:feat/rest-page-size

Conversation

@alexanderbianchi

Copy link
Copy Markdown

Which issue does this PR close?

Closes #2934.

What changes are included in this PR?

  • Honor rest-page-size for namespace and table listings, resolving server defaults, client configuration, and server overrides in that order.
  • Add REST_CATALOG_PROP_PAGE_SIZE and with_page_size(u32) to both REST catalog builders. Properties passed to load take precedence over the builder setter.
  • Validate the effective page size after the lazy configuration handshake. Leave pageSize unset when no source configures it.
  • Send pageSize on every page request and an empty initial pageToken to opt into pagination, while continuing to collect all results.
  • Document configuring a REST catalog for DataFusion and add an end-to-end test proving that namespaces and tables from subsequent pages are visible through the provider and SQL. DataFusion continues to use the generic Catalog interface; no REST-specific production dependency or separate pagination setting is introduced.
  • Update the REST public API snapshot.

Are these changes tested?

  • REST tests cover configuration precedence, missing configuration, invalid values, minimum/maximum sizes, multipart parents, opaque page-token encoding, and propagation across pages.
  • A DataFusion integration test uses a mock REST server to paginate namespace and table listings and checks the results through information_schema.tables.
  • cargo test -p iceberg-catalog-rest -p iceberg-datafusion --lib --tests: 181 tests passed.
  • cargo clippy -p iceberg-catalog-rest -p iceberg-datafusion --all-targets -- -D warnings: passed.
  • Formatting, git diff --check, and the generated REST public API snapshot were checked.

AI Disclosure

The implementation, tests, and documentation were developed with assistance from an AI coding agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RestCatalog] Support Page Sizes

1 participant