feat(glue): Support AWS STS AssumeRole for Glue catalog authentication by jlambatl · Pull Request #2396 · apache/iceberg-rust

jlambatl · 2026-05-01T23:17:36Z

Adds support for authenticating to the AWS Glue Data Catalog using temporary IAM credentials obtained via AWS STS AssumeRole. This enables cross-account Glue catalog access and aligns with AWS security best practices, which recommend temporary credentials over long-term static access keys.

Three new configuration properties are introduced, mirroring the naming convention already used by the S3 storage layer

Which issue does this PR close?

Closes feat(glue): Support AWS STS AssumeRole for Glue catalog authentication #2394

What changes are included in this PR?

crates/catalog/glue/src/utils.rs — Added three new public constants and extended create_sdk_config() to build an AssumeRoleProvider from aws_config::sts when a role ARN is configured. The base credential chain (static keys, profile, or default chain) is used to call STS, and the resulting temporary credentials are used for all Glue SDK calls.
crates/catalog/glue/src/catalog.rs — Propagates the three STS role properties into the S3 FileIO property map in GlueCatalog::new(), so that both Glue API calls and S3 file I/O operate under the same assumed role.
crates/catalog/glue/src/lib.rs — Exports the three new constants as part of the public API.
crates/catalog/glue/Cargo.toml — Added explicit aws-sdk-sts dependency (already present transitively) with a [package.metadata.cargo-machete] ignore entry to suppress the expected false positive from cargo-machete (the crate is consumed via aws_config::sts re-exports).
Cargo.toml — Added aws-sdk-sts to workspace dependencies.
crates/catalog/loader/tests/common/mod.rs — Added glue_catalog_assume_role() integration test helper, driven by environment variables, for use in #[ignore]-gated integration tests requiring a live AWS environment.
bump the aws-sdk-glue dependency version to the latest.

Are these changes tested?

4 new unit tests in crates/catalog/glue/src/utils.rs covering: ARN-only configuration, custom session name, all optional fields with a custom endpoint, and a regression guard confirming the existing static-credential path is unaffected.
All 18 tests in iceberg-catalog-glue pass.
cargo fmt, cargo clippy -D warnings, and cargo machete all pass.

… iceberg catalog crate

Kurtiscwright

Hi thank you for opening this PR!

…onflict

…ws utility crate

jlambatl · 2026-06-04T02:34:40Z

@Kurtiscwright I think that this PR is now ready for review again. I've moved the aws components that are shared between S3Tables and Glue into an integrations crate, fixed the machete items.

github-actions · 2026-07-05T00:37:42Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

# Conflicts: # Cargo.toml # crates/catalog/glue/public-api.txt # crates/catalog/glue/src/catalog.rs # crates/catalog/s3tables/src/utils.rs

jlambatl · 2026-07-07T04:57:01Z

resynced with the main branch and fixed the conflicts.

manuzhang · 2026-07-10T04:41:21Z

@jlambatl Please check my comments (disclosure: drafted by LLM) and rebase on main, which should resolve the security audit error.

# Conflicts: # crates/catalog/glue/src/catalog.rs

jlambatl · 2026-07-15T22:56:19Z

@manuzhang I've reviewed your feedback and confirmed that the concerns you raised do exist.

I've manually refactored the code and addressed the feedback. I've then had Codex and Gemini review it, which found some problems that have now been resolved. This Pr has become much larger than i anticipated, however I hope that it is now in a state that is more maintainable moving forward.

Keen to hear feedback, thanks in advance.

manuzhang

@jlambatl Left some comments with help of LLM. Please check again.

Previously, the catalog's own service endpoint override (e.g. a custom Glue or S3Tables/LocalStack endpoint passed via `endpoint_uri`) was applied to the base SdkConfig before it was used to configure the STS AssumeRoleProvider via `.configure(&base_config)`. This meant every AssumeRole call would silently target the catalog's endpoint instead of the real STS endpoint whenever a catalog endpoint override was set. Fix: build the base config used for STS authentication without any endpoint override, so STS always resolves to its real regional endpoint (or an AWS_ENDPOINT_URL_STS/profile override). The catalog's endpoint override is now applied only to the final SdkConfig returned for the catalog's own service client, after the STS-backed credentials provider has already been constructed. Addresses review comment on apache#2396 (review 4719317386).

`map_aws_to_s3_properties` previously only used the explicit `region_name` property to populate FileIO's `s3.region`. Whenever the region was instead resolved implicitly by the AWS SDK (via an AWS profile, the AWS_REGION env var, or IMDS), FileIO was left without a region, silently falling back to a default that may not match the SDK client actually used by the catalog. Fix: add a `resolved_region` parameter to `map_aws_to_s3_properties`, sourced from `sdk_config.region()` in both the Glue and S3Tables catalog constructors, used as a last-resort fallback after any explicit `s3.region`/`region_name` property. Regenerated public-api.txt for the iceberg-aws crate to reflect the new parameter. Addresses review comment on apache#2396 (review 4719317386).

…ilure RefreshingCredentialsProvider previously propagated any STS AssumeRole error the instant cached credentials crossed the 5-minute early-refresh buffer, even though the cached credentials could still be valid for real for several more minutes. A single transient STS error (throttling, a network blip, temporary IAM/STS unavailability) would therefore turn into a hard failure for every caller, instead of gracefully degrading. Fix: on a refresh failure, fall back to the cached credentials as long as their real (unbuffered) expiry is still in the future, logging a warning. Only propagate the error when there is no usable cached credential left (none cached, or actually expired). To avoid turning a sustained STS outage into a self-inflicted retry storm (every credential consumer re-attempting STS on every call, serialized behind a shared mutex held across each slow/retried attempt), failed refreshes also set a 30s backoff window during which cached-but-stale credentials are served without re-contacting STS. Adds the log crate as a dependency for the fallback warning. Addresses review comment on apache#2396 (review 4719317386).

…s-assume-role

manuzhang

@jlambatl Could you please resolve conflicts with main branch?

github-actions · 2026-08-30T00:26:37Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

jimmystewpot · 2026-08-30T01:47:46Z

Is anyone gonna be reviewing this PR?

initial commit for adding aws sts assume role support to the aws-glue…

cd15f4c

… iceberg catalog crate

Kurtiscwright suggested changes May 28, 2026

View reviewed changes

Comment thread crates/catalog/glue/Cargo.toml Outdated

Comment thread crates/catalog/glue/Cargo.toml Outdated

Comment thread crates/catalog/glue/src/utils.rs Outdated

jlambatl added 4 commits June 1, 2026 11:59

refactor aws sts into a utils package and clean up machete error

8d93ab8

Merge branch 'main' into aws-sts-assume-role and resolve Cargo.lock c…

2036111

…onflict

Merge upstream/main into aws-sts-assume-role and resolve conflicts

b4c282a

fix public api with exports moved from glue and s3tables to the new a…

12720a7

…ws utility crate

github-actions Bot added the stale label Jul 5, 2026

Merge remote-tracking branch 'upstream/main' into aws-sts-assume-role

d81608c

# Conflicts: # Cargo.toml # crates/catalog/glue/public-api.txt # crates/catalog/glue/src/catalog.rs # crates/catalog/s3tables/src/utils.rs

jlambatl requested a review from Kurtiscwright July 7, 2026 04:57

github-actions Bot removed the stale label Jul 8, 2026

manuzhang reviewed Jul 10, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

manuzhang reviewed Jul 10, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

manuzhang reviewed Jul 10, 2026

View reviewed changes

Comment thread crates/catalog/s3tables/src/catalog.rs Outdated

manuzhang reviewed Jul 10, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

Merge remote-tracking branch 'upstream/main' into aws-sts-assume-role

5599bf3

# Conflicts: # crates/catalog/glue/src/catalog.rs

Merge branch 'main' into aws-sts-assume-role

4358f56

jlambatl requested a review from manuzhang July 15, 2026 22:56

manuzhang reviewed Jul 17, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

manuzhang reviewed Jul 17, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

manuzhang reviewed Jul 17, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

manuzhang suggested changes Jul 17, 2026

View reviewed changes

jlambatl added 4 commits July 21, 2026 18:23

Merge remote-tracking branch 'origin/aws-sts-assume-role' into aws-st…

7989dd0

…s-assume-role

jlambatl requested a review from manuzhang July 22, 2026 05:55

manuzhang suggested changes Jul 22, 2026

View reviewed changes

manuzhang reviewed Jul 22, 2026

View reviewed changes

Comment thread crates/integrations/aws/src/config.rs Outdated

jlambatl requested a review from manuzhang July 24, 2026 08:50

github-actions Bot added the stale label Aug 30, 2026

github-actions Bot removed the stale label Aug 31, 2026

jlambatl added 3 commits September 8, 2026 11:43

Merge remote-tracking branch 'upstream/main' into aws-sts-assume-role

0caa2ce

test: isolate STS endpoint configuration

dfe8650

merge and fix from upstream

2c5aa69

jlambatl force-pushed the aws-sts-assume-role branch from 7dbc525 to 2c5aa69 Compare September 8, 2026 09:08

Uh oh!

Conversation

jlambatl commented May 1, 2026

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Kurtiscwright left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlambatl commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jul 5, 2026

Uh oh!

jlambatl commented Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manuzhang commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlambatl commented Jul 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

jimmystewpot commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manuzhang commented Jul 10, 2026 •

edited

Loading