feature: add extra virtual host style endpoints support

[LinPr]

This PR add the user custom endpoints that need to send request with virtual host style

reference issues:
#794
#518

Here I add a command line flag and also support fetch config from environment variable,

--addressing-style value use virtual host style or path style endpoint: (path, virtual) (default: path)

the address-style config is compatible with the aws s3 cli config，here is the aws document , and you can just search the address_style key word in the page. https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

NAME:
   s5cmd - Blazing fast S3 and local filesystem execution tool

USAGE:
   s5cmd [global options] command [command options] [arguments...]

COMMANDS:
   ls              list buckets and objects
   cp              copy objects
   rm              remove objects
   mv              move/rename objects
   mb              make bucket
   rb              remove bucket
   select          run SQL queries on objects
   du              show object size usage
   cat             print remote object content
   pipe            stream to remote from stdin
   run             run commands in batch
   sync            sync objects
   version         print version
   bucket-version  configure bucket versioning
   presign         print remote object presign url
   head            print remote object metadata
   help, h         Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --addressing-style value       use virtual host style or path style endpoint: (path, virtual) (default: path) [$S3_ADDRESSING_STYLE]
   --credentials-file value       use the specified credentials file instead of the default credentials file
   --dry-run                      fake run; show what commands will be executed without actually executing them (default: false)
   --endpoint-url value           override default S3 host for custom services [$S3_ENDPOINT_URL]
   --help, -h                     show help (default: false)
   --install-completion           get completion installation instructions for your shell (only available for bash, pwsh, and zsh) (default: false)
   --json                         enable JSON formatted output (default: false)
   --log value                    log level: (trace, debug, info, error) (default: info)
   --no-sign-request              do not sign requests: credentials will not be loaded if --no-sign-request is provided (default: false)
   --no-verify-ssl                disable SSL certificate verification (default: false)
   --numworkers value             number of workers execute operation on each object (default: 256)
   --profile value                use the specified profile from the credentials file
   --request-payer value          who pays for request (access requester pays buckets)
   --retry-count value, -r value  number of times that a request will be retried for failures (default: 10)
   --stat                         collect statistics of program execution and display it at the end (default: false)
   --use-list-objects-v1          use ListObjectsV1 API for services that don't support ListObjectsV2 (default: false)

And I well tested it, using this on our daily work, no bug being found till now.

I can give a example here :

1. If I want to upload a file to remote object storage , Firstly, I need to set up config with environment variable

export AWS_ACCESS_KEY_ID='xxxxxxxxxxx'
export AWS_SECRET_ACCESS_KEY='xxxxxxxxxx'
export S3_ENDPOINT_URL='http://oss-cn-hangzhou.aliyuncs.com'

1. Secondly, because I want to sent request with virtual host style, I need a another env or command line flag

export S3_ADDRESSING_STYLE="virtual"      

1. Now you can upload or down load file with the s5cmd command

/s5cmd cp ./testfile s3://restic-backup-good/hello/
or I can run:
/s5cmd  --addressing-style=virtual cp ./testfile s3://restic-backup-good/hello/

output:
cp ./testfile s3://restic-backup-good/hello/testfile 

let’s see the request detail on wireshark. it just send the request with virtual host style , a bucket name as prefix of host

[salanki]

Getting this merged would be very nice.

[ppcololo]

any updates?

[jprobichaud]

I would second the importance of this PR. We are using CoreWeave wth their CAIOS storage, which requires the virtual style, and not being able to use s5cmd is very painful !

[mannreis]

@ilkinulas @sonmezonur are there any hopes on getting this merged? It would be really nice to have this upstream!

[narenandu]

Could we get this PR merged or even #859

[Themis3000]

@jprobichaud

I would second the importance of this PR. We are using CoreWeave wth their CAIOS storage, which requires the virtual style, and not being able to use s5cmd is very painful !

You can still use s5cmd with other hosts. For example this is how I use s5cmd with mega's s4 storage:

s5cmd --endpoint-url https://s3.ca-montreal.megas4.com ls

Note that the --endpoint-url option must be specified before the command.