aws s3 の ec2へのmountには、s3fsやgoofysを使用していましたが、 awsより公式の mount-s3 が公開されていたので、お試し。
- 直接httpを経由せず、aws s3を使用できる為、s3fsやgoofys同様、操作は楽です
- 速度テストには、1MB x 1,000個のファイルを使用しましたが、EBSには遠く及びません
aws s3 の ec2へのmount
mount-s3をオプションなしで実行した場合、ファイルの削除や上書きができない為、 「--allow-delete」「--allow-overwrite」ありでマウントしています。
$ export AWS_ACCESS_KEY_ID=ないしょ $ export AWS_SECRET_ACCESS_KEY=ないしょ $ mkdir mount_s3 $ mount-s3 --allow-delete --allow-overwrite end0tknrnstest mount_s3
mount-s3経由でアップロード - 1MB x 1,000コ → 約7分
[ec2-user@ip-172-31-5-210 tmp]$ date; cp -r org_dummy_files/* mount_s3/; date Wed Aug 21 07:14:01 AM UTC 2024 Wed Aug 21 07:21:12 AM UTC 2024
mount-s3経由でダウンロード - 1MB x 1,000コ → 約2分
$ date; cp mount_s3/* new_dummy_files/; date Wed Aug 21 07:22:35 AM UTC 2024 Wed Aug 21 07:24:15 AM UTC 2024
mount-s3経由で削除 - 1MB x 1,000コ → 約1分
$ date; rm *; date Wed Aug 21 07:24:43 AM UTC 2024 Wed Aug 21 07:25:41 AM UTC 2024
その他 - 今回使用したmount-s3のバージョンとヘルプ
[ec2-user@ip-172-31-5-210 mount_s3]$ mount-s3 --version
mount-s3 1.8.0
[ec2-user@ip-172-31-5-210 mount_s3]$ mount-s3 --help
Mountpoint for Amazon S3
Usage: mount-s3 [OPTIONS] <BUCKET_NAME> <DIRECTORY>
Arguments:
<BUCKET_NAME> Name of bucket to mount
<DIRECTORY> Directory to mount the bucket at
Options:
-f, --foreground Run as foreground process
-h, --help Print help
-V, --version Print version
Bucket options:
--prefix <PREFIX>
Prefix inside the bucket to mount, ending in '/' [default: mount the entire bucket]
--region <REGION>
AWS region of the bucket [default: auto-detect region]
--endpoint-url <ENDPOINT_URL>
S3 endpoint URL [default: auto-detect endpoint]
--force-path-style
Force path-style addressing
--transfer-acceleration
Use S3 Transfer Acceleration when accessing S3. This must be enabled on the bucket.
--dual-stack
Use dual-stack endpoints when accessing S3
--requester-pays
Set the 'x-amz-request-payer' to 'requester' on S3 requests
--bucket-type <BUCKET_TYPE>
Type of S3 bucket to use [default: inferred from bucket name] [possible values: general-purpose, directory]
--storage-class <STORAGE_CLASS>
Set the storage class for new objects
--expected-bucket-owner <AWS_ACCOUNT_ID>
Account ID of the expected bucket owner. If the bucket is owned by a different account, S3 requests fail with an access denied error.
--sse <SSE>
Server-side encryption algorithm to use when uploading new objects [possible values: aws:kms, aws:kms:dsse, AES256]
--sse-kms-key-id <AWS_KMS_KEY_ARN>
AWS Key Management Service (KMS) key ARN to use with KMS server-side encryption when uploading new objects. Key ID, Alias and Alias ARN are all not supported.
--upload-checksums <ALGORITHM>
Checksum algorithm to use for S3 uploads [default: crc32c] [possible values: crc32c, off]
AWS credentials options:
--no-sign-request Do not sign requests. Credentials will not be loaded if this argument is provided.
--profile <PROFILE> Use a specific profile from your credential file.
Mount options:
--read-only Mount file system in read-only mode
--allow-delete Allow delete operations on file system
--allow-overwrite Allow overwrite operations on file system
--auto-unmount Automatically unmount on exit
--allow-root Allow root user to access file system
--allow-other Allow other users, including root, to access file system
--uid <UID> Owner UID [default: current user's UID]
--gid <GID> Owner GID [default: current user's GID]
--dir-mode <DIR_MODE> Directory permissions [default: 0755]
--file-mode <FILE_MODE> File permissions [default: 0644]
Client options:
--maximum-throughput-gbps <N> Maximum throughput in Gbps [default: auto-detected on EC2 instances, 10 Gbps elsewhere]
--max-threads <N> Maximum number of FUSE daemon threads [default: 16]
--part-size <SIZE> Part size for multi-part GET and PUT in bytes [default: 8388608]
--read-part-size <SIZE> Part size for GET in bytes [default: 8388608]
--write-part-size <SIZE> Part size for multi-part PUT in bytes [default: 8388608]
Logging options:
-l, --log-directory <DIRECTORY> Write log files to a directory [default: logs written to syslog]
--log-metrics Enable logging of summarized performance metrics
-d, --debug Enable debug logging for Mountpoint
--debug-crt Enable debug logging for AWS Common Runtime
--no-log Disable all logging. You will still see stdout messages.
Caching options:
--cache <DIRECTORY>
Enable caching of object content to the given directory and set metadata TTL to 60 seconds
--metadata-ttl <SECONDS|indefinite|minimal>
Time-to-live (TTL) for cached metadata in seconds [default: minimal, or 60 seconds if --cache is set]
--max-cache-size <MiB>
Maximum size of the cache directory in MiB [default: preserve 5% of available space]
Advanced options:
--user-agent-prefix <PREFIX> Configure a string to be prepended to the 'User-Agent' HTTP request header for all S3 requests
