s5cmd

Project Overview

s5cmd is a very fast parallel S3 and local filesystem execution tool that supports a multitude of operations, including tab completion and wildcard support, making it handy for object storage workflows with large numbers of files.

Key Features

• List buckets and objects
• Upload, download, or delete objects
• Move, copy, or rename objects
• Set Server Side Encryption using AWS KMS
• Set Access Control List (ACL) for objects/files
• Print object contents to stdout
• Select JSON records from objects using SQL expressions
• Create or remove buckets
• Summarize objects sizes, grouping by storage class
• Wildcard support for all operations
• Multiple arguments support for delete operation
• Command file support to run commands in batches at very high execution speeds
• Dry run support
• S3 Transfer Acceleration support
• Google Cloud Storage and any other S3 API compatible service support

Target Audience

• Data engineers and developers who need high-performance object storage operations
• Administrators handling large numbers of files on S3 or compatible storage services

Quick Start

1. Download pre-built binaries for Linux, macOS, or Windows from the Releases page
2. macOS users can install via Homebrew: brew install peak/tap/s5cmd
3. Use the s5cmd command to perform object storage operations

Core Advantages

• Blazing speed: 32x faster than s3cmd and 12x faster than aws-cli for uploads
• Bandwidth saturation: Can saturate a 40Gbps link (~4.3 GB/s) for downloads, while s3cmd and aws-cli only reach 85 MB/s and 375 MB/s respectively
• Broad compatibility: Supports AWS S3, Google Cloud Storage, and any S3-compatible service
• Feature-rich: Supports wildcards, batch command files, and structured logging

System Requirements

• OS: Linux, macOS, Windows
• Architectures: x86_64, arm64
• Additional Dependencies: None (pre-built binaries)