aws-sdk-pandas

Project Overview

AWS SDK for pandas (package name awswrangler) is an open-source Python library from AWS Professional Services, positioned as pandas on AWS: it connects DataFrames directly to AWS data services. It provides high-level APIs for S3 (Parquet, CSV, JSON, Excel), Athena, Redshift, Glue Catalog, Timestream, DynamoDB, OpenSearch, Neptune, and more — reading databases, landing data in a lake, and running SQL queries take only a line or two. With Modin and Ray, the same code can scale out to a cluster.

The official high-speed channel between pandas and AWS data services.

Key Features

• S3 data lake I/O: wr.s3.to_parquet / read_parquet work with datasets on S3 and register Glue tables automatically
• Athena queries: wr.athena.read_sql_query runs SQL and returns results straight into a DataFrame
• Redshift integration: connections from Glue Catalog, with query, COPY/UNLOAD, and upsert support
• Multiple databases: bulk read/write for MySQL, PostgreSQL, SQL Server, and Oracle
• Timestream time series: write and query between DataFrames and Amazon Timestream
• DynamoDB and OpenSearch: wrappers for key-value data and search clusters
• Scale-out execution: distribute processing across clusters with Modin and Ray
• Fully typed: type annotations throughout, checked with mypy, IDE-friendly

Target Audience

• Data engineers on AWS
• Data scientists analyzing with pandas
• Developers building ETL pipelines
• Teams running data jobs on Lambda / Glue / EMR

Quick Start

1. Install: pip install awswrangler (optional modules need extras like pip install 'awswrangler[redshift]')
2. Configure AWS credentials and start using:

import awswrangler as wr
import pandas as pd

df = pd.DataFrame({'id': [1, 2], 'value': ['foo', 'boo']})

# Write to the data lake and register a Glue table
wr.s3.to_parquet(df=df, path='s3://bucket/dataset/', dataset=True,
                 database='my_db', table='my_table')

# Query back into a DataFrame via Athena
df = wr.athena.read_sql_query('SELECT * FROM my_table', database='my_db')

3. See the official docs and the tutorials directory of Notebooks for more

Core Advantages

• Minimal code: operations that take dozens of boto3 lines usually take one or two
• Officially maintained: service coverage and compatibility handled by the AWS team
• Local to cloud consistency: one API moves seamlessly from notebooks to production pipelines

System Requirements

Notes

• Optional modules since 3.0: install extras explicitly, e.g. awswrangler[redshift] or awswrangler[mysql]; the default install includes core features only
• Incurs AWS costs: Athena scans and S3 requests are billed by usage — watch data scan volume before querying
• Lambda deployment: an official Lambda Layer is provided, so no manual dependency packaging is needed