Skip to main content
MySQL → AWS S3

Sync MySQL to AWS S3 —
snapshots and streaming CDC.

Land MySQL tables in S3 as date-partitioned Parquet for Athena, Spark or Glue, and stream binlog change events as JSON Lines. Snapshot large tables first, then keep them current.

TL;DR

rsync.ai reads MySQL as a snapshot plus binlog CDC (ROW format) and writes to S3 with Hive-style dt=YYYY-MM-DD partitions: Parquet (Snappy) for batch loads, JSON Lines for CDC change events. Works with MySQL 5.7+, RDS, Aurora and Cloud SQL. Authenticates with an IAM access key. Column-level masking is applied before the S3 write. Batch Parquet output is readable by Athena, Spark and Glue.

  • Parquet output with Snappy compression — Athena, Spark, Glue compatible
  • Hive-style date partitioning (dt=YYYY-MM-DD) out of the box
  • Chunked snapshot for large tables, then streaming CDC
  • PII columns masked or nulled before the S3 write
Start freeSee all MySQL destinations
How it works

How to sync MySQL to AWS S3 — 5 steps

From binlog setup to Parquet files in S3 — typically under 15 minutes.

  1. 1

    Enable MySQL binlog

    Set `binlog_format=ROW` and `log_bin=ON` in your MySQL config. Create a replication user with REPLICATION SLAVE and REPLICATION CLIENT privileges. On RDS or Aurora, enable automated backups and set the `binlog_format` parameter to ROW. Binlog retention should be at least 3 days — enough buffer for a rsync.ai restart without losing events.

    binlog_format=ROW · log_bin=ON · REPLICATION SLAVE + REPLICATION CLIENT
  2. 2

    Connect MySQL source

    Provide host, port, replication username, password, and the database name. rsync.ai connects as a MySQL replica and records the starting binlog position or GTID set before taking the initial snapshot. SSL/TLS and SSH tunnels are supported for MySQL instances inside a private VPC.

    mysql://rsync_cdc:pass@host:3306/mydb
  3. 3

    Connect AWS S3

    Provide your S3 bucket name, region, and an IAM access key ID and secret. The IAM user's policy needs `s3:PutObject`, `s3:GetObject`, and `s3:ListBucket` on your target bucket. You can optionally specify a key prefix (e.g. `mysql/prod/`).

    s3://your-bucket/mysql/ — IAM access key
  4. 4

    Describe the sync in plain English

    Tell rsync.ai what you want: 'Snapshot the MySQL orders table to S3 Parquet daily, then stream changes hourly.' or 'Replicate all tables in mydb to S3 continuously as CDC events.' The AI pipeline planner decides whether to use a full snapshot, incremental CDC, or both — and proposes a partition layout for Athena compatibility.

    No SQL · No YAML · No DAGs
  5. 5

    Approve the S3 path layout and start

    rsync.ai shows the proposed S3 key structure for each table (e.g. `s3://bucket/{prefix}/{dataset}/mydb/orders/dt=2026-10-11/part-000001.parquet`). You can adjust the prefix, the file format (Parquet or JSON Lines), and, for CDC change files, the partition granularity (hourly, daily or monthly). Approve, and the pipeline starts — snapshot first, then streaming CDC batches on the schedule you set.

    s3://bucket/{prefix}/{dataset}/{database}/{table}/dt=YYYY-MM-DD/part-NNNNNN.parquet

MySQL → S3 output layout

Each MySQL table maps to a partitioned Parquet prefix in your S3 bucket. Paths shown for a 2026-10-11 run.

MySQL tableS3 pathNotes
orderss3://bucket/mysql/mydb/orders/dt=2026-10-11/part-000001.parquetSnapshot and incremental batches are Parquet files under a dt= date folder. CDC changes, including deletes, are written as JSON Lines change events; there is no _deleted flag column.
productss3://bucket/mysql/mydb/products/dt=2026-10-11/part-000001.parquetFull snapshot on first run, then CDC change files (JSON Lines). Parquet schema is derived from the MySQL column types.
customerss3://bucket/mysql/mydb/customers/dt=2026-10-11/part-000001.parquetPII columns (email, phone) can be nulled or hashed before writing to S3. Configured per-column in the pipeline.
sessionss3://bucket/mysql/mydb/sessions/dt=2026-10-11/part-000001.parquetHigh-volume table. rsync.ai batches CDC events into JSON Lines files rather than writing one object per event.
transactionss3://bucket/mysql/mydb/transactions/dt=2026-10-11/part-000001.parquetAppend-only. rsync.ai writes INSERT events only. DECIMAL(18,8) preserved as Parquet DECIMAL with matching precision/scale.
audit_logs3://bucket/mysql/mydb/audit_log/dt=2026-10-11/part-000001.parquetAppend-only audit table. Parquet files are Snappy-compressed. Each batch folder also holds `_MANIFEST.json` and `_SUCCESS` markers; the underscore prefix means Athena and Spark skip them when reading the table.

rsync.ai vs. Fivetran, Airbyte, mysqldump+cron for MySQL → S3

How the options stack up for getting MySQL data into your data lake.

Featurersync.aiyouFivetranAirbytemysqldump+cron
Real-time MySQL binlog CDC to S3
Plain-English pipeline setup
Parquet output with Athena partitioning
Self-host in your own VPC
Source-available connector code
Column-level PII masking before S3 write
No per-row / per-MAR pricing
Schema evolution handling

MySQL to S3 — frequently asked

What file formats does rsync.ai use when writing MySQL data to S3?

Parquet (Snappy-compressed) is the recommended format for batch loads: best compression ratio and predicate pushdown in Athena and Spark. JSON Lines (one JSON object per row, optionally gzip-compressed) is available for tools that can't read Parquet, and is what CDC change events are written as. CSV is also available but loses type information.

How are S3 files partitioned and can I query them with Athena?

Batch files are written to `s3://bucket/{prefix}/{dataset}/{database}/{table}/dt=YYYY-MM-DD/part-NNNNNN.parquet`: Hive-style `dt=` partitions that a Glue Crawler discovers automatically and `MSCK REPAIR TABLE` can register without one. CDC change files can be partitioned by hour, day or month. All Parquet files include Parquet statistics so Athena can skip non-matching files.

Which AWS credentials does rsync.ai need for S3?

rsync.ai connects to S3 with an IAM access key ID and secret. Create a dedicated IAM user whose policy is limited to your target bucket and prefix (`s3:PutObject`, `s3:GetObject`, `s3:ListBucket`) and rotate the key on your normal schedule.

How does rsync.ai handle large MySQL tables — full snapshot vs. streaming?

For large tables (millions of rows), rsync.ai takes a chunked snapshot — it reads the table in primary-key-ordered chunks and writes each chunk as a Parquet file to S3. While the snapshot runs, binlog events are buffered. Once the snapshot completes, rsync.ai switches to streaming mode and applies buffered + new CDC events. The switch is atomic — you get a consistent point-in-time snapshot plus all changes since.

How does rsync.ai handle MySQL schema changes (ALTER TABLE) for S3 output?

When rsync.ai detects a DDL event in the binlog, it writes the current in-flight Parquet file, then starts a new file with the updated schema. Older Parquet files retain the old schema. Athena handles schema evolution gracefully if you define your table with column-by-name projection. New columns and column type changes are applied and recorded in the pipeline's history; only a dropped table waits for a person to decide what happens next.

Is the MySQL → S3 output compatible with Apache Spark and AWS Glue?

Yes. The Parquet files rsync.ai writes use standard Parquet column statistics and Hive-style partitioning, which Spark, Glue ETL, AWS Glue Crawler, Presto, and DuckDB all understand natively. For batch-loaded Parquet, Spark can read the table prefix directly and discover the dt= partitions: `spark.read.parquet('s3://bucket/{prefix}/{dataset}/mydb/orders/')`. Each batch folder also holds `_MANIFEST.json` and `_SUCCESS` markers, which readers skip because of the underscore prefix.

Can I redact or mask PII columns before they land in S3?

Yes. In the pipeline configuration you mark individual columns as Null, Hashed (SHA-256), or Truncated. Masking is applied in rsync.ai's processing layer before the file is written, so the raw value is not written to S3. This is especially useful for the customers table where email and phone should not land in a data lake that many team members can query.

Is rsync.ai self-hosted for MySQL → S3?

rsync.ai Cloud is live today at app.rsync.ai — nothing to deploy. You can self-host the full stack today — one command (`curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync.ai/main/install.sh | bash`) brings it up inside your own VPC, so your MySQL credentials, AWS credentials, and row data never leave your network. rsync.ai is source-available under the rsync.ai Source-Available License: self-hosting is free for your company's own internal use, while reselling it or hosting it for others as a service needs a commercial license.

Run your first MySQL to AWS S3 pipeline free

The first 10 GB of data moved is free. Describe the sync in plain English and review the plan before anything runs.

Start free

Related MySQL integrations

MySQL → PostgreSQLMySQL → Google SheetsPostgreSQL → AWS S3All MySQL integrations