Skip to main content
PostgreSQL → AWS S3

Export PostgreSQL to S3
as Parquet or JSON Lines.

Land PostgreSQL tables in S3 for data lake ingestion, compliance archiving or long-term storage, described in plain English. Batch loads are date-partitioned Parquet; CDC changes are written as JSON Lines.

TL;DR

rsync.ai reads PostgreSQL via logical replication (CDC mode) or consistent COPY (snapshot mode) and writes to S3 as Parquet (Snappy) for batch loads or JSON Lines for CDC changes. Partitions are Hive-style (dt=YYYY-MM-DD), with hourly partitions available for CDC files. Schema is inferred from Postgres column types. Initial snapshot + incremental CDC — or scheduled full snapshot. Works with self-hosted Postgres, RDS, Aurora, Supabase, Neon.

  • Parquet (Snappy-compressed) or JSON Lines — your choice
  • Date or hour partitioning — Hive-style for Athena / Glue
  • CDC streaming or scheduled snapshots — or both
  • Self-host in your own VPC — source-available under the rsync.ai Source-Available License
Start freeAll PostgreSQL integrations
How it works

How to sync PostgreSQL to AWS S3 — 5 steps

From enabling logical replication to Parquet files landing in your bucket.

  1. 1

    Enable logical replication on PostgreSQL

    Set `wal_level=logical` in postgresql.conf (or via the RDS/Aurora parameter group with `rds.logical_replication=1`). Set `max_replication_slots` ≥ 2. Create a publication for the tables you want to export: `CREATE PUBLICATION rsync_pub FOR ALL TABLES;`. For batch snapshot mode only (no CDC streaming), you can skip the replication slot — rsync.ai will use a consistent COPY snapshot instead.

    wal_level=logical · pgoutput · or batch snapshot mode
  2. 2

    Connect PostgreSQL

    Provide host, port (default 5432), user, password, and database. For CDC streaming mode also provide the replication slot name and publication name. The user needs SELECT on target tables (snapshot mode) or REPLICATION privilege (CDC mode). rsync.ai supports self-hosted Postgres 10+, Amazon RDS, Aurora, Google Cloud SQL, Supabase, and Neon.

    CDC mode: REPLICATION role · Snapshot mode: SELECT only
  3. 3

    Connect AWS S3

    Provide the S3 bucket name, region, and an IAM access key ID and secret. The IAM user's policy needs `s3:PutObject`, `s3:GetObject`, and `s3:ListBucket` on your bucket and prefix.

    IAM access key and secret
  4. 4

    Describe the sync in plain English

    Type what you want: 'Snapshot all tables from the public schema to S3 as Parquet daily at 2am, then stream CDC changes hourly.' rsync.ai determines the S3 path layout, file format, and partitioning strategy. You can also specify a custom path prefix, format (Parquet or JSON Lines), or gzip compression for JSON Lines (Parquet files use Parquet's built-in Snappy compression).

    No SQL · No YAML · Parquet / JSON Lines
  5. 5

    Approve path layout and start the pipeline

    rsync.ai shows the proposed S3 path structure — e.g. `s3://your-bucket/{prefix}/{dataset}/{database}/{table}/dt=2026-10-11/part-000001.parquet` — and the Parquet schema inferred from PostgreSQL column types. Review partitioning (date, hour), file naming, and column types. Approve and the pipeline takes an initial consistent snapshot, then streams incremental changes on the schedule you set.

    s3://bucket/{prefix}/{dataset}/{database}/{table}/dt=YYYY-MM-DD/part-NNNNNN.parquet

PostgreSQL → S3 path layout

Default path structure rsync.ai proposes. Customize prefix, format, and partitioning before approving.

PostgreSQL tableS3 path (Parquet)Notes
public.userss3://bucket/postgres/public/users/dt=YYYY-MM-DD/part-000001.parquetUUID columns stored as STRING in Parquet. TIMESTAMPTZ→TIMESTAMP (UTC).
public.orderss3://bucket/postgres/public/orders/dt=YYYY-MM-DD/part-000001.parquetNUMERIC(10,2)→DECIMAL(10,2) in Parquet. JSONB columns stored as STRING.
public.productss3://bucket/postgres/public/products/dt=YYYY-MM-DD/part-000001.parquetTEXT[]→repeated STRING group. JSONB metadata→STRING.
public.eventss3://bucket/postgres/public/events/dt=YYYY-MM-DD/part-000001.parquetHigh-volume table — hourly partitioning recommended. BIGSERIAL→INT64.
public.sessionss3://bucket/postgres/public/sessions/dt=YYYY-MM-DD/part-000001.parquetTIMESTAMPTZ expires_at→TIMESTAMP. UUID columns→STRING.
public.audit_logss3://bucket/postgres/public/audit_logs/dt=YYYY-MM-DD/part-000001.parquetJSONB row_data→STRING. Append-only table — no UPDATE/DELETE CDC needed.

rsync.ai vs. Fivetran, Airbyte, custom pg_dump scripts for PostgreSQL → S3

What you give up — and gain — choosing rsync.ai for Postgres to S3 pipelines.

Featurersync.aiyouFivetranAirbytepg_dump scripts
Real-time CDC streaming to S3
Plain-English pipeline setup
Parquet output with schema manifest
Self-host in your own VPC
Source-available connector code
No per-row / per-MAR pricing
Resumable snapshots (no restart on failure)
PII masking before S3 write

PostgreSQL to AWS S3 — frequently asked

What file formats does rsync.ai write to S3?

Parquet (recommended, Snappy-compressed) and newline-delimited JSON (JSON Lines); CSV is also available. Parquet is recommended for data lake use cases — it's columnar, compressed, and natively supported by Athena, Redshift Spectrum, BigQuery, Databricks, and DuckDB. JSON is useful for event streaming or downstream systems that consume JSON. CSV is available for legacy tooling that doesn't speak Parquet.

How does S3 path partitioning work?

By default rsync.ai writes Hive-style date partitions: `s3://bucket/{prefix}/{dataset}/{database}/{table}/dt=YYYY-MM-DD/part-NNNNNN.parquet`, which Athena and Glue crawlers pick up. CDC change files can also be partitioned by hour (`dt=YYYY-MM-DD/hour=HH/`) or month. You set the strategy in plain English or in the approve step.

Which AWS credentials does rsync.ai need for S3?

rsync.ai connects to S3 with an IAM access key ID and secret. Create a dedicated IAM user with a policy scoped to your bucket and path prefix (`s3:PutObject`, `s3:GetObject`, `s3:ListBucket`) and rotate the key regularly.

How does rsync.ai handle large tables during the initial snapshot?

The initial snapshot uses a consistent PostgreSQL COPY (or a repeatable read transaction for logical replication mode) so you get a point-in-time consistent snapshot even for tables with billions of rows. rsync.ai writes large tables as a series of batches. Each batch is uploaded under a pending prefix and only counts as committed once its manifest and `_SUCCESS` marker are written, so a failed run never leaves a half-written batch looking complete.

How does schema evolution work in Parquet files?

Parquet files are immutable once written. When rsync.ai detects a new column in PostgreSQL (via schema drift detection), it starts writing new files with the expanded schema. Old files retain the old schema. Tools like Athena, Spark, and DuckDB handle schema evolution via schema merging (union of all columns, NULL-filled for missing fields). rsync.ai also writes a JSON schema manifest alongside each batch so downstream tools can discover column additions.

What happens if a snapshot is interrupted mid-table?

Each batch is uploaded under a pending prefix and committed by writing its manifest and `_SUCCESS` marker. Before writing, rsync.ai checks whether the batch's manifest already exists, so a batch that already committed is not written twice, and an interrupted batch is simply written again.

Can rsync.ai mask PII columns in Parquet files written to S3?

Yes. Mark any column as `masked` in the pipeline configuration — rsync.ai replaces the value with a SHA-256 hash or NULL before writing to S3. The raw value is not written to S3. This keeps email, phone and payment fields out of your data lake while you retain full event history. Whether hashed or masked values satisfy GDPR or CCPA is a question for your compliance team.

Is rsync.ai self-hosted for PostgreSQL to S3?

rsync.ai Cloud is live today at app.rsync.ai — nothing to deploy. You can self-host the full stack today — one command (`curl -sSL https://raw.githubusercontent.com/rsync-ai/rsync.ai/main/install.sh | bash`) brings it up inside your own VPC, so your PostgreSQL credentials and AWS keys never leave your network. rsync.ai is source-available under the rsync.ai Source-Available License: self-hosting is free for your company's own internal use, while reselling it or hosting it for others as a service needs a commercial license.

Run your first PostgreSQL to AWS S3 pipeline free

The first 10 GB of data moved is free. Describe the sync in plain English and review the plan before anything runs.

Start free

Other PostgreSQL destinations

PostgreSQL → MySQLPostgreSQL → Google SheetsAll PostgreSQL integrations