Convex storage: Stop losing data in containers
Convex backends ship with local filesystem storage. That default traps your data inside ephemeral containers. Escaping requires seven distinct S3 buckets to separate modules, files, and search indexes. You cannot simply mount a volume and walk away; the architecture demands a hard split between compute and persistence.
Traffic routing hinges on explicit environment variables like S3_STORAGE_FILES_BUCKET and S3_STORAGE_MODULES_BUCKET. While AWS is the standard target, the S3_ENDPOINT_URL variable unlocks compatibility with R2 or other AWS S3 API drop-in replacements. This flexibility lets teams swap providers without touching application logic, provided they respect strict bucket naming conventions.
Moving between local and cloud storage breaks the "it just works" promise. You must manually export a snapshot via the CLI, spin up a fresh backend instance, and then import that snapshot. This two-step snapshot export and import sequence is non-negotiable. It guarantees data integrity while swapping the underlying storage engine, avoiding the corruption risks of direct file manipulation.
The Role of S3 Buckets in Convex Backend Architecture
S3 Storage vs Default Filesystem in Convex Backend
By default, Convex backends hoard file data on the filesystem within the docker container. This couples your data lifecycle to the container's uptime. Moving to external object storage severs this link, forcing persistence logic to stand independent of compute resources. Administrators must define specific environment variables to steer traffic toward remote buckets instead of local disk paths. Credentials like `AWS_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` sit alongside dedicated bucket names for exports, modules, files, and search indexes. Deployments using R2 or other drop-in replacements compatible with the AWS S3 API require the optional `S3_ENDPOINT_URL` variable to route requests correctly.
| Feature | Default Filesystem | S3 Object Storage |
|---|---|---|
| Persistence Scope | Container Lifecycle | Independent of Compute |
| Configuration | Zero-config default | Requires env vars |
| Portability | Host-bound | Region-agnostic |
Don't expect an automatic sync when switching providers. The workflow is manual and deliberate. Teams execute a snapshot export to a local path, reconfigure environment variables, then run a snapshot import to populate the new backend instance. This explicit cutover maintains data consistency. You can find the required configuration schema in the `get-convex/convex-backend` GitHub repository under the `self-hosted/advanced/s3_storage.md` path.
Configuring AWS Buckets for Convex User Files and Snapshots
Local filesystems inside Docker containers hold file data until operators switch to external storage solutions. Decoupling persistence from compute lifecycle prevents data loss when containers restart or scale.
Defining specific environment variables routes traffic to the correct destinations. The configuration mandates setting `S3_STORAGE_FILES_BUCKET` for application assets and `S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET` for data recovery operations. Additional buckets handle modules and search indexes separately. Teams using S3-compatible storage providers like the provider or the provider must set the `S3_ENDPOINT_URL` variable to override default AWS region logic.
You pay for this flexibility with increased configuration complexity. Data portability across cloud regions remains possible through snapshot export and import commands. Once configured, these buckets function as the primary storage layer for any Convex instance. Operators gain control over retention policies and access controls that local disks cannot provide. The setup takes longer, but the resulting infrastructure design offers far greater long-term flexibility.
Mechanics of Snapshot Export and Import Processes
Snapshot Export and Import Command Mechanics
Running `npx convex export --path` initiates the migration by serializing local filesystem data into a portable snapshot file. This command captures the complete state of the backend, ensuring that no user files or search indexes are lost during the transition. Operators must then configure a fresh backend instance with the target S3 Storage environment variables before proceeding. The second phase executes `npx convex import --replace-all` to ingest the serialized data into the new S3-backed environment. This two-step process effectively decouples the data layer from the underlying storage infrastructure.
| Step | Command | Action |
|---|---|---|
| 1 | `npx convex export` | Serializes local data to a file |
| 2 | `npx convex import` | Ingests file into S3 backend |
The snapshot export creates a consistent point-in-time view, while the import operation reconstructs this state on the remote provider. The procedure ensures that switching between local disks and object storage remains deterministic rather than relying on complex, error-prone manual copying.
Executing Data Migration to Fresh S3 Backends
Operators initiate data transfer by running `npx convex export --path` to serialize the local state into a portable file. This command captures the complete backend context, ensuring no user files or search indexes are lost during the transition. The subsequent phase requires provisioning a fresh backend instance configured with target S3 Storage environment variables before ingestion. Administrators then execute `npx convex import --replace-all` alongside the export path to ingest data into the new object store.
- Export local data to a set path.
- Configure AWS credentials and bucket names for the new provider.
- Run the import command with the replace-all flag.
The replace-all flag is destructive; it wipes the target backend storage before importing, preventing data merging errors. The operation decouples the application logic from the underlying filesystem, enabling scalable architectures.
| Parameter | Requirement | Purpose |
|---|---|---|
| `AWS_REGION` | Mandatory | Specifies region for S3 configuration |
| `S3_ENDPOINT_URL` | Optional | Enables R2 or compatible APIs |
| `--replace-all` | Mandatory | Clears target before import |
This approach guarantees that the fresh backend starts with a clean, consistent state derived directly from the source snapshot.
Configuring AWS Credentials for S3 Integration
Defining AWS Environment Variables for S3 Storage
Transitioning the Convex backend from local filesystem persistence to scalable object storage demands specific environment variables before startup. Standard authentication starts the process with `AWS_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY`. These primitives authorize the application to interact with the AWS S3 API. Distinct buckets handle different data types. Operators define `S3_STORAGE_EXPORTS_BUCKET` for snapshots. They set `S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET` for imports. Code resides in `S3_STORAGE_MODULES_BUCKET`. User files go to `S3_STORAGE_FILES_BUCKET`. Search indexes live in `S3_STORAGE_SEARCH_BUCKET`.
Teams using compatible alternatives like the provider add the optional `S3_ENDPOINT_URL` variable to redirect traffic away from AWS infrastructure. This flexibility supports hybrid architectures but demands precise bucket naming. Local storage simplifies initial development yet switching between local storage and S3 storage requires running a snapshot export and import cycle to migrate data. The backend uses these values to determine storage location; omitting the required variables results in the backend storing file data on the filesystem within the docker container by default. Validating these definitions in a staging environment prevents data portability issues before production rollout.
Applying S3 Endpoint URLs for R2 Compatibility
Setting the S3_ENDPOINT_URL environment variable redirects the backend from AWS global infrastructure to alternative object stores like Cloudflare R2. This configuration becomes mandatory when using drop-in replacements that adhere to the AWS S3 API but operate on different network addresses. The client library targets standard AWS endpoints without this specific directive.
Cost-optimized architectures often deploy this variable alongside standard credentials to apply diverse storage providers supporting S3 protocols without altering application logic. The variable accepts a full URI string, allowing smooth integration with these providers. Validating this endpoint early in the deployment pipeline helps prevent latent connectivity issues during high-volume snapshot imports. Protocol scheme mismatches (HTTP vs HTTPS) in the endpoint URL trigger certificate validation errors that halt startup sequences. Automatic region discovery may not apply when using custom endpoints, requiring manual verification of the target bucket's location and settings.
Teams migrating from local filesystems must keep the endpoint consistent across both export and import phases to maintain data integrity. The migration process involves running `npx convex export` to create a snapshot, setting up the fresh backend with the new storage provider variables, and then running `npx convex import --replace-all` to load the data.
About
Alex Kumar is a Senior Platform Engineer and Infrastructure Architect at Rabata.io, where he specializes in Kubernetes storage architecture and cost optimization. His daily work involves designing resilient, cloud-native systems that rely heavily on scalable object storage, making him uniquely qualified to guide developers through S3 bucket configuration. In this article, Alex uses his hands-on experience with persistent storage and CSI drivers to explain how to transition backend file data from local Docker volumes to reliable S3-compatible storage. At Rabata.io, an S3-compatible provider focused on high-performance and significant cost savings compared to AWS, Alex routinely helps enterprises eliminate vendor lock-in while maintaining true API compatibility. By connecting his practical infrastructure challenges with Rabata's mission to democratize enterprise-grade storage, he provides actionable insights for engineers seeking to optimize their cloud storage strategies without sacrificing speed or reliability.
Conclusion
Inconsistent endpoint configuration creates silent data fragmentation. It does not always crash the system immediately. When teams rely on default filesystem storage within containers during development, they incur a hidden operational debt where data portability depends entirely on manual snapshot cycles. This friction increases significantly when migrating between local environments and cloud providers like the provider, as protocol mismatches halt startup sequences before validation logic ever executes. You must treat the S3_ENDPOINT_URL as a critical architectural boundary, not merely a connection string.
Implement a strict rule where every local development environment mirrors the production endpoint configuration from day one. Do not wait for a substantial migration event to test these variables; instead, enforce this alignment before any code reaches the staging branch. This approach eliminates the need for risky data export and import loops later in the lifecycle. Start by auditing your current `.env` files this week to ensure the endpoint variable is explicitly set for all local instances, even if they currently point to AWS. Verifying that your backend rejects startup without this specific directive prevents the silent fallback to ephemeral container storage. By hardening this configuration early, you ensure that your storage layer remains portable across any S3-compatible provider without requiring application logic changes.
Frequently Asked Questions
You must configure seven distinct S3 buckets for full cloud integration. This setup separates modules, files, and search indexes rather than using a single unified store for all data types.
The primary cost is increased configuration complexity compared to local disk usage. Teams trade zero-config defaults for manual environment variable setup to gain region-agnostic portability and durable object storage.
Yes, you can use R2 by setting the S3_ENDPOINT_URL variable. This enables compatibility with AWS S3 API drop-in replacements without altering your core application logic or storage workflows.
Skipping the export breaks data integrity during the provider switch. You must run the export command then initialize a fresh backend instance to prevent corruption from direct file manipulation.
You must set S3_STORAGE_FILES_BUCKET and S3_STORAGE_MODULES_BUCKET to direct traffic correctly. These variables ensure application assets and code modules route to their specific, separate durable object stores.
References
