Cloud Storage Integration

FlashFS provides integration with cloud storage services, allowing you to export snapshots to and restore them from cloud storage providers like Amazon S3, Google Cloud Storage (GCS), or any S3-compatible storage service (like MinIO).

Overview

The cloud storage integration enables:

  • Exporting snapshots to cloud storage for backup or archival purposes
  • Restoring snapshots from cloud storage to a local directory
  • Compression and decompression of snapshots during transfer
  • Support for multiple cloud storage providers

Supported Storage Providers

FlashFS currently supports the following cloud storage providers:

  • Amazon S3: Native support for Amazon S3 storage
  • Google Cloud Storage (GCS): Native support for Google Cloud Storage
  • S3-Compatible Storage: Support for MinIO, Ceph, and other S3-compatible storage services

Configuration

Cloud storage configuration is primarily done through environment variables:

Amazon S3 Configuration

# Required
export S3_ACCESS_KEY=your-access-key
export S3_SECRET_KEY=your-secret-key

# Optional
export S3_ENDPOINT=https://s3.amazonaws.com  # Custom endpoint for S3-compatible storage
export S3_REGION=us-east-1                   # AWS region (default: us-east-1)
export S3_INSECURE=false                     # Use insecure connection (default: false)
export S3_FORCE_PATH_STYLE=false             # Use path-style addressing (default: false)

Google Cloud Storage Configuration

# Required for service account authentication
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

Usage

Exporting Snapshots

The export command allows you to export snapshots to cloud storage:

flashfs export [destination] [flags]

Where destination is a URL in the format s3://bucket-name/prefix/ or gcs://bucket-name/prefix/.

Examples

Export a specific snapshot to S3:

flashfs export s3://mybucket/flashfs-backups/ --snapshot snapshot1.snap

Export all snapshots to GCS:

flashfs export gcs://my-bucket-name/ --all

Export to MinIO or other S3-compatible storage:

export S3_ENDPOINT=https://minio.example.com
export S3_ACCESS_KEY=your-access-key
export S3_SECRET_KEY=your-secret-key
export S3_FORCE_PATH_STYLE=true
flashfs export s3://mybucket/backups/ --all

Flags

  • --dir string: Directory containing snapshots (default "snapshots")
  • --snapshot string: Name of the snapshot to export
  • --all: Export all snapshots
  • --compress: Compress snapshots before uploading (default true)

Restoring Snapshots

The restore command allows you to restore snapshots from cloud storage:

flashfs restore [source] [flags]

Where source is a URL in the format s3://bucket-name/prefix/ or gcs://bucket-name/prefix/.

Examples

Restore a specific snapshot from S3:

flashfs restore s3://mybucket/flashfs-backups/ --snapshot snapshot1.snap

Restore all snapshots from GCS:

flashfs restore gcs://my-bucket-name/ --all

Flags

  • --dir string: Directory to restore snapshots to (default "snapshots")
  • --snapshot string: Name of the snapshot to restore
  • --all: Restore all snapshots
  • --decompress: Decompress snapshots after downloading (default true)

Implementation Details

The cloud storage integration is implemented using the Thanos Object Storage library, which provides a unified interface for interacting with different cloud storage providers.

Compression

Snapshots are compressed using the Zstandard compression algorithm before being uploaded to cloud storage. This reduces the amount of data transferred and the storage space required.

Error Handling

The cloud storage integration includes robust error handling for common issues:

  • Authentication errors (missing or invalid credentials)
  • Network connectivity issues
  • Permission issues
  • Storage quota issues

Performance Considerations

  • Streaming Uploads/Downloads: Files are streamed to/from cloud storage to minimize memory usage
  • Compression: Zstandard compression provides a good balance between compression ratio and speed
  • Parallel Processing: Multiple snapshots can be exported/restored in parallel

Troubleshooting

Common Issues

  1. Authentication Errors:
  2. Ensure that the required environment variables are set correctly

  3. Check that the credentials have the necessary permissions

  4. Network Issues:

  5. Check your network connectivity

  6. Verify that the endpoint URL is correct

  7. Permission Issues:

  8. Ensure that the credentials have the necessary permissions to read/write to the bucket

  9. Storage Quota Issues:

  10. Check that you have sufficient storage quota in your cloud storage account

Logging

The cloud storage integration includes detailed logging to help diagnose issues. You can view the logs by running the commands with the --verbose flag.