Object Storage Management

In modern SaaS applications, file storage is an essential feature. Users need to upload avatars, documents, images and other files, and applications need to manage these resources securely and efficiently. NextDevKit provides a complete object storage solution.

🤔 Why Object Storage?

Problems with Traditional File Storage

Limitations of local file storage:

• Poor scalability: Limited server disk space, difficult to handle large volumes of files
• Performance bottlenecks: Large file transfers consume server bandwidth and computing resources
• Security risks: Files directly exposed on servers, vulnerable to malicious access
• High maintenance costs: Need to handle complex issues like file backup and disaster recovery

Advantages of Object Storage

Object Storage solves these problems:

✅ Unlimited scaling: Theoretically unlimited storage space
✅ High availability: Automatic backup and disaster recovery mechanisms
✅ Global distribution: CDN acceleration for fast access
✅ Security control: Fine-grained access permission control
✅ Cost optimization: Pay-as-you-use, no pre-provisioned resources needed

🏗️ NextDevKit Storage Architecture

NextDevKit adopts an abstracted storage architecture supporting multiple storage service providers:

Architecture Design Principles

1. Provider Abstraction: Unified interface supporting multiple storage services
2. Pre-signed URLs: Direct client uploads reducing server load
3. Image Proxy: Secure image access and caching
4. Type Safety: Complete TypeScript support

Directory Structure

Core Module Functions:

• actions.ts: Server Actions handling file upload requests
• types.ts: Storage-related TypeScript type definitions
• providers/: Storage service provider abstraction layer
• image-proxy/: Image proxy routes for secure access to stored images

Supported Storage Services

NextDevKit supports by default:

• AWS S3: Most popular object storage service
• Cloudflare R2: S3-compatible, lower cost
• Extensible: Based on unified interface, easily add other service providers

🔐 Pre-signed URL Core Concepts

What are Pre-signed URLs?

Pre-signed URLs are temporary, signed URLs that allow clients to directly access storage services without exposing storage credentials.

Pre-signed URL Workflow:

1. Client Request: Client requests upload permission from NextDevKit server
2. Generate Signature: Server calls storage service API to generate pre-signed URL
3. Return URL: Server returns temporary URL to client
4. Direct Upload: Client uses pre-signed URL to upload file directly to storage service
5. Upload Complete: Storage service confirms successful upload

Advantages of Pre-signed URLs

1. Security:

• Temporary validity with automatic expiration
• No exposure of storage credentials
• Limited operation permissions

2. Performance:

• Client direct upload without going through application server
• Reduced server bandwidth consumption
• Improved upload speed

3. Control:

• Flexible expiration time settings
• File type and size restrictions
• Specified storage location

⚙️ Environment Configuration

NextDevKit supports multiple storage services. Choose the appropriate configuration based on your deployment platform.

For specific S3 and R2 configuration, please refer to NextDevKit Storage documentation.

NextDevKit Application Configuration

Regardless of which storage service you choose, you need to configure in the application:

export const appConfig = {
  storage: {
    provider: "s3",  // S3 compatible protocol
    bucketNames: {
      avatars: process.env.NEXT_PUBLIC_AVATARS_BUCKET_NAME || "avatars",
    },
  },
} as const;

Storage Selection for Different Deployment Templates

Cloudflare Workers Template:

• Use Cloudflare R2
• Reason: Same ecosystem, lower latency, better cost

SST AWS Template:

• Use AWS S3
• Reason: Native AWS integration, IAM permission management

Next.js Standard Deployment (Vercel/Docker):

• Choose Cloudflare R2 or AWS S3
• Decision based on cost and geographic location needs

AWS S3 Setup

Environment Variables

STORAGE_REGION=your_region # e.g. us-east-1
STORAGE_ACCESS_KEY_ID=your_access_key_id
STORAGE_SECRET_ACCESS_KEY=your_secret_access_key

# Bucket Names (optional, can be configured in app config)
NEXT_PUBLIC_AVATARS_BUCKET_NAME=your-bucket-name

AWS S3 Bucket Setup

1. Create AWS Account: Sign up at aws.amazon.com

2. Create S3 Bucket:

   • Go to S3 Object Storage
   • Create a new bucket
   • Disable "Block all public access" if needed

3. Configure Bucket Policy:

   • Go to Bucket Permissions
   • Add the following policy:

   {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
      ]
    }

4. Configure CORS:

   • Go to Bucket CORS
   • Add the following CORS configuration:

   [
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "PUT",
            "GET",
            "HEAD",
            "POST"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
   ] 

5. Generate API Token:

   • Create API token with S3 permissions
   • Use as access key credentials

Cloudflare R2 Setup

Environment Variables

STORAGE_REGION=auto
STORAGE_ACCESS_KEY_ID=your_access_key_id
STORAGE_SECRET_ACCESS_KEY=your_secret_access_key
STORAGE_ENDPOINT=https://your-account-id.r2.cloudflarestorage.com

# Bucket Names (optional, can be configured in app config)
NEXT_PUBLIC_AVATARS_BUCKET_NAME=your-bucket-name

Cloudflare R2 Bucket Setup

1. Create Cloudflare Account: Sign up at cloudflare.com

2. Create R2 Bucket:

   • Go to R2 Object Storage
   • Create a new bucket
   • Set the custom domain to your bucket

3. Configure CORS:

   • Go to Bucket CORS
   • Add the following CORS configuration:

   [
      {
        "AllowedOrigins": [
          "*"
        ],
        "AllowedMethods": [
          "PUT",
          "GET",
          "HEAD"
        ],
        "AllowedHeaders": [
          "Content-Type" // must be set
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
      }
    ]

4. Create a new API Token:

   • Go to R2/API/Manage API Tokens
   • Click Create User API Token
   • Set permissions to Object Read & Write to the bucket
   • Create the API Token, get the Access Key ID and Secret Access Key

📤 File Upload Functionality Explained

Server Action: Get Upload URL

NextDevKit provides the getSignedUploadUrl Server Action:

export const getSignedUploadUrl = actionClient
  .inputSchema(z.object({
    bucket: z.string().min(1),
    key: z.string().min(1),
    contentType: z.string().min(1),
  }))
  .outputSchema(z.string())
  .action(async ({ parsedInput: { bucket, key, contentType } }) => {
    const storageProvider = getStorageProvider();
    return await storageProvider.getSignedUploadUrl({
      bucket,
      key,
      contentType,
    });
  });

Client-side File Upload Implementation

"use client";

import { getSignedUploadUrl } from "@/storage/actions";
import { appConfig } from "@/config";
import { useState } from "react";

export default function AvatarUpload() {
  const [uploading, setUploading] = useState(false);

  const handleFileUpload = async (file: File) => {
    setUploading(true);
    try {
      // 1. Generate unique file name
      const fileExtension = file.name.split('.').pop();
      const fileName = `user-${Date.now()}.${fileExtension}`;

      // 2. Get pre-signed upload URL
      const signedUrl = await getSignedUploadUrl({
        bucket: appConfig.storage.bucketNames.avatars,
        key: fileName,
        contentType: file.type,
      });

      // 3. Direct upload to storage service
      const uploadResponse = await fetch(signedUrl, {
        method: "PUT",
        body: file,
        headers: {
          "Content-Type": file.type,
        },
      });

      if (uploadResponse.ok) {
        console.log("File upload successful!", fileName);
        // 4. Update user information or interface
        await updateUserAvatar(fileName);
      }
    } catch (error) {
      console.error("Upload failed:", error);
    } finally {
      setUploading(false);
    }
  };

  return (
    <div className="space-y-4">
      <input
        type="file"
        accept="image/*"
        onChange={(e) => {
          const file = e.target.files?.[0];
          if (file) handleFileUpload(file);
        }}
        disabled={uploading}
        className="file:mr-4 file:py-2 file:px-4 file:rounded-md file:border-0"
      />
      {uploading && <p>Uploading...</p>}
    </div>
  );
}

Upload Process Analysis

Complete file upload process:

1. Frontend file selection: User selects file through input
2. Request upload permission: Call getSignedUploadUrl Server Action
3. Generate pre-signed URL: Server validates permissions and generates temporary URL
4. Direct upload: Frontend uses fetch to upload file to storage service
5. Update records: After successful upload, update database or user interface

Key advantages:

• Files don't go through NextJS server, saving bandwidth
• Upload failures don't affect application performance
• Supports large file uploads

🖼️ Image Proxy Functionality

What is Image Proxy?

Image Proxy is a proxy service that provides secure access and caching optimization for stored images.

NextDevKit Image Proxy Implementation

export const GET = async (
  _req: Request,
  { params }: { params: Promise<{ path: string[] }> }
) => {
  const { path } = await params;
  const [bucket, filePath] = path;

  // Verify bucket permissions
  if (bucket === appConfig.storage.bucketNames.avatars) {
    // Generate access URL
    const signedUrl = await getStorageProvider().getSignedUrl({
      bucket,
      key: filePath,
      expiresIn: 60 * 60, // 1 hour
    });

    // Redirect to signed URL with cache headers
    return NextResponse.redirect(signedUrl, {
      headers: { "Cache-Control": "max-age=3600" },
    });
  }

  return new Response("Not found", { status: 404 });
};

Using Image Proxy

import Image from "next/image";

export default function UserAvatar({ fileName }: { fileName: string }) {
  // Access images through Image Proxy
  const imageUrl = `/image-proxy/${appConfig.storage.bucketNames.avatars}/${fileName}`;

  return (
    <Image
      src={imageUrl}
      alt="User avatar"
      width={64}
      height={64}
      className="rounded-full"
    />
  );
}

Core Advantages of Image Proxy

1. Security:

• Hide real storage URLs
• Centralized access control
• Prevent direct access to storage services

2. Performance optimization:

• HTTP cache header optimization
• Reduce duplicate signature generation
• Support CDN caching

3. Flexibility:

• Can add image processing functionality
• Support differentiated handling for different buckets
• Easy monitoring and logging

🔧 Storage System Architecture Details

Provider Abstraction Layer

NextDevKit's storage system is based on the Provider pattern:

export interface StorageProvider {
  // Get file access URL
  getSignedUrl(params: SignedUrlParams): Promise<string>;
  
  // Get file upload URL
  getSignedUploadUrl(params: SignedUploadUrlParams): Promise<string>;
}

export interface SignedUrlParams {
  bucket: string;
  key: string;
  expiresIn: number;
}

export interface SignedUploadUrlParams {
  bucket: string;
  key: string;
  contentType: string;
}

📚 Summary

NextDevKit's object storage system provides:

💡 Key Technical Points

Pre-signed URL mechanism:

• Client direct upload reducing server load
• Temporary permissions improving security
• Support for large file uploads

Image Proxy functionality:

• Hide real storage URLs
• Cache optimization for better performance
• Centralized access control

Provider abstraction:

• Unified interface, easy to extend
• Support for S3, Cloudflare R2 and other services
• Flexible configuration through environment variables

Now you can build complete file storage functionality in NextDevKit projects, providing users with secure and efficient file management experience!

Reference Resources:

• Cloudflare R2 Documentation
• AWS S3 Documentation
• Pre-signed URL Best Practices