Kintone Backup System - Architecture Diagrams

Tài liệu chứa tất cả các mermaid diagrams mô tả kiến trúc và flow của hệ thống backup Kintone.

1. System Architecture Overview

1.1. High-Level Architecture

graph TB
    subgraph "Frontend Layer"
        A[Admin Dashboard<br/>Inertia + React]
    end

    subgraph "Application Layer"
        B[Laravel Application]
        C[API Controllers]
        D[Job Dispatcher]
    end

    subgraph "Scheduling Layer"
        E[Laravel Scheduler<br/>Cron Jobs]
        F[Task Scheduler<br/>Database]
    end

    subgraph "Queue Layer"
        G[Laravel Queue<br/>Database]
        H[AWS SQS<br/>Optional]
    end

    subgraph "Worker Layer"
        I[Queue Workers<br/>Laravel Workers]
        J[Backup Jobs]
    end

    subgraph "External Services"
        K[Kintone API]
        L[AWS S3]
    end

    subgraph "Storage"
        M[PostgreSQL/MySQL<br/>App Config & Logs]
        N[Local Temp Storage<br/>/temp-YYYY-MM-DD]
    end

    A --> B
    B --> C
    C --> D
    D --> F
    E --> F
    F --> G
    G --> I
    I --> J
    J --> K
    J --> N
    J --> L
    B --> M
    J --> M

1.2. Multi-App Concurrent Backup Flow

graph TD
    A[Scheduler: 10:00 AM] --> B{Check Scheduled Apps}
    B --> C[App A: 10:00 AM]
    B --> D[App B: 10:00 AM]
    B --> E[App C: 12:00 AM]

    C --> F[Queue: BackupJob App A]
    D --> G[Queue: BackupJob App B]

    F --> H[Worker 1: Process App A]
    G --> I[Worker 2: Process App B]

    H --> J[Kintone API App A]
    I --> K[Kintone API App B]

    H --> L[Temp: /temp-2025-01-15/app-a/]
    I --> M[Temp: /temp-2025-01-15/app-b/]

    L --> N[S3: 2025-01-15/app-a/]
    M --> O[S3: 2025-01-15/app-b/]

    style C fill:#e1f5ff
    style D fill:#e1f5ff
    style E fill:#fff4e1
    style H fill:#d4edda
    style I fill:#d4edda

2. Backup Flow Diagrams

2.1. Complete Backup Flow

sequenceDiagram
    participant Admin
    participant Laravel
    participant Scheduler
    participant Queue
    participant Worker
    participant Kintone
    participant TempStorage
    participant S3

    Admin->>Laravel: Configure App & Schedule
    Laravel->>Scheduler: Save schedule (10:00 AM)

    Note over Scheduler: Every minute, check schedules
    Scheduler->>Scheduler: Check if time matches (10:00 AM)
    Scheduler->>Queue: Dispatch BackupJob for App A
    Scheduler->>Queue: Dispatch BackupJob for App B

    Queue->>Worker: Process BackupJob (App A)

    Note over Worker: Phase 1: Fetch Records & Extract Data
    Worker->>Kintone: Get Records (offset=0, limit=500)
    Kintone-->>Worker: Return 500 records

    Worker->>Worker: Extract data → Append to CSV buffer
    Worker->>Worker: Extract fileKeys from records
    Worker->>Database: Save fileKeys to backup_files table

    Worker->>Kintone: Get Records (offset=500, limit=500)
    Kintone-->>Worker: Return 500 records
    Note over Worker: Repeat until all records fetched

    Worker->>TempStorage: Save CSV to /temp-2025-01-15/app-123/data.csv

    Note over Worker: Phase 2: Download Files
    Worker->>Database: Query fileKeys from backup_files
    Database-->>Worker: Return all fileKeys

    loop For each fileKey
        Worker->>Kintone: Download File (fileKey)
        Kintone-->>Worker: Return file content
        Worker->>TempStorage: Save file to /temp-2025-01-15/app-123/
    end

    Note over Worker: Phase 3: Zip All Files
    Worker->>Worker: Zip CSV + all files → backup.zip
    Worker->>TempStorage: Save backup.zip to /temp-2025-01-15/app-123/

    Note over Worker: Phase 4: Upload to S3
    Worker->>S3: Upload backup.zip to s3://bucket/2025-01-15/app-123/backup.zip

    Worker->>TempStorage: Cleanup temp files (CSV + files + zip)
    Worker->>Laravel: Update backup status (completed)

    Laravel->>Admin: Notification (success/failure)

2.2. Failure Flow - Fetch Records với Retry Logic

sequenceDiagram
    participant Worker
    participant Kintone
    participant Database
    participant TempStorage

    Note over Worker: Fetch Records Page 0 (Success)
    Worker->>Kintone: GET /records.json?offset=0&limit=500
    Kintone-->>Worker: Return 500 records (Success)
    Worker->>Worker: Extract data → Append to CSV buffer
    Worker->>Worker: Extract fileKeys
    Worker->>Database: INSERT INTO backup_files (fileKeys, status='pending')
    Worker->>Database: UPDATE backup_jobs SET processed_records += 500

    Note over Worker: Fetch Records Page 1 (Failed)
    Worker->>Kintone: GET /records.json?offset=500&limit=500
    Kintone-->>Worker: HTTP 429 (Too Many Requests)

    Worker->>Database: Log error: "Failed page 1, attempt 1"
    Worker->>Worker: Wait 1 second (exponential backoff)

    Worker->>Kintone: Retry GET /records.json?offset=500&limit=500
    alt Retry Success
        Kintone-->>Worker: Return 500 records (Success)
        Worker->>Worker: Extract data → Append to CSV buffer
        Worker->>Worker: Extract fileKeys
        Worker->>Database: Log: "Retry successful, fetched page 1"
        Worker->>Database: INSERT INTO backup_files (fileKeys)
        Worker->>Database: UPDATE processed_records += 500
        Note over Worker: Continue with next page
    else Retry Failed (Attempt 2)
        Kintone-->>Worker: HTTP 429 (Still failing)
        Worker->>Database: Log error: "Failed page 1, attempt 2"
        Worker->>Worker: Wait 2 seconds
        Worker->>Kintone: Retry GET /records.json (Attempt 3)
        alt Final Retry Success
            Kintone-->>Worker: Return 500 records (Success)
            Worker->>Worker: Extract data → Append to CSV buffer
            Worker->>Worker: Extract fileKeys
            Worker->>Database: Log: "Retry successful after 3 attempts"
            Worker->>Database: INSERT INTO backup_files (fileKeys)
            Worker->>Database: UPDATE processed_records += 500
            Note over Worker: Continue with next page
        else Final Retry Failed
            Kintone-->>Worker: HTTP 429 (Still failing)
            Worker->>TempStorage: Save partial CSV (records đã fetch được)
            Worker->>Database: UPDATE backup_jobs SET status='failed'
            Worker->>Database: Log error: "Failed after 3 retries, job failed"
            Note over Worker: Stop processing, job marked as failed
        end
    end

2.3. Download Files & Zip Flow - Success & Failure

sequenceDiagram
    participant Worker
    participant Database
    participant Kintone
    participant TempStorage

    Note over Worker: Query pending files
    Worker->>Database: SELECT * FROM backup_files WHERE status='pending'
    Database-->>Worker: Return 1000 file records

    loop For each file
        Worker->>Kintone: GET /file.json?fileKey=abc123
        alt Download Success
            Kintone-->>Worker: Return file content
            Worker->>TempStorage: Save file to /temp-2025-01-15/app-123/
            Worker->>Database: UPDATE backup_files SET status='downloaded'
            Worker->>Database: UPDATE backup_jobs SET processed_files += 1
        else Download Failed
            Kintone-->>Worker: HTTP 404 (File not found)
            Worker->>Database: UPDATE backup_files SET status='failed', error_message='HTTP 404'
            Worker->>Database: Log warning: "File download failed"
            Worker->>Worker: Retry download (max 3 times)
            alt Retry Success
                Kintone-->>Worker: Return file content
                Worker->>TempStorage: Save file to /temp-2025-01-15/app-123/
                Worker->>Database: UPDATE backup_files SET status='downloaded'
            else Retry Failed
                Worker->>Database: UPDATE backup_files SET status='failed'
                Worker->>Database: Log warning: "File skipped after retries"
                Note over Worker: Continue with next file (không fail job)
            end
        end
    end

    Note over Worker: Zip All Files (CSV + Downloaded Files)
    Worker->>TempStorage: Read CSV file
    Worker->>TempStorage: Read all downloaded files
    Worker->>Worker: Zip CSV + all files → backup.zip
    Worker->>TempStorage: Save backup.zip to /temp-2025-01-15/app-123/
    Worker->>Database: Log: "Backup zip created successfully"

    alt Zip Success
        Worker->>Database: Log: "Zip completed, ready to upload"
    else Zip Failed
        Worker->>Database: Log error: "Zip failed"
        Worker->>Database: UPDATE backup_jobs SET status='failed'
        Note over Worker: Job failed, stop processing
    end

2.4. Upload to S3 Flow - Success & Failure

sequenceDiagram
    participant Worker
    participant TempStorage
    participant S3
    participant Database

    Note over Worker: Upload Zip File
    Worker->>TempStorage: Read backup.zip file
    Worker->>S3: Upload backup.zip to s3://bucket/2025-01-15/app-123/backup.zip
    alt Upload Success
        S3-->>Worker: Upload successful
        Worker->>Database: Log: "Backup zip uploaded successfully"
        Worker->>Database: UPDATE backup_jobs SET s3_path='s3://bucket/2025-01-15/app-123/backup.zip'
    else Upload Failed
        S3-->>Worker: Network timeout
        Worker->>Database: Log error: "Backup zip upload failed, retrying..."
        Worker->>S3: Retry upload (max 3 times)
        alt Retry Success
            S3-->>Worker: Upload successful
            Worker->>Database: Log: "Backup zip upload retry successful"
            Worker->>Database: UPDATE backup_jobs SET s3_path='s3://bucket/2025-01-15/app-123/backup.zip'
        else Retry Failed
            Worker->>Database: UPDATE backup_jobs SET status='failed'
            Worker->>Database: Log error: "Backup zip upload failed after retries"
            Note over Worker: Job failed, stop processing
        end
    end

    Worker->>TempStorage: Cleanup temp files
    Worker->>Database: UPDATE backup_jobs SET status='completed'
    Worker->>Database: Log: "Backup completed successfully"