4kUpScalerWorker

What it is

A Python-based background worker that listens to an AWS SQS queue for S3 upload events. Upon receiving a notification, it downloads the specified image from S3, prepares it for processing (intended for 4K upscaling via InvokeAI), and manages the file lifecycle between storage and local processing directories.

Features

• Automated ingestion of images from AWS S3 buckets
• Event-driven processing via AWS SQS queues
• Local file staging for GPU-accelerated upscaling
• Batch upload utility for triggering multiple jobs
• Error handling for S3 upload failures

Quickstart

git clone https://github.com/davidbmar/4kUpScalerWorker.git
cd 4kUpScalerWorker
mkdir -p /root/.aws
# Copy your AWS config and credentials into /root/.aws/
python3 UpscalerWorker.py

Architecture

flowchart TD
    User[User] -->|Uploads Image| S3[S3 Bucket]
    S3 -->|Put Event| SNS[S3 Event Notification]
    SNS -->|Send Message| SQS[SQS Queue]
    Worker[UpscalerWorker.py] -->|Poll/Receive| SQS
    Worker -->|Download Object| S3
    Worker -->|Process/Upscale| LocalFS[Local File System]
    LocalFS -->|Upload Result| S3Processed[S3 Processed Bucket/Folder]

How it's built

Built with Python 3 and boto3 for AWS interactions. It relies on the InvokeAI environment for the actual upscaling logic. The architecture uses S3 for storage, S3 Event Notifications to trigger SQS messages, and a long-polling or periodic polling script (UpscalerWorker.py) to process jobs.

How it runs

sequenceDiagram
    participant U as User
    participant S3 as S3 Bucket
    participant SQS as SQS Queue
    participant W as UpscalerWorker
    
    U->>S3: PUT object (image.jpg)
    S3->>SQS: Send event notification
    loop Polling
        W->>SQS: ReceiveMessage
        SQS-->>W: Return message (if available)
    end
    W->>S3: GetObject (bucket, key)
    S3-->>W: Return file content
    W->>W: Save to local disk
    W->>W: Execute Upscaling (InvokeAI)
    W->>S3: Upload processed file (via s3_utils)

How to apply & reuse

Deploy this worker on a GPU-enabled instance (e.g., RunPod) with the InvokeAI image. Configure AWS credentials to allow reading from the source S3 bucket and writing to the destination. Use UploadToS3_forProcessing.py to batch-upload images to trigger the workflow.

At a glance