Audio Orchestrator

What it is

Audio Orchestrator is a backend service designed to decouple audio file ingestion from processing. It exposes REST APIs for workers to register and claim tasks, persists task state in PostgreSQL, and uses AWS SQS to distribute work items containing presigned S3 URLs to downstream processing workers.

Features

• REST API for worker authentication and task claiming
• Integration with AWS SQS for distributed task queuing
• Generation of presigned S3 URLs for secure file access
• PostgreSQL-backed task state management with retry logic
• Hybrid configuration via AWS Secrets Manager and YAML

Quickstart

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m orchestrator.main

Architecture

flowchart TD
    Client[Worker/Client] -->|HTTP/Bearer Token| API[Flask API Routes]
    API --> Auth{Authenticate}
    Auth -->|Valid| TS[Task Service]
    Auth -->|Invalid| Err[401 Error]
    TS --> DB[(PostgreSQL)]
    TS --> QS[Queue Service]
    QS --> SQS[AWS SQS]
    QS --> S3[AWS S3]
    S3 -->|Presigned URLs| QS
    Config[Settings] -->|Secrets| ASM[AWS Secrets Manager]
    Config -->|YAML| Local[Local File]

How it's built

Built with Python using Flask for the API layer, psycopg2 for PostgreSQL interaction, and boto3 for AWS integration (SQS, S3, Secrets Manager). Configuration is hybrid, loading sensitive credentials from AWS Secrets Manager and non-sensitive settings from a local YAML file. It uses a singleton Settings class and dataclasses for task modeling.

How it runs

sequenceDiagram
    participant W as Worker
    participant A as API Routes
    participant TS as Task Service
    participant DB as Database
    participant QS as Queue Service
    participant S3 as AWS S3
    participant SQS as AWS SQS

    W->>A: POST /claim-task (Bearer Token)
    A->>A: Authenticate Token
    A->>TS: get_next_task()
    TS->>DB: Query pending tasks
    DB-->>TS: Return Task Object
    TS->>QS: send_task_to_queue(task_id, key)
    QS->>S3: generate_presigned_urls()
    S3-->>QS: Return Get/Put URLs
    QS->>SQS: send_message(Task Details + URLs)
    SQS-->>QS: Message ID
    QS-->>TS: Success
    TS-->>A: Task Assigned
    A-->>W: 200 OK (Task Data)

How to apply & reuse

Deploy this service in an AWS environment where it can access Secrets Manager, SQS, and S3. Use it as the central control plane for any audio processing pipeline (e.g., transcription, enhancement) where you need reliable task tracking, retry logic, and secure file access delegation via presigned URLs.

At a glance