Background Jobs with Graphile Worker

The Inference Studio uses Graphile Worker for background job processing. Jobs are stored in PostgreSQL and processed asynchronously.

Running the Worker

In Docker Compose (Recommended): The worker is automatically started in a separate container when you run docker compose up.

# View worker logs
docker compose logs -f worker

# Restart worker
docker compose restart worker

Locally (for development):

cd inference-studio
npm run worker        # Run worker once
npm run worker:dev    # Run worker with auto-reload

The worker processes the following background tasks:

process-upload - Processes uploaded CSV/XLSX experiments: uploads data to S3, submits Ray job, sets status → PROCESSING
submit-review - Processes API experiments after user review: uploads data to S3, submits Ray job, sets status → PROCESSING
rerun-analysis - Re-runs Ray job for an existing experiment
send-email - Sends analysis-complete notification email via Resend

sync-traces-coordinator - Coordinates periodic syncing for all active API keys, fans out to sync-fetch-traces
sync-fetch-traces - Fetches raw traces from Langfuse for one API key
sync-enrich-observations - Enriches observations with cost/latency metrics
sync-categorize-traces - Categorizes traces and assigns tasks

You can view and manage jobs at /admin/jobs:

Worker-specific configuration:

GRAPHILE_WORKER_CONCURRENCY - Number of jobs to run in parallel (default: 5)
GRAPHILE_WORKER_POLL_INTERVAL - How often to poll for new jobs in ms (default: 1000)