> ## Documentation Index
> Fetch the complete documentation index at: https://docs.neurometric.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Background Jobs

> How background jobs work in Inference Studio.

# Background Jobs with Graphile Worker

The Inference Studio uses **Graphile Worker** for background job processing. Jobs are stored in PostgreSQL and processed asynchronously.

## Running the Worker

**In Docker Compose (Recommended):**
The worker is automatically started in a separate container when you run `docker compose up`.

```bash theme={null}
# View worker logs
docker compose logs -f worker

# Restart worker
docker compose restart worker
```

**Locally (for development):**

```bash theme={null}
cd inference-studio
npm run worker        # Run worker once
npm run worker:dev    # Run worker with auto-reload
```

## Available Jobs

The worker processes the following background tasks:

### Experiment Jobs

* **`process-upload`** - Processes uploaded CSV/XLSX experiments: uploads data to S3, submits Ray job, sets status → `PROCESSING`
* **`submit-review`** - Processes API experiments after user review: uploads data to S3, submits Ray job, sets status → `PROCESSING`
* **`rerun-analysis`** - Re-runs Ray job for an existing experiment
* **`send-email`** - Sends analysis-complete notification email via Resend

### Langfuse Trace Sync (Periodic)

* **`sync-traces-coordinator`** - Coordinates periodic syncing for all active API keys, fans out to `sync-fetch-traces`
* **`sync-fetch-traces`** - Fetches raw traces from Langfuse for one API key
* **`sync-enrich-observations`** - Enriches observations with cost/latency metrics
* **`sync-categorize-traces`** - Categorizes traces and assigns tasks

## Admin UI

You can view and manage jobs at `/admin/jobs`:

* Monitor job status (pending, active, completed, failed)
* Retry failed jobs
* View job payloads and errors
* Delete jobs

## Environment Variables

Worker-specific configuration:

* `GRAPHILE_WORKER_CONCURRENCY` - Number of jobs to run in parallel (default: 5)
* `GRAPHILE_WORKER_POLL_INTERVAL` - How often to poll for new jobs in ms (default: 1000)