Backend API Reference

The Kayra backend exposes a REST API on port 8000 by default. Interactive docs are available at /docs (Swagger UI) and /redoc (ReDoc) when DEBUG=true.

Route Map

Prefix	Area
`/health*`	Health & readiness probes
`/data/api/*`	Data source management
`/api/ingestion/*`	File upload & ingestion jobs
`/api/pipeline/*`	ML pipeline execution, preview, jobs, versions
`/api/eda/*`	Automated EDA & profiling
`/api/deployment/*`	Model deployment & inference
`/api/registry/*`	Model registry
`/api/monitoring/*`	Drift monitoring, error tracking, pipeline logs
`/ws/jobs`	WebSocket job event stream

Health

`GET /health/ping`

Minimal liveness probe — returns { "message": "pong" }. Fastest possible health check with no database involvement.

`GET /health`

Basic health probe for load balancers. Returns HTTP 200 when the server is running.

{
  "status": "healthy",
  "timestamp": "2025-06-30T12:00:00Z",
  "version": "0.6.3",
  "environment": "development",
  "uptime_seconds": 3600.0
}

`GET /health/detailed`

Detailed diagnostics. Checks database connectivity and Redis (if USE_CELERY=true).

{
  "status": "healthy",
  "database_status": "healthy",
  "cache_status": "healthy",
  "external_services": { "snowflake": "not_configured" }
}

`GET /health/ready`

Readiness probe — fits a tiny sklearn pipeline to verify the ML stack is functional. Returns 503 if the ML dependencies are broken.

Data Sources

`GET /data/api/sources`

List all data sources. Query params: limit (default 50), skip (default 0).

`GET /data/api/sources/usable`

List data sources that have completed ingestion and are ready for pipeline use.

`GET /data/api/sources/{source_id}`

Get a single data source by ID.

Note: The config field in all data source responses has sensitive keys (storage_options, AWS credentials, passwords) automatically redacted.

`GET /data/api/sources/{source_id}/sample`

Get a sample of rows from the data source.

`DELETE /data/api/sources/{source_id}`

Delete a data source and its associated files.

`GET /data/api/sources/{source_id}/export`

Export the full dataset as a downloadable file.

File Ingestion

`POST /api/ingestion/upload` — Rate limited: 10/min

Upload a dataset file. Starts an asynchronous ingestion job.

Request: multipart/form-data with a file field.

Response:

{ "job_id": "...", "status": "pending", "message": "File upload started", "file_id": "..." }

Allowed extensions: .csv, .xlsx, .xls, .parquet, .json, .txt, .pkl, .pickle, .feather, .h5, .hdf5

Size limit: MAX_UPLOAD_SIZE (default 10 GB).

`GET /api/ingestion/{source_id}/status`

Poll the status of an ingestion job. Returns pending, processing, completed, or failed.

`POST /api/ingestion/{source_id}/cancel`

Cancel an in-progress ingestion job.

ML Pipeline

All pipeline endpoints are under /api/pipeline/*.

`POST /api/pipeline/run` — Rate limited: 20/min

Submit a pipeline for asynchronous execution. Accepts a PipelineConfigModel JSON body describing the node graph.

Response:

{
  "message": "Pipeline execution started",
  "pipeline_id": "...",
  "job_id": "...",
  "job_ids": ["..."]
}

Multiple job_ids are returned when the pipeline is split into parallel branches. Concurrent submissions for the same dataset, terminal node and branch reuse a recent queued/running job within JOB_IDEMPOTENCY_WINDOW_SECONDS; the duplicate does not dispatch another task. Reservation covers the duplicate lookup, version allocation and committed insertion across API processes. SQLite serializes writers during this short transaction; PostgreSQL uses transaction advisory locks with the configured default READ COMMITTED isolation. Unsupported job types return HTTP 400. Completed jobs and requests outside the window can create new versions.

`POST /api/pipeline/preview`

Run the pipeline in preview mode (synchronous, limited to 1000 rows). Returns per-node output samples.

`POST /api/pipeline/schema-preview`

Returns predicted output column schema for a node without executing the pipeline.

`GET /api/pipeline/jobs`

List all training jobs with status and metrics.

`GET /api/pipeline/jobs/{job_id}`

Get detailed status, metrics, and logs for a specific job.

`GET /api/pipeline/jobs/{job_id}/trials`

Get the process-local live chart history for a running job: trials, metric, iterations, and iteration_metric. This backfills points missed before connecting to the WebSocket. Buffers are bounded and may be evicted.

Execution releases both buffers when it finishes, fails, or exits after cancellation. A successful cancellation also clears the local buffers after the database commit; any later points from a fitting thread are cleared when that execution exits. Completed-job charts use the saved metrics.trials and metrics.iterations returned by the job-details endpoint, so live-buffer cleanup does not remove their stored history.

`POST /api/pipeline/jobs/{job_id}/cancel`

Cancel a running or queued job.

Threshold tuning

POST /api/pipeline/jobs/{job_id}/thresholds/preview computes thresholds; POST /api/pipeline/jobs/{job_id}/thresholds/save saves and enables them. Supported metric values are accuracy, f1, precision, recall, and balanced_accuracy. Both endpoints reject roc_auc with HTTP 400 because probability-ranking AUC does not depend on the decision threshold.

Existing saved sets, including legacy roc_auc sets, can still be read via GET /api/pipeline/jobs/{job_id}/thresholds, enabled or disabled via POST .../thresholds/toggle, and removed via DELETE .../thresholds. See Threshold Tuning for the workflow.

`POST /api/pipeline/jobs/{job_id}/promote`

Mark a completed job as the "promoted winner" — sets the promoted_at timestamp.

`DELETE /api/pipeline/jobs/{job_id}/promote`

Remove the promoted status from a job (unpromote).

`GET /api/pipeline/jobs/{job_id}/evaluation`

Get full evaluation results for a completed job — model metrics, preprocessing artifacts, and tuning trial history (if applicable).

`GET /api/pipeline/jobs/node-summaries`

Returns per-node aggregated metrics across all jobs. Useful for comparing model types. Query param: limit (default 200).

Tuning Jobs

Method	Path	Description
`GET`	`/api/pipeline/jobs/tuning/latest/{node_id}`	Most recent tuning job for a canvas node
`GET`	`/api/pipeline/jobs/tuning/best/{model_type}`	Best-scoring tuning job for a model type
`GET`	`/api/pipeline/jobs/tuning/history/{model_type}`	Full tuning history for a model type

`GET /api/pipeline/registry`

Returns the full node registry — all available step types with their parameter schemas.

`GET /api/pipeline/stats`

Aggregate statistics across all jobs (counts by status, model type, etc.).

`GET /api/pipeline/datasets/list`

List datasets available for pipeline use.

`GET /api/pipeline/datasets/{dataset_id}/schema`

Get the column schema of a dataset.

`GET /api/pipeline/hyperparameters/{model_type}`

Get hyperparameter definitions for a model type.

`GET /api/pipeline/hyperparameters/{model_type}/defaults`

Get default hyperparameter values for a model type.

Pipeline Save / Load

Method	Path	Description
`POST`	`/api/pipeline/save/{dataset_id}`	Save a pipeline configuration for a dataset
`GET`	`/api/pipeline/load/{dataset_id}`	Load the current pipeline configuration for a dataset

Pipeline Versions

Method	Path	Description
`GET`	`/api/pipeline/versions/{dataset_source_id}`	List all saved versions for a dataset
`POST`	`/api/pipeline/versions/{dataset_source_id}`	Save a new named version
`PATCH`	`/api/pipeline/versions/{dataset_source_id}/{version_id}`	Update version metadata
`DELETE`	`/api/pipeline/versions/{dataset_source_id}/{version_id}`	Delete a version
`GET`	`/api/pipeline/versions/{dataset_source_id}/audit`	Version change audit log

Notebook Export

`POST /api/pipeline/pipeline/{dataset_id}/export-notebook`

Generate and download a Jupyter notebook (.ipynb) for a pipeline. Supports full and compact export modes. See Platform Walkthrough for details.

Deployment & Inference

`POST /api/deployment/deploy/{job_id}` — Rate limited: 10/min

Deploy a trained model from a completed job. Sets it as the active deployment.

`GET /api/deployment/active`

Get the currently active deployment.

`GET /api/deployment/history`

List deployment history. Query params: limit (default 50), skip (default 0).

`POST /api/deployment/deactivate`

Deactivate the current active deployment.

`POST /api/deployment/predict` — Rate limited: 60/min

Run inference using the active model.

Request:

{ "data": [{ "feature1": 1.5, "feature2": "value" }] }

data is an array of records — even for a single prediction, wrap it in a list.

Response:

{ "predictions": [...], "model_version": "job-id-here" }

Returns 404 if no deployment is active. Returns 400 if preprocessing changes the number of input rows, or a built-in LagFeatures/RollingAggregate step changes their order. The error names the stage; no partial prediction list is returned. Filter or sort inputs explicitly while retaining their mapping, or use row-preserving preprocessing such as Winsorize. External transformers and predictors are also checked for matching row counts.

Model Registry

The model registry is read-only — models are auto-registered when jobs complete and promoted via the jobs API.

`GET /api/registry/stats`

Aggregate registry statistics (model counts by type, total versions, etc.).

`GET /api/registry/models`

List all registered models.

`GET /api/registry/models/{model_type}/versions`

List all versions for a specific model type (e.g., random_forest_classifier).

`GET /api/registry/artifacts/{job_id}`

List all stored artifacts (model files, preprocessing pickles) for a job.

EDA & Profiling

`POST /api/eda/{dataset_id}/analyze` — Rate limited: 20/min

Trigger a full automated EDA analysis for a dataset. Returns a report_id immediately; the analysis runs asynchronously.

Response:

{ "report_id": 42, "status": "pending", "dataset_id": 7 }

`GET /api/eda/{dataset_id}/latest`

Get the most recently completed EDA report for a dataset (distributions, correlations, outliers, smart alerts).

`GET /api/eda/{dataset_id}/history`

List all past EDA reports for a dataset.

`GET /api/eda/reports/{report_id}`

Get a specific EDA report by ID.

`POST /api/eda/reports/{report_id}/cancel`

Cancel a running EDA analysis.

`POST /api/eda/{dataset_id}/decomposition`

Aggregate a measure for one level of the EDA decomposition tree, optionally grouped by split_col and restricted by filters.

Grouped response rows include name (display label), filter_value (the category identity), value and ratio. Use filter_value when drilling into a row: missing values have JSON null, while a real category named Unknown has the string "Unknown". For example, the following filter selects missing values in group:

{"column": "group", "operator": "==", "value": null}

Input columns can be named value, ratio, name or filter_value; those names do not conflict with response fields. A numeric column can also be used as both the grouping column and the measure.

Use "!=" with null to select non-missing rows. The UI labels the missing bucket Unknown (missing) and keeps literal Unknown separate. Older response rows without filter_value fall back to their display name; the legacy string "Unknown" still selects missing values on numeric columns. Reload or reopen the tree to obtain updated group identities; no stored-data migration is needed.

`GET /api/eda/jobs/all`

List all EDA jobs with status. Query params: limit (default 50), skip (default 0).

Monitoring

Drift Detection

`GET /api/monitoring/jobs`

List jobs that have reference data available for drift comparison.

`POST /api/monitoring/drift/calculate` — Rate limited: 20/min

Calculate data drift between a job's reference data and a new uploaded file.

Request: multipart/form-data - job_id — the training job whose reference data to compare against - file — current data file (CSV or Parquet) - dataset_name (optional) — dataset name to match reference data - threshold_psi, threshold_ks, threshold_wasserstein, threshold_kl (optional) — custom thresholds

Size limit: MAX_UPLOAD_SIZE (default 10 GB).

`GET /api/monitoring/drift/history/{job_id}`

List past drift check results for a job.

`GET /api/monitoring/drift/status`

Drift health summary across all monitored jobs.

Performance

`GET /api/monitoring/slow-nodes`

Returns the top N step types by cumulative execution time over the last N days. Query params: days (1–90, default 7), limit (1–50, default 10).

Error Tracking

The backend automatically records unhandled 5xx errors to the error_events database table.

`GET /api/monitoring/errors`

List recorded error events. Supports since, limit, show_resolved (default false) query params.

`GET /api/monitoring/errors/count`

Count of unresolved errors.

`GET /api/monitoring/errors/grouped`

Errors grouped by error_type.

`GET /api/monitoring/errors/timeline`

Error counts over time (for trend charts).

`GET /api/monitoring/errors/{error_id}`

Get a specific error event (includes full traceback — admin-only in production).

`PATCH /api/monitoring/errors/{error_id}/resolve`

Mark an error as resolved.

`PATCH /api/monitoring/errors/{error_id}/unresolve`

Reopen a resolved error.

`DELETE /api/monitoring/errors`

Delete all error events. ⚠️ Irreversible.

Pipeline Run Logs

`POST /api/monitoring/pipeline-logs`

Ingest pipeline execution log entries.

`GET /api/monitoring/pipeline-logs`

List pipeline run logs. Supports pipeline_id, since, limit (max 500) query params.

`PATCH /api/monitoring/jobs/{job_id}/description`

Update the description/notes for a monitored job.

`DELETE /api/monitoring/pipeline-logs`

Delete all pipeline run logs. ⚠️ Irreversible.

WebSocket — Job Events

`WS /ws/jobs`

A WebSocket stream that pushes real-time job status events to connected clients.

Connection: ws://127.0.0.1:8000/ws/jobs

Message envelope:

{ "channel": "jobs", "data": { "event": "progress", "job_id": "abc123", "status": "running", "progress": 42 } }

Messages are wrapped in a channel/data envelope. The data object is a JobEvent.

Event types (data.event):

`event`	When
`created`	A new job was submitted
`progress`	Job progress updated (0–100, in `data.progress`)
`status`	Job status changed (running → completed, etc.)
`deleted`	Job was deleted

JavaScript example:

const ws = new WebSocket("ws://127.0.0.1:8000/ws/jobs");
ws.onmessage = (msg) => {
  const { channel, data } = JSON.parse(msg.data);
  if (channel === "jobs") {
    console.log(`Job ${data.job_id}: ${data.event} (${data.status})`);
  }
};

Important: WebSocket events are best-effort hints. Maintain a polling fallback (GET /api/pipeline/jobs/{job_id}) for critical status checks.

Rate Limiting

Undecorated application routes use RATE_LIMIT_DEFAULT (200/minute by default), counted by client IP and URL path. Requests beyond the budget return HTTP 429 before the endpoint runs. The current in-memory counters are local to each API process; this is not a shared quota across workers.

Routes with explicit limits use their own budget instead of the default:

Endpoint	Limit
`POST /api/pipeline/run`	20/minute
`POST /api/eda/{dataset_id}/analyze`	20/minute
`POST /api/ingestion/upload`	10/minute
`POST /api/monitoring/drift/calculate`	20/minute
`POST /api/deployment/deploy/{job_id}`	10/minute
`POST /api/deployment/predict`	60/minute

Requests over the limit receive HTTP 429 Too Many Requests. Explicit exemptions remain exempt. CORS preflight, mounted static files and WebSocket connections do not consume an application route's default budget. Streaming response bodies and the existing security/CORS headers are preserved.

Error Responses

Error responses share a common shape, though some fields may be absent depending on the handler:

{
  "success": false,
  "error": "HTTP 404",
  "message": "The requested resource was not found",
  "request_id": "550e8400-e29b-41d4-a716-446655440000"
}

The request_id is also returned in the X-Request-ID response header for log correlation.

Common status codes:

Code	Meaning
`400`	Bad request (invalid parameters or business logic error)
`404`	Resource not found
`405`	Method not allowed
`413`	File too large
`422`	Validation error (malformed request body)
`429`	Rate limit exceeded
`500`	Internal server error

Backend API Reference

Route Map

Health

GET /health/ping

GET /health

GET /health/detailed

GET /health/ready

Data Sources

GET /data/api/sources

GET /data/api/sources/usable

GET /data/api/sources/{source_id}

GET /data/api/sources/{source_id}/sample

DELETE /data/api/sources/{source_id}

GET /data/api/sources/{source_id}/export

File Ingestion

POST /api/ingestion/upload — Rate limited: 10/min

GET /api/ingestion/{source_id}/status

POST /api/ingestion/{source_id}/cancel

ML Pipeline

POST /api/pipeline/run — Rate limited: 20/min

POST /api/pipeline/preview

POST /api/pipeline/schema-preview

GET /api/pipeline/jobs

GET /api/pipeline/jobs/{job_id}

GET /api/pipeline/jobs/{job_id}/trials

POST /api/pipeline/jobs/{job_id}/cancel

Threshold tuning

POST /api/pipeline/jobs/{job_id}/promote

DELETE /api/pipeline/jobs/{job_id}/promote

GET /api/pipeline/jobs/{job_id}/evaluation

GET /api/pipeline/jobs/node-summaries

Tuning Jobs

GET /api/pipeline/registry

GET /api/pipeline/stats

GET /api/pipeline/datasets/list

GET /api/pipeline/datasets/{dataset_id}/schema

GET /api/pipeline/hyperparameters/{model_type}

GET /api/pipeline/hyperparameters/{model_type}/defaults

Pipeline Save / Load

Pipeline Versions

Notebook Export

POST /api/pipeline/pipeline/{dataset_id}/export-notebook

Deployment & Inference

POST /api/deployment/deploy/{job_id} — Rate limited: 10/min

GET /api/deployment/active

GET /api/deployment/history

POST /api/deployment/deactivate

POST /api/deployment/predict — Rate limited: 60/min

Model Registry

GET /api/registry/stats

GET /api/registry/models

GET /api/registry/models/{model_type}/versions

GET /api/registry/artifacts/{job_id}

EDA & Profiling

POST /api/eda/{dataset_id}/analyze — Rate limited: 20/min

GET /api/eda/{dataset_id}/latest

GET /api/eda/{dataset_id}/history

GET /api/eda/reports/{report_id}

POST /api/eda/reports/{report_id}/cancel

POST /api/eda/{dataset_id}/decomposition

GET /api/eda/jobs/all

Monitoring

Drift Detection

GET /api/monitoring/jobs

POST /api/monitoring/drift/calculate — Rate limited: 20/min

GET /api/monitoring/drift/history/{job_id}

GET /api/monitoring/drift/status

Performance

GET /api/monitoring/slow-nodes

Error Tracking

GET /api/monitoring/errors

GET /api/monitoring/errors/count

GET /api/monitoring/errors/grouped

GET /api/monitoring/errors/timeline

GET /api/monitoring/errors/{error_id}

PATCH /api/monitoring/errors/{error_id}/resolve

PATCH /api/monitoring/errors/{error_id}/unresolve

DELETE /api/monitoring/errors

Pipeline Run Logs

POST /api/monitoring/pipeline-logs

`GET /health/ping`

`GET /health`

`GET /health/detailed`

`GET /health/ready`

`GET /data/api/sources`

`GET /data/api/sources/usable`

`GET /data/api/sources/{source_id}`

`GET /data/api/sources/{source_id}/sample`

`DELETE /data/api/sources/{source_id}`

`GET /data/api/sources/{source_id}/export`

`POST /api/ingestion/upload` — Rate limited: 10/min

`GET /api/ingestion/{source_id}/status`

`POST /api/ingestion/{source_id}/cancel`

`POST /api/pipeline/run` — Rate limited: 20/min

`POST /api/pipeline/preview`

`POST /api/pipeline/schema-preview`

`GET /api/pipeline/jobs`

`GET /api/pipeline/jobs/{job_id}`

`GET /api/pipeline/jobs/{job_id}/trials`

`POST /api/pipeline/jobs/{job_id}/cancel`

`POST /api/pipeline/jobs/{job_id}/promote`

`DELETE /api/pipeline/jobs/{job_id}/promote`

`GET /api/pipeline/jobs/{job_id}/evaluation`

`GET /api/pipeline/jobs/node-summaries`

`GET /api/pipeline/registry`

`GET /api/pipeline/stats`

`GET /api/pipeline/datasets/list`

`GET /api/pipeline/datasets/{dataset_id}/schema`

`GET /api/pipeline/hyperparameters/{model_type}`

`GET /api/pipeline/hyperparameters/{model_type}/defaults`

`POST /api/pipeline/pipeline/{dataset_id}/export-notebook`

`POST /api/deployment/deploy/{job_id}` — Rate limited: 10/min

`GET /api/deployment/active`

`GET /api/deployment/history`

`POST /api/deployment/deactivate`

`POST /api/deployment/predict` — Rate limited: 60/min

`GET /api/registry/stats`

`GET /api/registry/models`

`GET /api/registry/models/{model_type}/versions`

`GET /api/registry/artifacts/{job_id}`

`POST /api/eda/{dataset_id}/analyze` — Rate limited: 20/min

`GET /api/eda/{dataset_id}/latest`

`GET /api/eda/{dataset_id}/history`

`GET /api/eda/reports/{report_id}`

`POST /api/eda/reports/{report_id}/cancel`

`POST /api/eda/{dataset_id}/decomposition`

`GET /api/eda/jobs/all`

`GET /api/monitoring/jobs`

`POST /api/monitoring/drift/calculate` — Rate limited: 20/min

`GET /api/monitoring/drift/history/{job_id}`

`GET /api/monitoring/drift/status`

`GET /api/monitoring/slow-nodes`

`GET /api/monitoring/errors`

`GET /api/monitoring/errors/count`

`GET /api/monitoring/errors/grouped`

`GET /api/monitoring/errors/timeline`

`GET /api/monitoring/errors/{error_id}`

`PATCH /api/monitoring/errors/{error_id}/resolve`

`PATCH /api/monitoring/errors/{error_id}/unresolve`

`DELETE /api/monitoring/errors`

`POST /api/monitoring/pipeline-logs`

`GET /api/monitoring/pipeline-logs`

`PATCH /api/monitoring/jobs/{job_id}/description`

`DELETE /api/monitoring/pipeline-logs`

`WS /ws/jobs`