Pipeline Quickstart

This guide shows a production-style workflow: split → fit → evaluate → predict → save/load.

What data shape should I pass?

SkyulfPipeline.fit(...) supports two common inputs:

1. A single pd.DataFrame that includes the target column.

   • You pass target_column="..." and Skyulf splits features/target internally.
   • This is what the quickstart uses, because it's the simplest onboarding path.

2. A SplitDataset (recommended when you already have train/test splits).

   • Each split can be either a DataFrame (with the target column) or a tuple (X, y).

You do not need to manually build X and y for the pipeline unless you want to.

If you prefer a scikit-learn-style workflow (X/y + train_test_split), see:

• Validation vs scikit-learn (Proof)

Complete runnable example

This single snippet is intentionally end-to-end (no duplicated setup across steps).

import tempfile
from pathlib import Path

import pandas as pd

from skyulf import SkyulfPipeline

# 1) Define a pipeline config
# Each preprocessing step is:
#   {"name": "...", "transformer": "TransformerType", "params": {...}}
config = {
    "preprocessing": [
        {
            "name": "split",
            "transformer": "TrainTestSplitter",
            "params": {
                "test_size": 0.25,
                "validation_size": 0.0,
                "random_state": 42,
                "shuffle": True,
                "stratify": True,
                "target_column": "target",
            },
        },
        {
            "name": "impute",
            "transformer": "SimpleImputer",
            "params": {"strategy": "mean", "columns": ["age"]},
        },
        {
            "name": "encode",
            "transformer": "OneHotEncoder",
            "params": {"columns": ["city"], "drop_original": True, "handle_unknown": "ignore"},
        },
    ],
    "modeling": {
        "type": "random_forest_classifier",
        "params": {"n_estimators": 50, "random_state": 42},
    },
}

# 2) Training data (includes the target column)
df = pd.DataFrame(
    {
        "age": [10, 20, None, 40, 50, 60, None, 80],
        "city": ["A", "B", "A", "C", "B", "A", "C", "B"],
        "target": [0, 1, 0, 1, 1, 0, 1, 0],
    }
)

# 3) Fit (learn params on train split, apply to test split)
pipeline = SkyulfPipeline(config)
metrics = pipeline.fit(df, target_column="target")
print("Metrics keys:", list(metrics.keys()))

preprocessing = metrics["preprocessing"]
print(preprocessing["summary"]["fit_time"])
print(preprocessing["steps"]["1:impute"]["details"]["fill_values"])
# preprocessing["fit_time"], ["peak_memory_bytes"], ["rows_in"], and
# ["rows_out"] remain compatibility aliases for the summary values above.

# 4) Predict (feature-only dataframe)
incoming = pd.DataFrame({"age": [25, None], "city": ["A", "C"]})
preds = pipeline.predict(incoming)
print("Preds:")
print(preds)

# 5) Save / load
with tempfile.TemporaryDirectory() as tmp:
    model_path = Path(tmp) / "model.pkl"
    pipeline.save(model_path)
    loaded = SkyulfPipeline.load(model_path)
    preds2 = loaded.predict(incoming)

print("Preds after reload:")
print(preds2)

Inspecting a pipeline

Evaluation population after filtering

Explicit row filters, such as dropping missing rows, select the eligible population for both ordinary evaluation and tuning/CV. Fitted cleaning bounds come from training data. Serving preserves the requested row alignment.

Held-out reports record input_rows, scored_rows, and excluded_rows. The denominator begins when that test or validation split exists. For example, filtering 12 rows out of a 120-row test split reports 120 input, 108 scored, and 12 excluded. Filtering before splitting changes the source population; those earlier exclusions are not counted again as test exclusions. Training coverage describes the final training frame, not all rows loaded upstream. Consequently, the displayed metrics describe eligible observations and should not be interpreted as accuracy on the original unfiltered dataset.

A split with no eligible rows has no evaluation metrics and an explicit unavailable reason. Missing original counts remain unknown rather than zero. In Canvas, merging nonempty filtered evaluation branches without shared row lineage is rejected: equal counts do not establish that both branches contain the same rows. Merge branches before applying row filters. All-empty branches retain their coverage and remain visible in the report. This also rejects branches that inherit the same earlier filter: the recorded counts do not establish shared row identity, even when both branches happen to contain identical rows.

Pipeline description and diagram

A fitted (or unfitted) pipeline is fully introspectable — useful for logs, docs, PR descriptions, and model registries:

print(pipeline.describe())
# Human-readable summary: each preprocessing step with its params, then the model.

print(pipeline.to_mermaid())
# Mermaid flowchart of the topology (data -> steps -> model).
# Paste into GitHub docs/PRs, or render with any mermaid viewer.

card = pipeline.export_model_card()
# JSON-friendly dict: schema_version, fitted, fingerprint, preprocessing
# lineage, model + params, fit metrics, and `diagram` (the same mermaid
# string as to_mermaid()). Intended for audit logs and registries.

The same surfaces appear across the platform:

• Experiments page — every completed run persists its mermaid topology (metrics.pipeline_diagram); select runs and open the Pipeline Diagram tab.
• Notebook export — exported notebooks embed the topology as a rendered "Pipeline topology" markdown cell (compact and full modes).

Diagram labels are human-readable — internal node ids are never shown. Each node gets the step's display name, and a second line with the most relevant detail: a digest of the step params for config diagrams (strategy: mean), or the run's one-line summary for completed jobs (acc 0.87 · f1 0.84). Model nodes show the algorithm in readable form (Random Forest Classifier).

Mermaid labels are always emitted double-quoted, so node names containing parentheses, brackets, or quotes stay parseable.