Automated EDA & Profiling

Skyulf provides a powerful automated Exploratory Data Analysis (EDA) engine powered by Polars, designed to handle large datasets efficiently. The EDAAnalyzer generates comprehensive statistical profiles, detects data quality issues, and provides actionable insights.

Quick Start

The Easy Way (Automated Visualization)

Skyulf provides a EDAVisualizer helper that automatically generates a rich terminal dashboard and interactive plots.

Note: This requires the visualization extras: pip install skyulf-core[viz]

import polars as pl
from skyulf import EDAAnalyzer, EDAVisualizer

# 1. Load Data
df = pl.read_csv("your_dataset.csv")

# 2. Analyze
analyzer = EDAAnalyzer(df)
profile = analyzer.analyze(
    target_col="target",
    task_type="Classification", # Optional: Force "Classification" or "Regression"
    date_col="timestamp",       # Optional: Manually specify if auto-detection fails
    lat_col="latitude",         # Optional
    lon_col="longitude"         # Optional
)

# 3. Visualize
viz = EDAVisualizer(profile, df)
viz.summary()  # Prints Data Quality, Stats, Outliers, Target Analysis, Alerts to terminal
viz.plot()     # Opens Matplotlib plots (Distributions, Correlations, PCA, etc.)

The Manual Way (Accessing Raw Stats)

For custom reports or application integrations, use EDAProfile for the raw result returned by EDAAnalyzer.analyze(). This is the recommended public name, available from both skyulf and skyulf.profiling.

The older DatasetProfile import remains the exact same Pydantic class: EDAProfile is DatasetProfile. Existing JSON schemas retain the DatasetProfile title, and saved profiles retain their original pickle path. The pipeline transformer id "DatasetProfile" still identifies the separate inspection node; use that unchanged id in pipeline configurations.

import polars as pl
from skyulf import EDAAnalyzer, EDAProfile

# 1. Load your data into a Polars DataFrame
df = pl.read_csv("your_dataset.csv")

# 2. Initialize the Analyzer
analyzer = EDAAnalyzer(df)

# 3. Run the analysis
# You can optionally specify a target column for supervised analysis
profile: EDAProfile = analyzer.analyze(target_col="target_variable")

# 4. Access the results
print(f"Rows: {profile.row_count}")
print(f"Missing Cells: {profile.missing_cells_percentage}%")

# Access column-specific stats
age_stats = profile.columns["age"].stats
print(f"Mean Age: {age_stats.mean}")

# Check for data quality alerts
for alert in profile.alerts:
    print(f"[{alert.severity}] {alert.message}")

Features

1. Comprehensive Profiling

The analyzer automatically detects column types (Numeric, Categorical, Date, Text, Boolean) and computes relevant statistics: * Numeric: Mean, Median, Std Dev, Quantiles, Skewness, Kurtosis, Histogram. * Normality Tests: For performance, normality tests are performed on a sample of up to 5,000 rows. * Shapiro-Wilk: Used for N < 5,000. * Kolmogorov-Smirnov: Used for N >= 5,000 (on a 5k sample). * Categorical: Unique count, Mode, Frequency distribution. Unique and rare-label counts and the top categories exclude missing values, which are reported separately in the column's missing count and percentage. * Date: Min/Max date, Range, Year/Month distribution. * Text: Avg length, Common words, Sentiment Analysis (Positive/Neutral/Negative).

Native Polars Categorical and Enum columns are profiled as categorical, including their value frequencies. A column with dtype Null (only missing values and no declared type) is reported as Unknown: its missing count, missing percentage and quality alerts remain available, while type-specific statistics are omitted. An all-null column with a declared type, such as Float64, retains that type's profile.

Automatic date detection converts a string column only when the selected format parses every non-missing value in the full column. If a later value uses another format or cannot be parsed, the original strings remain available for profiling; parse failures do not become missing cells or trigger a false missing-data Drop recommendation. Normalize such a column explicitly before requesting temporal analysis. An all-null native Date/Datetime column retains its type and reports actual null minimum/maximum bounds. Regenerate saved profiles to apply these corrections.

String columns are treated as categorical when fewer than 5% of their non-missing values are distinct, or when they contain at most 20 distinct non-missing values with at least one repetition. This also recognizes small datasets with repeated class or city labels. Missing values do not count as labels or dilute the ratio; all-null strings and strings with no repeated values remain Text. A recognized string target receives classification statistics, associations and rule discovery when numeric features are available. Rerun saved analyses to refresh their inferred types and target results.

2. Smart Alerts

Skyulf automatically flags potential data quality issues: * High Null Rate: Columns with >50% missing values. * Constant Columns: Columns with only 1 unique value. * High Cardinality: Categorical columns with too many unique values. * High Correlation: Pairs of features with correlation > 0.95. * Multicollinearity (VIF): Detects features with Variance Inflation Factor > 5.0. Nearly duplicated features still produce warnings when the correlation matrix cannot be reliably inverted. The calculation then uses a separate regression for each feature, preserving low VIF for unrelated features. A value of 999 is a finite warning marker for effectively perfect dependence, not a precise estimate or a cap on other VIF values. Rerun saved analyses to refresh these results. * Class Imbalance: Target variables with skewed class distributions. * Data Leakage: Features with 1.0 correlation to the target.

Smart Recommendations

For highly skewed numeric columns, transformation advice follows the observed value range. Log or Box-Cox is suggested for strictly positive, right-skewed data. Columns containing zero or negative values, left-skewed columns, and profiles with an unknown minimum receive Yeo-Johnson advice instead. These are suggestions to evaluate; EDA does not transform the source data.

The clean-dataset message appears only when no drop, imputation, transformation, encoding or resampling action is recommended. It describes the outcome of these checks, without certifying that a dataset is ready for modeling. An informational balanced-target message can appear alongside it. Rerun saved analyses to refresh their recommendations.

3. Advanced Analysis

Outlier Detection: Uses Isolation Forest to identify anomalous rows.
PCA: Computes Principal Components to visualize high-dimensional data structure.
Clustering (Post-Hoc Analysis): Uses KMeans to automatically discover segments (clusters) in the numeric data and profiles them.
Causal Discovery: Uses the PC Algorithm with Fisher-Z tests to estimate a partially directed graph among numeric variables.
Decision Tree (Rule Discovery): Uses a surrogate Decision Tree model to extract human-readable rules.
- Classification: "If Age > 50 -> High Risk (Confidence: 85%)"
- Regression: "If Age > 50 -> Value = 120.5 (Samples: 100)"
- High Cardinality: Automatically groups rare classes into "Other" for readable trees (e.g. Top 10 Zip Codes).
Target Analysis: Analyzes relationships between features and the target variable (Correlations, ANOVA, Interactions).
Geospatial: Automatically detects Lat/Lon columns and computes bounding boxes.
Time Series: Detects seasonality and trends in datetime columns.

For a categorical or Boolean target, feature association uses the correlation ratio (eta), where 0 means no difference between group means and 1 means no within-group variation. Each numeric feature is evaluated using only rows where both that feature and the target are present. Missing values in another feature do not affect its score, and this calculation does not remove rows from your dataset. Features with no complete pairs are omitted; constant observed features keep a score of 0. Rows missing either value contribute neither to group sizes nor to the sums of squares used in this calculation.

Previously saved reports retain their recorded scores. Rerun EDA to refresh the association values and feature ranking after this correction.

Categorical and Boolean targets keep their original names and values. They appear in Target Analysis through category-aware associations and box plots; they are not converted to arbitrary numeric codes for Pearson correlations or causal discovery. For example, selecting species does not create a species_encoded variable. The web app explains this omission in the causal and correlation views, including when no numeric graph is available. Numeric targets classified as categories are treated the same way.

Target and grouping column names such as mean, n, min, median, count and group are supported without dropping their statistics or recommendations.

4. Task Type Control

By default, Skyulf automatically detects if your target is Classification (Categorical) or Regression (Numeric). However, you can override this behavior:

Force Classification: Useful for ID columns or numeric codes (e.g., Zip Code, Status Code) that should be treated as categories.
Force Regression: Useful for numeric ratings (e.g., 1–5) that you want to treat as measurements. String labels are not automatically converted into numeric ratings.

# Force Classification on a numeric ID column
profile = analyzer.analyze(target_col="zip_code", task_type="Classification")

An explicit classification task also excludes a numerically stored target from target Pearson and Fisher-Z analysis, while retaining categorical associations. An explicit regression task permits an existing numeric target even if its low number of distinct values led to a categorical profile type.

Visualization Support (`skyulf-core[viz]`)

PCA plots keep points whose target label is missing, showing them as gray crosses marked Unlabeled. If every label is missing, the plot still shows all points without a target colorbar. Categorical PCA and geospatial plots use a deterministic color mapping for the same set of category labels, even when row order or the Python process changes. Recreate existing plots with EDAVisualizer to use the corrected rendering; the saved profile can be reused. PCA treats text labels such as "nan" and "inf" as categories so their points remain visible.

The core library is designed to be lightweight. To use the EDAVisualizer and generate plots, you must install the optional visualization dependencies:

pip install skyulf-core[viz]

This installs: * matplotlib: For generating plots. * rich: For beautiful terminal dashboards.

EDA charts in the web app

The PCA & Clusters, Bivariate scatter charts and Geospatial map share category colors. Colors stay stable when rows are reordered, provided the displayed category set stays the same. Adding or removing categories can change the mapping; the palette repeats for large category sets. Category labels use English alphabetical ordering, including case and accents, so the browser's language does not change their color or marker assignment.

The 2D and 3D scatter views use the same five repeating marker shapes: circle, square, diamond, plus (+) and diagonal cross (x). The legend shows the shape actually drawn for each category. Missing-label points use gray plus markers in both views.

Missing target labels form a separate gray Unlabeled group. Real values such as Other, Unlabeled, constructor and __proto__ remain ordinary categories. If Unlabeled already exists, the missing group is named Unlabeled (missing), with a numeric suffix if needed to keep names distinct. All-missing targets still show every point and a legend entry. Maps use circles and show their category legend, including a filter for long lists; they retain category colors above twenty groups.

Reload the web app to use these rendering fixes with existing saved profiles. The chart data table and CSV export retain the original label values.

Filtering & Exclusion

You can refine your analysis by filtering rows or excluding columns without modifying your original dataframe.

# Run analysis on a subset of data
profile = analyzer.analyze(
    target_col="price",
    # Filter rows where 'region' is 'US' and 'age' > 18
    filters=[
        {"column": "region", "operator": "==", "value": "US"},
        {"column": "age", "operator": ">", "value": 18}
    ],
    # Exclude 'id' and 'timestamp' columns from the report
    exclude_cols=["id", "timestamp"]
)

Column exclusions also apply to explicit lat_col and lon_col selections. If either selected coordinate is excluded, profile.geospatial is None, so its points, bounds and centroid are omitted. Exclusions persist when you reuse an analyzer; create a new EDAAnalyzer(df) to start with all columns again. Rerun saved analyses to refresh their stored reports after an update.

Exclusions also take precedence over the selected target: an excluded target does not contribute associations, causal nodes, inferred task type or decision tree rules. Its name remains in profile.target_col to record the requested configuration; profile.causal_target_exclusion_reason is "excluded".

Advanced Analysis Modules

The EDAAnalyzer automatically runs several advanced analysis modules if the data supports them. Here is how to use and interpret each one.

1. Outlier Detection

Uses Isolation Forest to identify anomalous rows in your dataset. This is useful for cleaning data or detecting fraud/errors.

For more than 50,000 rows, detection uses a reproducible sample of 50,000 rows. Each returned index is the zero-based position in the analyzed dataframe after row filters, before sampling. Without filters, it is the position in the input dataframe. It is never the position within the randomized sample. The outlier count and percentage describe the analyzed sample when sampling is used.

if profile.outliers:
    print(f"Total Outliers: {profile.outliers.total_outliers}")
    print(f"Outlier Percentage: {profile.outliers.outlier_percentage}%")

    # Inspect the top most anomalous rows
    for outlier in profile.outliers.top_outliers[:5]:
        print(f"Row Index: {outlier.index}, Score: {outlier.score}")

        # See why it was flagged (feature contribution)
        if outlier.explanation:
            print("  Reason:", outlier.explanation)

2. Multivariate Analysis (PCA)

Principal Component Analysis (PCA) projects high-dimensional data into 2D or 3D space to visualize clusters and structure.

if profile.pca_data:
    # Access the projected points (x, y, z)
    for point in profile.pca_data[:5]:
        print(f"x: {point.x}, y: {point.y}, Label: {point.label}")

3. Causal Discovery

Skyulf integrates causal-learn to run PC with Fisher-Z tests on eligible numeric measurements. PC can leave relationships unoriented, so the result is a partially directed graph. It is an observational estimate for exploring hypotheses, not proof that one variable causes another.

Categorical and Boolean variables are omitted from this method. A categorical target such as species remains available in Target Analysis; it is not given an arbitrary ordering to place it in the causal graph. Fisher-Z's linear-Gaussian assumptions still matter for the numeric variables you analyze.

The graph uses at least 50 and up to the first 5,000 complete rows across its selected columns. It includes all eligible variables up to a maximum of 15. Above that limit, an eligible selected numeric target is retained along with up to 14 features ranked by absolute Pearson correlation with it. Without an eligible numeric target, the 15 highest-variance numeric variables are selected. A feature named target or label does not override your choice.

New reports record profile.causal_graph.selection_method as "all", "target_correlation" or "variance". If the selected target cannot enter numeric analysis, profile.causal_target_exclusion_reason is "categorical", "excluded" or "unsupported"; this explanation is available even when the graph itself is None. Both fields are optional when reading older reports.

if profile.causal_graph:
    print(f"Graph has {len(profile.causal_graph.nodes)} nodes and {len(profile.causal_graph.edges)} edges.")

    for edge in profile.causal_graph.edges:
        # Types: "directed" (->), "undirected" (--), "bidirected" (<->)
        print(f"{edge.source} {edge.type} {edge.target}")

Directed edges preserve the orientation returned by PC. The web graph keeps distinct column names such as petal width and petal_width separate and shows both arrowheads for a bidirected edge. Loading a report with an empty graph clears the previous graph.

Reload the app for rendering corrections. Run Analyze again to regenerate saved causal graphs, target statistics, rules and outlier row indices: existing stored reports retain the results produced by their original analysis.

4. Geospatial Analysis

Automatically detects Latitude/Longitude columns and computes bounding boxes and centroids.

if profile.geospatial:
    print(f"Bounds: {profile.geospatial.min_lat}, {profile.geospatial.min_lon} to {profile.geospatial.max_lat}, {profile.geospatial.max_lon}")
    print(f"Centroid: {profile.geospatial.centroid_lat}, {profile.geospatial.centroid_lon}")

5. Time Series Analysis

Detects datetime columns and analyzes trends and seasonality. It automatically adjusts the resampling interval (e.g., 1s, 1h, 1d) based on the data duration to provide meaningful trend lines.

if profile.timeseries:
    print(f"Time Column: {profile.timeseries.date_col}")

    # Access trend data
    for point in profile.timeseries.trend[:5]:
        print(f"Date: {point.date}, Value: {point.values}")

6. Smart Date Detection

Skyulf uses a robust "Tournament Strategy" to correctly identify date formats in ambiguous columns (e.g., distinguishing between MM/DD/YYYY and DD/MM/YYYY).

How it works: 1. Heuristic Check: First, it checks column names for keywords like date, time, created_at to avoid expensive parsing on non-date columns. 2. Candidate Testing: It attempts to parse a sample of the data using multiple formats: * ISO Standard (YYYY-MM-DD) * Common US Format (MM/DD/YYYY) * Common EU Format (DD/MM/YYYY) 3. Entropy Selection: For ambiguous cases (e.g., 01/02/2023 could be Jan 2nd or Feb 1st), it calculates the Month Entropy (number of unique months) for each candidate format. * It selects the format that yields the highest number of unique months, assuming that real-world data typically spans more than just January.

Examples

We provide complete, runnable scripts in the docs/examples/scripts/ directory. These examples demonstrate how to use Skyulf as a library with rich terminal output and visualizations.

Prerequisites for Examples

To run these examples, you'll need a few extra libraries for visualization and pretty printing:

pip install matplotlib rich scikit-learn

1. Time Series Analysis

Script: docs/examples/scripts/eda_timeseries.py

This example generates synthetic sales data with a trend and weekly seasonality. It demonstrates: - Automatic date column detection. - Extracting and plotting the Trend component. - Identifying seasonality patterns (e.g., day of week).

Output: - A terminal table showing trend values. - A plot window showing the trend.

2. Geospatial Analysis

Script: docs/examples/scripts/eda_geospatial.py

This example simulates store locations around New York City. It demonstrates: - Automatic detection of Latitude/Longitude columns. - Computing the Centroid and Bounding Box. - Visualizing the store locations and centroid on a map.

Output: - A terminal summary of spatial stats. - A scatter plot window.

3. Comprehensive Workflow (Iris Dataset)

Script: docs/examples/scripts/eda_comprehensive.py

A full-featured example using the classic Iris dataset. It covers the entire Skyulf EDA pipeline: - Data Quality: Missing values and duplicates. - Numeric Stats: Skewness, Kurtosis, and Normality Tests (Shapiro-Wilk/D'Agostino). - Outlier Detection: Finding anomalies with Isolation Forest. - Distributions: Histograms for numeric features. - Correlations: Heatmap of feature relationships. - Target Analysis: Boxplots showing feature distribution by target class. - Scatter Matrix: Pairwise scatter plots of features. - PCA: Projecting data into 2D space for visualization. - Causal Discovery: Inferring causal relationships between variables. - Smart Alerts: Automatic warnings about data issues.

Output: - A detailed "Dashboard" in the terminal. - Plot windows for Distributions, Correlations, Target Analysis, Scatter Matrix, and PCA.

4. Automated Analysis (The Easy Way)

Script: docs/examples/scripts/eda_automated.py

This example demonstrates the simplified workflow using the EDAVisualizer class. Instead of writing manual plotting code, you can generate a full report in just 3 lines.

Output: - A rich terminal dashboard. - All standard plots (Distributions, Correlations, PCA, etc.).

5. Time Series & Geospatial (Manual Config)

Script: docs/examples/scripts/eda_timeseries_geo.py

This example shows how to handle cases where auto-detection might fail or when you want to be explicit. It demonstrates: - Manually specifying date_col, lat_col, and lon_col in the analyze() method. - Visualizing both Time Series trends and Geospatial maps in the same report.

Output: - A terminal summary including Time Series and Geospatial sections. - Plots for Time Series Trend and Geospatial Distribution.

Performance

The profiler is built on Polars, making it significantly faster than Pandas-based alternatives like pandas-profiling or ydata-profiling, especially for datasets with millions of rows.

Data Drift & Monitoring

Skyulf includes Data Drift Detection module to monitor how your data changes over time, ensuring model reliability in production.

Concept

Data Drift occurs when the statistical properties of the input data (production) change compared to the data used to train the model (reference). This can lead to model degradation.

Metrics

Skyulf calculates drift using four key statistical metrics for every numerical column:

Wasserstein Distance (Earth Mover's Distance): Measures the "work" needed to transform one distribution into the other. Good for detecting shifts in shape and location. The reported value is the distance divided by the reference column's standard deviation, so one threshold means the same thing whatever unit the column is in; the untransformed distance is kept in raw_value.
KS Test (Kolmogorov-Smirnov): A non-parametric test that compares cumulative distribution functions. The drift decision is made on the statistic (the maximum distance between the two CDFs, 0–1); the p-value is reported alongside it as ks_test_p_value for diagnostics but never decides drift, because it shrinks with sample size.
PSI (Population Stability Index): A widely used industry standard for measuring population shifts.
- PSI < 0.1: No significant drift.
- 0.1 <= PSI < 0.25: Moderate drift.
- PSI >= 0.25: Significant drift (Action required).
KL Divergence (Kullback-Leibler): Measures how one probability distribution diverges from a second, expected probability distribution.

Schema Drift

The system also monitors for structural changes: * Missing Columns: Critical alerts for features present in training but missing in production. * New Columns: Alerts for unexpected features appearing in production data.

Both count toward report.drifted_columns_count, so that figure covers distribution drift and schema drift. A missing column has no values left to compare and therefore never appears in column_drifts — the count is the only place it is reported.

Using the Data Drift UI

Navigate to the Data Drift page in the ML Canvas.
Step 1: Select a Reference Job (Training Job) from the list. This loads the statistical profile of the data used during training.
Step 2: Upload your Current Data (Production Data) as a CSV or Parquet file.
Analyze: Click "Run Analysis".
Review Report:
- Check the Drift Score and Schema Alerts at the top.
- Expand any column row to see detailed metrics and Interactive Histograms comparing the two distributions side-by-side.

Using the Library (Python API)

Script: docs/examples/scripts/data_drift_check.py

You can also use the drift detection engine programmatically within your own scripts or pipelines using skyulf-core.

import polars as pl
from skyulf import DriftCalculator

# 1. Load Data (Reference vs Current)
# Reference: The data your model was trained on
# Current: The new data from production
ref_df = pl.read_csv("training_data.csv")
curr_df = pl.read_csv("production_data.csv")

# 2. Initialize Calculator
calculator = DriftCalculator(ref_df, curr_df)

# 3. Calculate Drift
# You can optionally override default thresholds
report = calculator.calculate_drift(thresholds={"psi": 0.2, "wasserstein": 0.1})

# 4. Inspect Results
print(f"Drifted Columns: {report.drifted_columns_count}")

# Check for Schema Drift
if report.missing_columns:
    print(f"⚠️ Missing Columns: {report.missing_columns}")
if report.new_columns:
    print(f"ℹ️ New Columns: {report.new_columns}")

# Check for Statistical Drift
for col_name, drift_info in report.column_drifts.items():
    if drift_info.drift_detected:
        print(f"\n🚨 Drift detected in '{col_name}':")
        for metric in drift_info.metrics:
            status = "FAIL" if metric.has_drift else "PASS"
            print(f"  - {metric.metric}: {metric.value:.4f} [{status}]")

        if drift_info.suggestions:
            print(f"  💡 Suggestion: {drift_info.suggestions[0]}")