Optimization and Training Strategies

In highFIS, the model definition is decoupled from the optimization loops. This separation of concerns allows the model layers to focus purely on PyTorch forward execution, while training protocols handle batching, learning rate schedules, parameter locking/freezing, and specialized multi-phase algorithms.

All training strategies inherit from the abstract class BaseTrainer and decouple the optimization loop from the scikit-learn estimator interface.

The Trainer Architecture

Each estimator class in highFIS delegates its .fit() call to a dedicated trainer. The trainer handles the standard training loop (epoch-wise validation, early stopping, logging) as well as advanced phase transitions.

graph TD
    Estimator[Estimator.fit] --> Trainer[BaseTrainer]
    Trainer --> GT[GradientTrainer]
    Trainer --> DGT[DGTrainer]
    Trainer --> FSRET[FSRETrainer]

    GT -->|Single-phase| GD[Standard Mini-batch SGD/Adam]
    DGT -->|Three-phase| GateOpt[Data-Guided Antecedent & Gate Optimization]
    FSRET -->|Three-phase| FSREOpt[Feature Selection & Rule Extraction Optimization]

1. GradientTrainer

The GradientTrainer implements standard single-phase mini-batch gradient descent. It is used by baseline models like TSK, HTSK, LogTSK, DombiTSK, ADMTSK, AYATSK, ADATSK, and ADPTSK.

Note: All model parameters (antecedent means/widths, rule weights, and consequent weights) are optimized simultaneously using standard PyTorch optimizers (e.g., Adam) and learning rate schedulers.

Configuration Parameters

You can configure the trainer directly through the estimator's constructor arguments: * epochs: Total training epochs (default: 100). * learning_rate: Step size for weight updates (default: 0.01). * batch_size: Size of mini-batches — see Batch size below. Accepts an integer, None for full-batch training, or "auto" (the default) to follow the source paper's protocol for that model family. * weight_decay: L2 regularization strength, applied to the consequent parameters (antecedent and rule parameters are left decay-free). * eval_metrics_every: Evaluate training metrics every n epochs (default: 1); 0 skips the training-metric pass entirely. This only affects the "train_<metric>" entries in history_ — validation metrics, which drive early stopping, are evaluated every epoch regardless. Raising it (or setting 0) is a cheap speed-up when you do not need per-epoch training curves. * scheduler_class / scheduler_params: Learning-rate schedule; see below.

Numerical precision

Models train in whatever floating-point precision is PyTorch's default when fit is called. The default is single precision (float32); to train and predict in double precision, set it before fitting:

import torch

torch.set_default_dtype(torch.float64)
clf.fit(X, y)  # parameters, inputs and predictions are all float64

Batch size

batch_size takes three kinds of value:

The papers these models come from prescribe different protocols, so there is no single sensible number; "auto" follows each one:

After fit, the size actually used is available as batch_size_, while the batch_size parameter keeps the value you passed:

from highfis import HDFISProdClassifier

clf = HDFISProdClassifier(n_mfs=3, epochs=5, random_state=42)
clf.fit(X_train, y_train)
print("configured:", clf.batch_size, "| used:", clf.batch_size_)

Example: Training with GradientTrainer and Inspecting History

This example shows how to configure a model using standard gradient optimization, and then inspect the resulting training logs.

from sklearn.datasets import make_classification
from sklearn.preprocessing import MinMaxScaler
from highfis import HTSKClassifier

# Generate mock classification data
X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_scaled = MinMaxScaler().fit_transform(X)

# Instantiate the HTSK Classifier with custom optimization settings
clf = HTSKClassifier(
    n_mfs=3,
    mf_init="kmeans",
    epochs=120,
    learning_rate=0.005,
    batch_size=64,
    weight_decay=1e-5,
    verbose=True,
    random_state=42
)

# Fit the model (delegates training to GradientTrainer)
clf.fit(X_scaled, y)

# The training history is captured in clf.history_
print("Final training loss:", clf.history_["train_total_loss"][-1])
print("Epochs executed:", clf.history_["stopped_epoch"])

2. DGTrainer (Double-Gated Training)

Specialized for DGTSK and DGALETSK model families, the DGTrainer implements the three-phase Data-Guided (DG) training protocol. This protocol is critical for establishing sparse high-dimensional structures by systematically pruning feature and rule gates.

                  ┌─────────────────────────────────────────┐
                  │ Phase 1: Antecedent & Consequent Warmup  │
                  │ (Gate parameters are frozen)             │
                  └────────────────────┬────────────────────┘
                                       │
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │ Phase 2: Gate Optimization              │
                  │ (Antecedent parameters are frozen)      │
                  └────────────────────┬────────────────────┘
                                       │
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │ Phase 3: Joint Fine-Tuning              │
                  │ (All parameters unfrozen and optimized) │
                  └─────────────────────────────────────────┘

The Three Phases

1. Warm-up Phase (fit_warmup_phase): Antecedent membership functions and consequents are optimized to match the target signal. The structural gates are frozen to prevent premature pruning.
2. Gate Selection Phase (fit_dg_phase): The gate parameters (\(\lambda, \theta\)) are unfrozen and optimized under an L1 or entropy penalty to induce sparsity, while antecedents remain frozen.
3. Joint Fine-Tuning (fit_joint_phase): All parameters are unfrozen and optimized jointly to polish the final sparse model.

Key DGTrainer Settings

• dg_epochs / finetune_epochs: Duration of the gate optimization and fine-tuning phases.
• dg_learning_rate / finetune_learning_rate: Separate step sizes for phase-specific updates.
• zeta_lambda / zeta_theta: Pruning threshold grid search options (defaults: [0.0, 0.25, 0.5, 0.75, 1.0]).
• use_lse: Re-estimates consequent weights using Least Squares Estimation (LSE) during search.
• structural_pruning: Hard-prunes the PyTorch neural network parameters of dropped rules/features to speed up forward/backward passes.

Example: Double-Gated Sparse Feature Selection

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from highfis import DGTSKClassifier
from highfis.optim import DGTrainer

# Create high-dimensional data (e.g. 50 features, only 10 informative)
X, y = make_classification(n_samples=1000, n_features=50, n_informative=10, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define custom three-phase trainer settings
custom_trainer = DGTrainer(
    dg_epochs=30,
    dg_learning_rate=0.01,
    zeta_lambda=[0.1, 0.3, 0.5, 0.7],  # Custom feature gate threshold grid
    zeta_theta=[0.2, 0.4, 0.6],        # Custom rule gate threshold grid
    use_lse=True,
    finetune_epochs=150,
    finetune_learning_rate=0.005,
    verbose=True
)

# Instantiate the estimator and assign our trainer
clf = DGTSKClassifier(
    n_mfs=3,
    trainer=custom_trainer,
    random_state=42
)

# Train the model (automatically cycles through DG optimization, threshold search, and fine-tuning)
clf.fit(X_train, y_train, x_val=X_val, y_val=y_val)

# Introspect the pruned structural statistics (recorded during threshold search)
threshold = clf.history_["threshold"]
print("Retained Feature Indices:", threshold["surviving_feature_indices"])
print("Retained Rules:", len(threshold["surviving_rule_indices"]))

3. FSRETrainer (Feature Selection & Rule Extraction)

The FSRETrainer implements a custom three-phase optimization protocol tailored for FSRE-ADATSK estimators. It targets the simultaneous optimization of softmin aggregation parameters and the structural feature/rule gates.

• Phase 1 (Feature Selection): Focuses on determining the initial membership function layouts and optimizing feature gates \(\lambda_d\).
• Phase 2 (Rule Extraction): Locks selected features, expands the system to the full Rule Base (\(En-FRB\)), and optimizes rule gates \(\theta_r\) under pruning penalties.
• Phase 3 (Refinement/Fine-tuning): Locks the pruned architecture and updates the remaining membership parameters and consequent weights to restore accuracy.

Key FSRETrainer Settings

• fs_epochs / re_epochs / finetune_epochs: Durations for the three phases.
• zeta_lambda: Feature pruning threshold coefficient (default: 0.5). Larger values retain more features. Recommend 0.4 for high-dimensional datasets.
• zeta_theta: Rule pruning threshold coefficient (default: 0.3). Recommend 0.5 for high-dimensional datasets.
• structural_pruning: Hard-prunes the structural tensors in the model to optimize execution speed.

Example: Feature Selection and Rule Extraction in Softmin TSK

from sklearn.datasets import make_classification
from highfis import FSREADATSKClassifier
from highfis.optim import FSRETrainer

# Load classification data
X, y = make_classification(n_samples=800, n_features=20, random_state=42)

# Configure the three-phase FSRE trainer
fsre_trainer = FSRETrainer(
    fs_epochs=20,
    fs_learning_rate=0.01,
    re_epochs=20,
    re_learning_rate=0.01,
    finetune_epochs=100,
    finetune_learning_rate=0.005,
    zeta_lambda=0.4,   # Pruning threshold coefficient for features
    zeta_theta=0.5,    # Pruning threshold coefficient for rules
    verbose=True
)

# Instantiate the model with our customized FSRE training loop
clf = FSREADATSKClassifier(
    n_mfs=3,
    trainer=fsre_trainer,
    random_state=42
)

# Fit the classifier
clf.fit(X, y)

Optimizer selection

The optimizer is chosen automatically from the model family, so there is no optimizer_class constructor argument. ADATSK models train with SGD; ADPTSK, ADMTSK, DombiTSK and AYATSK use Adam; the remaining single-phase models use AdamW with weight decay applied only to the consequent parameters. The DG-TSK and DG-ALETSK estimators additionally expose an optimizer_type argument ("sgd", "adam", or "adamw") to follow their respective papers.

To supply a fully custom optimizer instance, build a GradientTrainer yourself and pass it via trainer= (only the DG and FSRE estimators accept a trainer= argument), or call GradientTrainer.fit(model, x, y, optimizer=my_optimizer) directly.

Learning-rate schedules

Every estimator accepts a scheduler_class plus scheduler_params. You pass the scheduler class, not an instance — the scheduler must bind to the optimizer, and the optimizer is only built inside fit, so a pre-constructed scheduler would decay an optimizer that is never stepped.

from torch.optim.lr_scheduler import StepLR
from highfis import HTSKClassifier

clf = HTSKClassifier(
    epochs=150,
    learning_rate=1e-2,
    scheduler_class=StepLR,
    scheduler_params={"step_size": 50, "gamma": 0.5},
    random_state=42,
)
clf.fit(X_train, y_train)

# The realised rate for each epoch is recorded in history_.
print("Learning rate per epoch:", clf.history_["lr"])

For the multi-phase DG-TSK, DG-ALETSK and FSRE-ADATSK estimators each phase builds its own optimizer, so the class is instantiated once per phase.

Note: scheduler_class is a class object, which torch.load(weights_only=True) cannot unpickle, so it is not written into checkpoints. A model reloaded from disk is not resuming training and does not need it.

4. Training History and Metric Logging

Each trainer populates the estimator's history_ attribute after fitting the model. Internally, the optimization loop computes all metrics on the training and validation sets using pure PyTorch on the active device (e.g., CPU or CUDA GPU) to avoid expensive, synchronous data transfers to the host memory during training. Only the final scalar metrics are converted to Python floats when written to the history dictionary at the end of each epoch.

History Structure and Keys

The history_ dictionary contains the following keys depending on the trainer and parameters:

• "train": The loss value computed on the training set for each epoch.
• "ur": The Uniform Regularization loss value computed on the training set for each epoch.
• "val": The validation loss computed on the validation set for each epoch (only present if validation_data or x_val/y_val is provided).
• "train_<metric>" / "val_<metric>": The epoch-wise performance of custom or default metrics.
• "stopped_epoch": The final epoch index executed (especially useful when early stopping is triggered).

Default and Custom Metrics

If no metrics are explicitly specified, the trainers automatically configure task-appropriate defaults: * Classification: Defaults to "accuracy" (creating "train_accuracy" and "val_accuracy" keys). * Regression: Defaults to "mse" (creating "train_mse" and "val_mse" keys).

The "train_<metric>" keys are only present when eval_metrics_every is greater than 0 (the default is 1). The "val_<metric>" keys are present whenever a validation set is supplied. Passing metrics=[] to .fit() disables both.

Reading history_ correctly

Each series carries one canonical name — the loss decomposition is train_total_loss = train_loss + ur_weight * train_ur_loss:

!!! warning "The last row is not always the fitted model"

With a validation set and `restore_best=True` (the default), training continues past
the best epoch and the returned model is rolled back to it — so `history_[...][-1]`
describes a **different** model than the one you hold. Index with `best_epoch`
instead.

from highfis import HTSKClassifier

clf = HTSKClassifier(n_mfs=3, epochs=30, random_state=42)
clf.fit(X_train, y_train, x_val=X_val, y_val=y_val)

best = clf.history_["best_epoch"]
print("best epoch:", best)
print("last row (a discarded model):", round(clf.history_["train_accuracy"][-1], 3))
print("the returned model:          ", round(clf.history_["train_accuracy"][best], 3))

best_epoch is an epoch index, so it lines up with the per-epoch series. Without a validation set it is None — the last row is the returned model. If you set eval_metrics_every above 1, the train_<metric> series are subsampled and no longer indexed by epoch.

To track additional metrics, pass them as a list of strings to the .fit() method:

# Train the model while tracking accuracy and macro F1 score
clf.fit(
    X_train, y_train,
    x_val=X_val, y_val=y_val,
    metrics=["accuracy", "f1_macro"]
)

# Inspect the captured metrics
print("Training Macro F1 per epoch:", clf.history_["train_f1_macro"])
print("Validation Accuracy per epoch:", clf.history_["val_accuracy"])

5. Uniform Regularization (UR)

In Takagi-Sugeno-Kang (TSK) fuzzy neural networks, mini-batch gradient descent can sometimes lead to rule starvation or rule dominance, where a subset of rules dominates the output while others are never activated. This reduces the generalization performance of the model.

Uniform Regularization (UR) resolves this by penalizing the deviation of the average rule activations from a target uniform distribution over each batch.

Mathematical Formulation

The Uniform Regularization loss \(L_{\text{UR}}\) added to the primary task loss is formulated as:

Where: * \(R\) is the number of fuzzy rules. * \(\bar{a}_r\) is the average normalized activation (firing strength) of rule \(r\) across all samples in the current mini-batch: $$ \bar{a}r = \frac{1}{B} \sum{b=1}^{B} \bar{w}_r(x_b) $$ * \(t\) is the target uniform activation level. Typically, this is set to \(1/R\), meaning every rule is expected to contribute equally on average.

Configuration Parameters

Uniform Regularization is configured using two hyperparameters on the estimator: * ur_weight: The regularization strength (default: 0.0). If set to 0.0, no UR penalty is added to the loss function during backpropagation. However, the value of the metric is still computed and logged to history_["train_ur_loss"] for monitoring. * ur_target: The target activation level \(t\) (default: None, which defaults to \(1/R\)).

Interpretation of UR Values

• ur \(\approx 0.0\): The rule activations are perfectly and uniformly distributed across the batch. Every rule is contributing.
• ur \(> 0.5\): The rule activations are highly unbalanced. This suggests that either:
  1. Only a few rules are doing all the work (dominating), while the others are starved (inactive).
  2. Many rules are activating in a highly correlated way, reducing the diversity of the rule base.

Scientific Reference

For more details on the rationale and effectiveness of Uniform Regularization, refer to:

Y. Cui, D. Wu and J. Huang, "Optimize TSK Fuzzy Systems for Classification Problems: Minibatch Gradient Descent With Uniform Regularization and Batch Normalization," in IEEE Transactions on Fuzzy Systems, vol. 28, no. 12, pp. 3065-3075, Dec. 2020, doi: 10.1109/TFUZZ.2020.2967282.