DG-ALETSK

DG-ALETSK combines adaptive Ln-Exp softmin aggregation with feature and rule gating, enabling simultaneous feature selection and rule extraction in high-dimensional fuzzy models.

Reference

G. Xue, J. Wang, B. Yuan and C. Dai, "DG-ALETSK: A High-Dimensional Fuzzy Approach With Simultaneous Feature Selection and Rule Extraction," in IEEE Transactions on Fuzzy Systems, vol. 31, no. 11, pp. 3866–3880, Nov. 2023, doi: 10.1109/TFUZZ.2023.3270445.

Overview

DG-ALETSK is a high-dimensional TSK fuzzy model that jointly learns:

• feature selection via antecedent gates;
• rule extraction via consequent gates;
• a differentiable adaptive Ln-Exp softmin antecedent;
• a zero-order DG phase followed by first-order fine tuning.

The highFIS implementation supports both classification and regression via DGALETSKClassifier and DGALETSKRegressor.

Mathematical Formulation

Antecedent membership

Each antecedent uses Gaussian membership functions:

with learned antecedent centers \(m_{r,d}\) and spreads \(\sigma_{r,d}>0\).

Feature gating

DG-ALETSK embeds one gate per input feature in the antecedent. The paper's enhanced gate function (eq. 24) with \(k=10\) is used:

The gated membership term is computed by raising the base membership value to the power of the gate value (paper eq. 12):

When \(M(\lambda_d)=1\) the gate is fully open (membership unchanged); when \(M(\lambda_d)=0\) the gate is closed (\(\tilde{\mu}_{r,d}=1\), so the feature contributes nothing to the softmin).

Adaptive Ln-Exp softmin

DG-ALETSK replaces the standard product T-norm with the Adaptive Ln-Exp (ALE) softmin. The firing strength of rule \(r\) is computed directly from paper eq. 22:

The adaptive exponent \(\hat{q}\) is recomputed on every forward pass: it is not a learned parameter. Choosing \(\xi = 700\) guarantees \(\exp(\hat{q}\cdot\max_d\tilde{\mu}) = e^{-700} > 0\) in IEEE 754 double precision (underflow boundary \(\approx e^{-745}\)), preventing numerical underflow while driving the softmin to closely approximate \(\min_d \tilde{\mu}_{r,d}\). Because \(\hat{q} \le -700\) and \(\tilde{\mu} \in (0,1]\), the output satisfies \(f_r \in (0, 1]\) by construction.

Rule gates and consequents

DG-ALETSK also embeds one gate per rule in the consequent. For zero-order classification, each rule \(r\) produces gated class logits:

For regression, the same gate multiplies a scalar rule output.

Output aggregation

Normalized rule strengths are computed as:

The final model output is the weighted sum of gated rule consequents:

Threshold search and pruning

Gate thresholds are computed from the learned gate values and two coefficients \(\zeta_{\lambda}\) and \(\zeta_{\theta}\):

Features and rules with gate values below these thresholds are pruned.

Code ↔ Paper Correspondence

Fidelity to the source article

The reproduction runs the protocol of the article and prints the result next to its Table V.

The classifier follows the article on the points below. On the SRBCT data (83 samples, 2308 genes), with the protocol of the article, the defaults give an accuracy of 0.949 with about 13 genes and 4 rules, against 0.950, 15.6 and 10.1 in the article.

Remaining differences:

• Threshold search. The article scores the candidate thresholds on a validation set of 10% of the training samples and refits the consequents by least squares for each candidate. highFIS scores them on the validation set passed to fit, or on the training data when there is none, keeps the first of the candidates that tie, and does not refit by default (use_lse=False). On samples as small as Colon and SRBCT, a 10% validation set and the least-squares refit both gave worse results when tried.
• Start of fine-tuning. The article resets the constant terms to the labels of the extracted rules; highFIS carries over the constant terms trained in the gate phase.
• Loss scale. The article divides the summed squared error by 2N; highFIS uses MSELoss, which also divides by the number of classes. With the Adam optimizer this family uses, the scale has little effect.
• The regressor is an extension. The article only treats classification. DGALETSKRegressor shares the point-based rule base, the spreads and the rule cap of the classifier, starts each rule from the target of its sample, and does not refit by least squares by default, because that refit diverges on a point-based rule base. On the Friedman-1 problem it selects informative features but stays below ridge regression (R² about 0.50 against 0.62); treat it as experimental.

Implementation notes

• DGALETSKClassifier now defaults to the paper profile for classification: optimizer_type='adam' (paper Section IV), rule_base='pfrb', dg_epochs=10, finetune_epochs=50, learning_rate=0.01, freeze_antecedents_finetune=False (the fine-tune phase optimises all parameters, including MF centres and spreads), zeta_lambda=zeta_theta=[0.05, 0.1, 0.15, 0.2, 0.25, 0.3], and use_lse=False.
• optimizer_type accepts 'adam' (default, paper), 'sgd', and 'adamw'; any other value raises ValueError (previously 'adam' silently became AdamW).
• For P-FRB classification, zero-order consequent biases are initialised from class labels before DG training (one-hot per sampled rule point, Eq. 25), and this label-initialised bias is carried over to the first-order consequent when converting for fine-tuning (rather than being reset to zero). This is essential to avoid collapse to the majority class on high-dimensional data.
• use_en_frb=True starts from an enhanced rule base (en FRB), but the paper's point-based FRB (P-FRB) is not constructed by default.
• The DG-ALETSK paper justifies P-FRB as a way to initialize an abundant candidate rule base from training samples, enabling the gate-based DG phase to perform rule extraction and feature selection in tandem.
• Estimator wrappers support rule_base='pfrb', which builds a point-based FRB from training samples and uses a CoCo rule base over the resulting sample-centered Gaussian MFs. Use pfrb_max_rules to cap the number of sample-based rules when the training set is large.
• When pfrb_max_rules=None in DGALETSKClassifier, the default cap follows the paper's high-dimensional policy: 100 rules (or 50 when D >= 10000).
• DGALETSKClassifier and DGALETSKRegressor train a zero-order model in fit_dg_phase(). The recommended Phase 2 workflow differs by task:
  • Classification (use_lse=False): search_thresholds evaluates the zero-order model on the validation set directly. Call fit_finetune afterwards to convert to first-order and train. By default (freeze_antecedents_finetune=False) the fine-tune phase optimises all system parameters — MF centres and spreads plus the consequents — as in the paper; only the feature-selection gates (λ) stay frozen.
  • Regression (use_lse=True): search_thresholds prunes gates and fits first-order consequents via least squares in one step. The LSE result is the final model — do not call fit_finetune afterwards, as it would reset the LSE-fitted weights and retrain from zero.
• By default (freeze_antecedents_finetune=False), fit_finetune trains the MF parameters (centres and spreads) together with the consequents, matching the DG-ALETSK paper: "In the FT phase, all of the system parameters including centers, spreads and consequent parameters are optimized." The feature-selection gates (λ) are kept frozen. Set freeze_antecedents_finetune=True to additionally freeze the MFs.
• Loss function: the paper uses MSE throughout, including for classification (applied to one-hot targets). highFIS follows that paper contract in the DG path: classification uses MSE with one-hot targets and regression uses MSE on scalar targets.
• The feature gate uses ExpGate(k=10) (\(M(\lambda)=1-e^{-10\lambda^2}\), paper eq. 24), the enhanced gate function introduced in DG-ALETSK. Gate values are applied to antecedent memberships as \(\mu^{M(\lambda_d)}\) (paper eq. 12), i.e., exponential gating.
• Threshold search is implemented by deep-copying the current model, pruning candidate copies, optionally refitting first-order consequents via least squares, and selecting the best validation score.

highFIS API summary

• fit_dg_phase(x, y, **kwargs) — train the DG-ALETSK zero-order model.
• convert_to_first_order() — convert the trained zero-order consequent to a gated first-order consequent.
• compute_thresholds(zeta_lambda, zeta_theta) — compute pruning thresholds from current gate activations.
• apply_thresholds(tau_lambda, tau_theta) — prune low-value gates.
• search_thresholds(x, y, *, zeta_lambda, zeta_theta, x_val, y_val, use_lse, inplace, ...) — grid-search over (zeta_lambda, zeta_theta) pairs and select the best gate thresholds by validation score. When use_lse=True, each candidate also fits first-order consequents via least squares before scoring (recommended for regression). When use_lse=False, the zero-order model is scored directly (recommended for classification). With inplace=True, the winning thresholds are applied to self.
• fit_finetune(x, y, **kwargs) — convert the pruned zero-order model to first-order (carrying over the label-initialised consequent bias), reset the linear consequent weights to zero, and retrain. By default the MF centres and spreads are trained together with the consequents (only the λ-gates stay frozen); pass freeze_antecedents_finetune=True to freeze the MFs. Call this only after search_thresholds(use_lse=False) (classification path). Do not call it after search_thresholds(use_lse=True), which already produces final first-order consequents via LSE.

Examples

Classification

For classification, Phase 2 uses use_lse=False so that threshold candidates are ranked by the zero-order model's accuracy on the validation set. After selecting the best thresholds, fit_finetune converts the model to first-order and retrains the consequents together with the MF centres and spreads (only the feature gates are kept frozen).

from highfis.models import DGALETSKClassifierModel
from highfis.memberships import GaussianMF

input_mfs = {
    "x1": [GaussianMF(mean=-1.0, sigma=1.0), GaussianMF(mean=1.0, sigma=1.0)],
    "x2": [GaussianMF(mean=-1.0, sigma=1.0), GaussianMF(mean=1.0, sigma=1.0)],
}

model = DGALETSKClassifierModel(
    input_mfs,
    n_classes=2,
    use_en_frb=False,
)

history = model.fit_dg_phase(X_train, y_train, epochs=30, learning_rate=1e-3)

# Phase 2a: select thresholds by evaluating the zero-order model.
result = model.search_thresholds(
    X_train,
    y_train,
    zeta_lambda=[0.0, 0.25, 0.5, 0.75, 1.0],
    zeta_theta=[0.0, 0.25, 0.5],
    x_val=X_val,
    y_val=y_val,
    use_lse=False,   # evaluate zero-order quality; do not fit first-order here
    inplace=True,
)
print(result)

# Phase 2b: convert to first-order and fine-tune (MFs + consequents trained; λ-gates frozen).
model.fit_finetune(X_train, y_train, epochs=60, learning_rate=1e-3)

Regression

For regression, Phase 2 uses use_lse=True so that each threshold candidate is evaluated after fitting first-order consequents via least squares. The best candidate's LSE-fitted model is the final result — do not call fit_finetune, which would reset those weights.

from highfis.models import DGALETSKRegressorModel
from highfis.memberships import GaussianMF

input_mfs = {
    "x1": [GaussianMF(mean=-1.0, sigma=1.0), GaussianMF(mean=1.0, sigma=1.0)],
    "x2": [GaussianMF(mean=-1.0, sigma=1.0), GaussianMF(mean=1.0, sigma=1.0)],
}

model = DGALETSKRegressorModel(
    input_mfs,
    use_en_frb=False,
)

history = model.fit_dg_phase(X_train, y_train, epochs=60, learning_rate=1e-3)

# Phase 2: select thresholds and fit first-order consequents via LSE in one step.
result = model.search_thresholds(
    X_train,
    y_train,
    zeta_lambda=[0.0, 0.25, 0.5, 0.75, 1.0],
    zeta_theta=[0.0, 0.25, 0.5],
    x_val=X_val,
    y_val=y_val,
    use_lse=True,    # fit first-order via LSE; this IS the final model
    inplace=True,
)
print(result)
# No fit_finetune here — LSE consequents are already optimal.