FSRE-AdaTSK

Article: G. Xue, Q. Chang, J. Wang, K. Zhang and N. R. Pal, "An adaptive neuro-fuzzy system with integrated feature selection and rule extraction for high-dimensional classification problems", IEEE Transactions on Fuzzy Systems, vol. 31, no. 7, pp. 2167-2181, 2023.

Script: examples/reproductions/fsre_adatsk_2022.py

The same article proposes AdaTSK, the classifier without feature selection, which has its own reproduction.

Protocol of the Article

Section IV-C of the article, for its Table IV.

Result

Output of the script, on one thread:

Table IV: FSRE-AdaTSK (accuracy in percent / selected features / extracted rules)
std of accuracy: over the runs, and over the means of the repetitions
dataset                 article              highFIS  std of accuracy
Iris           96.5 / 2.1 / 6.1  96.40 / 2.00 / 6.20      4.46 / 0.33
Appendicitis   86.3 / 3.7 / 3.1  84.96 / 3.14 / 3.24      8.40 / 1.34
Pima           75.7 / 3.0 / 3.2  76.30 / 3.40 / 3.28      4.45 / 0.30
Glass          65.5 / 4.1 / 6.0  65.35 / 5.66 / 6.00      9.71 / 0.92
Wine           97.3 / 6.3 / 6.3  96.18 / 8.22 / 6.80      5.80 / 1.11
Heart          82.5 / 7.4 / 2.9  82.07 / 8.88 / 2.96      6.84 / 2.51
Wdbc           95.4 / 6.2 / 4.9  95.26 / 7.12 / 4.36      2.61 / 0.37
Spectfheart    79.2 / 3.4 / 5.1  77.17 / 5.40 / 5.70      6.81 / 1.80

Summary of the last model fitted
FSREADATSKClassifier
  task               classification, 2 classes
  features           44 seen in fit, 3 used by the model
  rules              5 (rule base: coco)
  fuzzy sets         5 per feature, ADATSKGaussianMF
  firing strength    AdaSoftminRuleLayer
  defuzzifier        SoftmaxLogDefuzzifier
  consequents        GatedClassificationConsequentLayer
  parameters         63 trainable (15 antecedent, 48 consequent)
Training
  batch size         full batch
  feature selection  1000 epochs, learning rate 0.05, training loss 1.0000 -> 0.1899, training accuracy 0.8714
  rule extraction    600 epochs, learning rate 0.05, training loss 1.0000 -> 0.2915, training accuracy 0.8216
  fine-tuning        1000 epochs, learning rate 0.05, training loss 0.5190 -> 0.2293, training accuracy 0.8216

The table has eight of the twelve low-dimensional datasets of the article. Seven accuracies are within 1.4 points of the article and Spectfheart is two points below, in every case a fraction of the standard deviation between runs. The numbers of extracted rules are reproduced on the eight datasets, within 0.6 of a rule. The number of selected features is the article's on Iris, Appendicitis and Pima, and one or two above it on the others.

The standard deviations of the article are 0.4 to 1.5 points on these datasets, the size of a deviation between the means of the five repetitions and not between the 50 runs. The second value of the last column is that quantity: 0.3 to 2.5 points.

Scale of the Inputs

The article does not say how the inputs are scaled, and the choice decides the number of rules. Its membership function has no spread, so the scale of a feature sets how far apart its fuzzy sets are. With five sets on a feature scaled to \([0, 1]\), the centre of a set has a degree of 0.94 in the neighbouring set, the rules of the enhanced rule base fire almost alike, and their gates open together. On standardized inputs the same degree is typically between 0.1 and 0.4, and the gates of a few rules open well before the others.

The same protocol with the inputs scaled to \([0, 1]\), 300 iterations of rule extraction and 3000 of fine-tuning:

On that scale no length of the rule-extraction phase brought Wine below about fifteen rules.

Length of the Rule-Extraction Phase

The gates of the rules keep opening as the phase goes on, so the number of extracted rules grows with the number of iterations, which the article does not state. On standardized inputs, with the full protocol:

The value of 600 was chosen on these three datasets, as the one whose numbers of rules are closest to the article; the other five datasets were run with it afterwards, without further choice. The accuracy changes by half a point or less between the two.

Differences from the Article

• Scale of the inputs, learning rate and iterations. Not stated in the article; chosen here as described above.
• Defaults of the estimator. FSREADATSKClassifier defaults to three fuzzy sets in every phase and to normalization of the consequent inputs, a setting meant for inputs scaled to \([0, 1]\). With the defaults and that scale the same protocol gives 96.9 / 3.0 / 7.5 on Iris and 98.9 / 9.1 / 11.7 on Wine.
• Selection on the magnitude of the gate. The gate is an odd function; highFIS compares the magnitude of each gate with the threshold.
• Folds. The article does not say whether its folds are stratified; the script uses stratified folds. Glass has a class of nine samples, fewer than the ten folds, so some test parts have no sample of that class.
• Datasets. The article uses nineteen datasets, seven of them with more than 1000 features, for which it changes the threshold coefficients to 0.4 and 0.5 and keeps the centres fixed in the first two phases. Iris, Wine and Wdbc ship with scikit-learn; Appendicitis, Pima, Glass, Heart and Spectfheart come from OpenML (identifiers 1456, 37, 41, 53 and 1600) with the samples, features and classes of the article.

Reproducibility

The script fixes the seeds, uses deterministic algorithms and one thread, and ends by printing the versions of the libraries and the processor. Another processor or another version of the numerical libraries can change the last digits.

Run It

python examples/reproductions/fsre_adatsk_2022.py

Five datasets are downloaded from OpenML on the first run. A run takes about an hour and a half.