weka.attributeSelection

## Class ClassifierSubsetEval

- java.lang.Object
- - weka.attributeSelection.ASEvaluation
  - - weka.attributeSelection.HoldOutSubsetEvaluator
    - - weka.attributeSelection.ClassifierSubsetEval

- All Implemented Interfaces:
    java.io.Serializable, ErrorBasedMeritEvaluator, SubsetEvaluator, CapabilitiesHandler, CapabilitiesIgnorer, CommandlineRunnable, OptionHandler, RevisionHandler

      public class ClassifierSubsetEval
      extends HoldOutSubsetEvaluator
      implements OptionHandler, ErrorBasedMeritEvaluator

  Classifier subset evaluator:
  Evaluates attribute subsets on training data or a seperate hold out testing set. Uses a classifier to estimate the 'merit' of a set of attributes.
  Valid options are:

       -B <classifier>
        class name of the classifier to use for accuracy estimation.
        Place any classifier options LAST on the command line
        following a "--". eg.:
         -B weka.classifiers.bayes.NaiveBayes ... -- -K
        (default: weka.classifiers.rules.ZeroR)

       -T
        Use the training data to estimate accuracy.

       -H <filename>
        Name of the hold out/test set to
        estimate accuracy on.

       -percentage-split
        Perform a percentage split on the training data.
        Use in conjunction with -T.

       -P
        Split percentage to use (default = 90).

       -S
        Random seed for percentage split (default = 1).

       -E <DEFAULT|ACC|RMSE|MAE|F-MEAS|AUC|AUPRC|CORR-COEFF>
        Performance evaluation measure to use for selecting attributes.
        (Default = default: accuracy for discrete class and rmse for numeric class)

       -IRclass <label | index>
        Optional class value (label or 1-based index) to use in conjunction with
        IR statistics (f-meas, auc or auprc). Omitting this option will use
        the class-weighted average.

       Options specific to scheme weka.classifiers.rules.ZeroR:

       -output-debug-info
        If set, classifier is run in debug mode and
        may output additional info to the console

       -do-not-check-capabilities
        If set, classifier capabilities are not checked before classifier is built
        (use with caution).

       -num-decimal-places
        The number of decimal places for the output of numbers in the model (default 2).

       -batch-size
        The desired batch size for batch prediction  (default 100).

  Version:
    $Revision: 10332 $
  Author:
    Mark Hall (mhall@cs.waikato.ac.nz)
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `EVAL_ACCURACY`
    `static int` | `EVAL_AUC`
    `static int` | `EVAL_AUPRC`
    `static int` | `EVAL_CORRELATION`
    `static int` | `EVAL_DEFAULT`
    `static int` | `EVAL_FMEASURE`
    `static int` | `EVAL_MAE`
    `static int` | `EVAL_PLUGIN`
    `static int` | `EVAL_RMSE`
    `static Tag[]` | `TAGS_EVALUATION` Holds all tags for metrics

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `ClassifierSubsetEval()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `buildEvaluator(Instances data)` Generates a attribute evaluator.
    `java.lang.String` | `classifierTipText()` Returns the tip text for this property
    `double` | `evaluateSubset(java.util.BitSet subset)` Evaluates a subset of attributes
    `double` | `evaluateSubset(java.util.BitSet subset, Instance holdOut, boolean retrain)` Evaluates a subset of attributes with respect to a single instance.
    `double` | `evaluateSubset(java.util.BitSet subset, Instances holdOut)` Evaluates a subset of attributes with respect to a set of instances.
    `java.lang.String` | `evaluationMeasureTipText()` Returns the tip text for this property
    `Capabilities` | `getCapabilities()` Returns the capabilities of this evaluator.
    `Classifier` | `getClassifier()` Get the classifier used as the base learner.
    `SelectedTag` | `getEvaluationMeasure()` Gets the currently set performance evaluation measure used for selecting attributes for the decision table
    `java.io.File` | `getHoldOutFile()` Gets the file that holds hold out/test instances.
    `java.lang.String` | `getIRClassValue()` Get the class value (label or index) to use with IR metric evaluation of subsets.
    `java.lang.String[]` | `getOptions()` Gets the current settings of ClassifierSubsetEval
    `java.lang.String` | `getRevision()` Returns the revision string.
    `int` | `getSeed()` Get the random seed used to randomize the data before performing a percentage split
    `java.lang.String` | `getSplitPercent()` Get the split percentage to use
    `boolean` | `getUsePercentageSplit()` Get whether to perform a percentage split on the training data for evaluation
    `boolean` | `getUseTraining()` Get if training data is to be used instead of hold out/test data
    `java.lang.String` | `globalInfo()` Returns a string describing this attribute evaluator
    `java.lang.String` | `holdOutFileTipText()` Returns the tip text for this property
    `java.lang.String` | `IRClassValueTipText()` Returns the tip text for this property
    `java.util.Enumeration<Option>` | `listOptions()` Returns an enumeration describing the available options.
    `static void` | `main(java.lang.String[] args)` Main method for testing this class.
    `java.lang.String` | `seedTipText()` Returns the tip text for this property
    `void` | `setClassifier(Classifier newClassifier)` Set the classifier to use for accuracy estimation
    `void` | `setEvaluationMeasure(SelectedTag newMethod)` Sets the performance evaluation measure to use for selecting attributes for the decision table
    `void` | `setHoldOutFile(java.io.File h)` Set the file that contains hold out/test instances
    `void` | `setIRClassValue(java.lang.String val)` Set the class value (label or index) to use with IR metric evaluation of subsets.
    `void` | `setOptions(java.lang.String[] options)` Parses a given list of options.
    `void` | `setSeed(int s)` Set the random seed used to randomize the data before performing a percentage split
    `void` | `setSplitPercent(java.lang.String sp)` Set the split percentage to use
    `void` | `setUsePercentageSplit(boolean p)` Set whether to perform a percentage split on the training data for evaluation
    `void` | `setUseTraining(boolean t)` Set if training data is to be used instead of hold out/test data
    `java.lang.String` | `splitPercentTipText()` Returns the tip text for this property
    `java.lang.String` | `toString()` Returns a string describing classifierSubsetEval
    `java.lang.String` | `usePercentageSplitTipText()` Returns the tip text for this property
    `java.lang.String` | `useTrainingTipText()` Returns the tip text for this property

    - ### Methods inherited from class weka.attributeSelection.ASEvaluation

      `clean, doNotCheckCapabilitiesTipText, forName, getDoNotCheckCapabilities, makeCopies, postExecution, postProcess, preExecution, run, runEvaluator, setDoNotCheckCapabilities`

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### EVAL_DEFAULT

          public static final int EVAL_DEFAULT

      See Also:
        Constant Field Values

    - #### EVAL_ACCURACY

          public static final int EVAL_ACCURACY

      See Also:
        Constant Field Values

    - #### EVAL_RMSE

          public static final int EVAL_RMSE

      See Also:
        Constant Field Values

    - #### EVAL_MAE

          public static final int EVAL_MAE

      See Also:
        Constant Field Values

    - #### EVAL_FMEASURE

          public static final int EVAL_FMEASURE

      See Also:
        Constant Field Values

    - #### EVAL_AUC

          public static final int EVAL_AUC

      See Also:
        Constant Field Values

    - #### EVAL_AUPRC

          public static final int EVAL_AUPRC

      See Also:
        Constant Field Values

    - #### EVAL_CORRELATION

          public static final int EVAL_CORRELATION

      See Also:
        Constant Field Values

    - #### EVAL_PLUGIN

          public static final int EVAL_PLUGIN

      See Also:
        Constant Field Values

    - #### TAGS_EVALUATION

          public static final Tag[] TAGS_EVALUATION

      Holds all tags for metrics

  - ### Constructor Detail

    - #### ClassifierSubsetEval

          public ClassifierSubsetEval()

  - ### Method Detail

    - #### globalInfo

          public java.lang.String globalInfo()

      Returns a string describing this attribute evaluator
      Returns:
        a description of the evaluator suitable for displaying in the explorer/experimenter gui

    - #### listOptions

          public java.util.Enumeration<Option> listOptions()

      Returns an enumeration describing the available options.
      Specified by:
        `listOptions` in interface `OptionHandler`
      Returns:
        an enumeration of all the available options.

    - #### setOptions

          public void setOptions(java.lang.String[] options)
                          throws java.lang.Exception

      Parses a given list of options.

      Valid options are:

           -B <classifier>
            class name of the classifier to use for accuracy estimation.
            Place any classifier options LAST on the command line
            following a "--". eg.:
             -B weka.classifiers.bayes.NaiveBayes ... -- -K
            (default: weka.classifiers.rules.ZeroR)

           -T
            Use the training data to estimate accuracy.

           -H <filename>
            Name of the hold out/test set to
            estimate accuracy on.

           -percentage-split
            Perform a percentage split on the training data.
            Use in conjunction with -T.

           -P
            Split percentage to use (default = 90).

           -S
            Random seed for percentage split (default = 1).

           -E <DEFAULT|ACC|RMSE|MAE|F-MEAS|AUC|AUPRC|CORR-COEFF>
            Performance evaluation measure to use for selecting attributes.
            (Default = default: accuracy for discrete class and rmse for numeric class)

           -IRclass <label | index>
            Optional class value (label or 1-based index) to use in conjunction with
            IR statistics (f-meas, auc or auprc). Omitting this option will use
            the class-weighted average.

           Options specific to scheme weka.classifiers.rules.ZeroR:

           -output-debug-info
            If set, classifier is run in debug mode and
            may output additional info to the console

           -do-not-check-capabilities
            If set, classifier capabilities are not checked before classifier is built
            (use with caution).

           -num-decimal-places
            The number of decimal places for the output of numbers in the model (default 2).

           -batch-size
            The desired batch size for batch prediction  (default 100).

      Specified by:
        `setOptions` in interface `OptionHandler`
      Parameters:
        `options` - the list of options as an array of strings
      Throws:
        `java.lang.Exception` - if an option is not supported

    - #### seedTipText

          public java.lang.String seedTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### setSeed

          public void setSeed(int s)

      Set the random seed used to randomize the data before performing a percentage split
      Parameters:
        `s` - the seed to use

    - #### getSeed

          public int getSeed()

      Get the random seed used to randomize the data before performing a percentage split
      Returns:
        the seed to use

    - #### usePercentageSplitTipText

          public java.lang.String usePercentageSplitTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### setUsePercentageSplit

          public void setUsePercentageSplit(boolean p)

      Set whether to perform a percentage split on the training data for evaluation
      Parameters:
        `p` - true if a percentage split is to be performed

    - #### getUsePercentageSplit

          public boolean getUsePercentageSplit()

      Get whether to perform a percentage split on the training data for evaluation
      Returns:
        true if a percentage split is to be performed

    - #### splitPercentTipText

          public java.lang.String splitPercentTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### setSplitPercent

          public void setSplitPercent(java.lang.String sp)

      Set the split percentage to use
      Parameters:
        `sp` - the split percentage to use

    - #### getSplitPercent

          public java.lang.String getSplitPercent()

      Get the split percentage to use
      Returns:
        the split percentage to use

    - #### setIRClassValue

          public void setIRClassValue(java.lang.String val)

      Set the class value (label or index) to use with IR metric evaluation of subsets. Leaving this unset will result in the class weighted average for the IR metric being used.
      Parameters:
        `val` - the class label or 1-based index of the class label to use when evaluating subsets with an IR metric

    - #### getIRClassValue

          public java.lang.String getIRClassValue()

      Get the class value (label or index) to use with IR metric evaluation of subsets. Leaving this unset will result in the class weighted average for the IR metric being used.
      Returns:
        the class label or 1-based index of the class label to use when evaluating subsets with an IR metric

    - #### IRClassValueTipText

          public java.lang.String IRClassValueTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### evaluationMeasureTipText

          public java.lang.String evaluationMeasureTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### getEvaluationMeasure

          public SelectedTag getEvaluationMeasure()

      Gets the currently set performance evaluation measure used for selecting attributes for the decision table
      Returns:
        the performance evaluation measure

    - #### setEvaluationMeasure

          public void setEvaluationMeasure(SelectedTag newMethod)

      Sets the performance evaluation measure to use for selecting attributes for the decision table
      Parameters:
        `newMethod` - the new performance evaluation metric to use

    - #### classifierTipText

          public java.lang.String classifierTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### setClassifier

          public void setClassifier(Classifier newClassifier)

      Set the classifier to use for accuracy estimation
      Parameters:
        `newClassifier` - the Classifier to use.

    - #### getClassifier

          public Classifier getClassifier()

      Get the classifier used as the base learner.
      Returns:
        the classifier used as the classifier

    - #### holdOutFileTipText

          public java.lang.String holdOutFileTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### getHoldOutFile

          public java.io.File getHoldOutFile()

      Gets the file that holds hold out/test instances.
      Returns:
        File that contains hold out instances

    - #### setHoldOutFile

          public void setHoldOutFile(java.io.File h)

      Set the file that contains hold out/test instances
      Parameters:
        `h` - the hold out file

    - #### useTrainingTipText

          public java.lang.String useTrainingTipText()

      Returns the tip text for this property
      Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

    - #### getUseTraining

          public boolean getUseTraining()

      Get if training data is to be used instead of hold out/test data
      Returns:
        true if training data is to be used instead of hold out data

    - #### setUseTraining

          public void setUseTraining(boolean t)

      Set if training data is to be used instead of hold out/test data
      Parameters:
        `t` - true if training data is to be used instead of hold out data

    - #### getOptions

          public java.lang.String[] getOptions()

      Gets the current settings of ClassifierSubsetEval
      Specified by:
        `getOptions` in interface `OptionHandler`
      Returns:
        an array of strings suitable for passing to setOptions()

    - #### getCapabilities

          public Capabilities getCapabilities()

      Returns the capabilities of this evaluator.
      Specified by:
        `getCapabilities` in interface `CapabilitiesHandler`
      Overrides:
        `getCapabilities` in class `ASEvaluation`
      Returns:
        the capabilities of this evaluator
      See Also:
        `Capabilities`

    - #### buildEvaluator

          public void buildEvaluator(Instances data)
                              throws java.lang.Exception

      Generates a attribute evaluator. Has to initialize all fields of the evaluator that are not being set via options.
      Specified by:
        `buildEvaluator` in class `ASEvaluation`
      Parameters:
        `data` - set of instances serving as training data
      Throws:
        `java.lang.Exception` - if the evaluator has not been generated successfully

    - #### evaluateSubset

          public double evaluateSubset(java.util.BitSet subset)
                                throws java.lang.Exception

      Evaluates a subset of attributes
      Specified by:
        `evaluateSubset` in interface `SubsetEvaluator`
      Parameters:
        `subset` - a bitset representing the attribute subset to be evaluated
      Returns:
        the error rate
      Throws:
        `java.lang.Exception` - if the subset could not be evaluated

    - #### evaluateSubset

          public double evaluateSubset(java.util.BitSet subset,
                                       Instances holdOut)
                                throws java.lang.Exception

      Evaluates a subset of attributes with respect to a set of instances. Calling this function overrides any test/hold out instances set from setHoldOutFile.
      Specified by:
        `evaluateSubset` in class `HoldOutSubsetEvaluator`
      Parameters:
        `subset` - a bitset representing the attribute subset to be evaluated
        `holdOut` - a set of instances (possibly separate and distinct from those use to build/train the evaluator) with which to evaluate the merit of the subset
      Returns:
        the "merit" of the subset on the holdOut data
      Throws:
        `java.lang.Exception` - if the subset cannot be evaluated

    - #### evaluateSubset

          public double evaluateSubset(java.util.BitSet subset,
                                       Instance holdOut,
                                       boolean retrain)
                                throws java.lang.Exception

      Evaluates a subset of attributes with respect to a single instance. Calling this function overides any hold out/test instances set through setHoldOutFile.
      Specified by:
        `evaluateSubset` in class `HoldOutSubsetEvaluator`
      Parameters:
        `subset` - a bitset representing the attribute subset to be evaluated
        `holdOut` - a single instance (possibly not one of those used to build/train the evaluator) with which to evaluate the merit of the subset
        `retrain` - true if the classifier should be retrained with respect to the new subset before testing on the holdOut instance.
      Returns:
        the "merit" of the subset on the holdOut instance
      Throws:
        `java.lang.Exception` - if the subset cannot be evaluated

    - #### toString

          public java.lang.String toString()

      Returns a string describing classifierSubsetEval
      Overrides:
        `toString` in class `java.lang.Object`
      Returns:
        the description as a string

    - #### getRevision

          public java.lang.String getRevision()

      Returns the revision string.
      Specified by:
        `getRevision` in interface `RevisionHandler`
      Overrides:
        `getRevision` in class `ASEvaluation`
      Returns:
        the revision

    - #### main

          public static void main(java.lang.String[] args)

      Main method for testing this class.
      Parameters:
        `args` - the options