Classifier performance on subset of data

Question

I'm using Weka to perform classification on a set of labelled web pages, and measuring classifier performance with AUC. I have a separate six-level factor that is not used in classification, and I'd like to know how well classifiers perform on each level of the factor.

What techniques or measures should I use to test classifier performance on a subset of data?

Answer 1

I'm not sure if this is exactly what you are asking, but people often use cross-validation to break a single set of data into multiple training/testing subsets to better evaluate learning performance.

The basic idea (for e.g. 10-fold cross-validation) is to:

1. randomly split your data into training and testing sets
2. train a classifier on the training set
3. evaluate its performance on the testing set
4. repeat steps 1-3 nine more times with different random training/testing splits

The overall performance of the classifier is its average performance on all 10 testing sets.

I looked around a bit and found some examples of how to perform cross-validation programmatically or via the Weka UI.

Answer 2

The steps that Nate Kohl recommended are all correct. Another, very important question is the function for measuring the performance. In my experience, maximizing the AUC can sometimes lead to substantial bias of the classifier. I prefer using Matthews Correlation Coeficient (MCC) for binary classifiers or Cohen's kappa for categorical classifiers with more than two possible values