TWiki> Kernel Web>LearningKernels (revision 2)EditAttach

Automatic Kernel Selection

Here we give some examples showing how to automatically create custom kernels using data. The kernels are generally created from a certain combination of base kernels, which can be specified by the user.

The examples here will use the electonics sentiment analysis dataset, which we include with precomputed ngram features and binary as well as real-valued labels. The data is arranged in LIBSVM format, which is compatible with all the learning algorithms used in these examples. The results shown here should be easily reproducible and serve as a good first exercise.

Feature Weighted Kernels

The examples below consider the case when each base kernel corresponds to a single features. Such a set of base kernels occurs naturally when, for example, learning rational kernels as explained in Cortes et al. (MLSP 2008).

Correlation Kernel Example: The correlation based kernel weights each input feature by a quantity proportional to its correlation with the training labels. The following command will generate weighed features:

klweightfeatures  --weight_type=corr --features --sparse --num_train=1000 electronics.class 0.1
  > electronics.class.corr

The --features flag forces the output of explicit feature vectors, rather than the kernel matrix, and the --sparse flag forces the use of sparse data-structure, which are both desirable in this case since the ngram-features are sparse. The --num_train flag indicates that the kernel selection algorithm should use only the first 1000 data-points for training, and thus allows us to use the remaining points as a holdout set for evaluating performance. The --weight_type flag selects which type of kernel selection algorithm is used. The first argument indicates the input dataset and the second argument regularizes the kernel which, in the case of the correlation kernel, restricts the kernel trace to equal 0.1.

The weighted features can then be used to train and test an svm model via libsvm or liblinear:

Separate train and test:

head -n1000 electronics.class.corr > electronics.class.corr.train
tail -n+1001 electronics.class.corr > electronics.class.corr.test

Train:

svm-train ...

Test:

svm-predict ...

L2 Regularized Linear Combination: Here we optimally weight the input features in order to maximize the kernel ridge regression (KRR) objective, subject to the L2 regularization constraint: ||mu - mu0|| < Lambda||, where mu is the vector of squared weights and mu0 and Lambda are user specified arguments.

klweightfeatures  --weight_type=lin2 --features --sparse --num_train=1000 electronics.reg 1024
  > electronics.reg.lin2

We then train and test use (KRR), with input and output arguments that have been made to closely resemble libsvm. To see a full list of command line arguments, run krr-train without any parameters.

Separate train and test:

head -n1000 electronics.class.corr > electronics.class.corr.train
tail -n+1001 electronics.class.corr > electronics.class.corr.test

Train:

krr-train ...

Test:

krr-predict ...

Kernel Combinations w/ Explicit Features

Here we consider the case of combining several general base kernels that admit explicit feature mappings. In the case that these features are sparse, for example, we are able to very efficiently compute combinations in high dimensional features spaces.

In this example we find the best linear combination of 5 ngram kernels (1-gram, 2-gram, ..., 5-gram) with respect to the SVM objective.

klcombinefeatures ...

This will produce a kernels with many features, but which are sparse, thus liblinear is a good choice for training a model:

Separate train and test:

head -n1000 electronics.class.corr > electronics.class.corr.train
tail -n+1001 electronics.class.corr > electronics.class.corr.test

Train:

train ...

Test:

predict ...

General Kernel Combinations

-- AfshinRostamizadeh - 24 Aug 2009

Edit | Attach | Watch | Print version | History: r13 | r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r2 - 2009-08-24 - AfshinRostamizadeh
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback