lemga::LPBoost

LPBoost Class Reference

LPBoost (Linear-Programming Boosting). More...

#include <lpboost.h>

Inheritance diagram for LPBoost:

Inheritance graph
[legend]
Collaboration diagram for LPBoost:

Collaboration graph
[legend]
List of all members.

Public Member Functions

 LPBoost ()
 LPBoost (const Boosting &s)
 LPBoost (std::istream &is)
virtual const id_t & id () const
virtual LPBoost * create () const
 Create a new object using the default constructor.
virtual LPBoost * clone () const
 Create a new object by replicating itself.
virtual bool set_aggregation_size (UINT)
virtual void train ()
 Train with preset data set and sample weight.
REAL C () const
 The regularization constant C.
void set_C (REAL _C)
 Set the regularization constant C.

Detailed Description

LPBoost (Linear-Programming Boosting).

With a similar idea to the original LPBoost [1], which solves

\begin{eqnarray*} \min && -\rho + D \sum_i \xi_i \\ \textrm{s.t.}&& y_i\left(\sum_t\alpha_t h_t(x_i)\right)\ge \rho-\xi_i, \\ && \xi_i\ge 0, \quad \alpha_t\ge 0, \quad \sum_t\alpha_t=1. \end{eqnarray*}

we instead implement the algorithm to solve

\begin{eqnarray*} \min && \sum_t \alpha_t + C \sum_i \xi_i \\ \textrm{s.t.}&& y_i \left(\sum_t \alpha_t h_t(x_i)\right) \ge 1 - \xi_i,\\ && \xi_i \ge 0, \quad \alpha_t \ge 0. \end{eqnarray*}

by column generation. Note that the dual problem is

\begin{eqnarray*} \max && \sum_i u_i \\ \textrm{s.t.}&& \sum_i u_i y_i h_t(x_i) \le 1, \qquad (*)\\ && 0 \le u_i \le C. \end{eqnarray*}

Column generation corresponds to generating the constraints (*). We actually use individual upper bound $C_i$ proportional to example's initial weight.

If we treat $w_i$, the normalized version of $u_i$, as the sample weight, and $\Sigma_u = \sum_i u_i$ as the normalization constraint, (*) is the same as

\[ \Sigma_u (1 - 2 e(h_t, w)) \le 1, \]

which means

\[ e(h_t, w) \ge \frac12 (1 - \Sigma_u^{-1}).\qquad (**) \]

Assume that we have found $h_1, \dots, h_T$ so far, solving the dual problem with $T$ (*) constraints gives us $\Sigma_u$. If for every remaining $h$ in $\cal{H}$, $ e(h, w) \ge \frac12 (1 - \Sigma_u^{-1}),$ the duality condition tells us that even if we set $\alpha=0$ for those remaining $h$, the solution is still optimal. Thus, we can train the weak learner with sample weight $w$ in each iteration, and terminate if the best hypothesis has satisfied (**).

[1] A. Demiriz, K. P. Bennett, and J. Shawe-Taylor. Linear programming boosting via column generation. Machine Learning, 46(1-3):225-254, 2002.

Definition at line 59 of file lpboost.h.

Constructor & Destructor Documentation

LPBoost (   )  [inline, explicit]
 

Definition at line 63 of file lpboost.h.

References LPBoost::set_C().

Referenced by LPBoost::clone(), and LPBoost::create().

LPBoost (  const Boosting &  s  )  [inline]
 

Definition at line 64 of file lpboost.h.

References Boosting::convex, and LPBoost::set_C().

LPBoost (  std::istream &  is  )  [inline, explicit]
 

Definition at line 65 of file lpboost.h.

Member Function Documentation

REAL C (   )  const [inline]
 

The regularization constant C.

Definition at line 76 of file lpboost.h.

virtual LPBoost* clone (   )  const [inline, virtual]
 

Create a new object by replicating itself.

Returns:
A pointer to the new copy.
The code for a derived class Derived is always 
 return new Derived(*this);
Though seemingly redundant, it helps to copy an object without knowing the real type of the object.
See also:
C++ FAQ Lite 20.6

Reimplemented from Boosting.

Definition at line 69 of file lpboost.h.

References LPBoost::LPBoost().

virtual LPBoost* create (   )  const [inline, virtual]
 

Create a new object using the default constructor.

The code for a derived class Derived is always 

 return new Derived();

Reimplemented from Boosting.

Definition at line 68 of file lpboost.h.

References LPBoost::LPBoost().

virtual const id_t& id (   )  const [virtual]
 

Returns:
Class ID string (class name)

Reimplemented from Boosting.

virtual bool set_aggregation_size (  UINT   )  [inline, virtual]
 

Todo:
Implement something similar to that in CGBoost.

Reimplemented from Aggregating.

Definition at line 72 of file lpboost.h.

void set_C (  REAL  _C  )  [inline]
 

Set the regularization constant C.

Definition at line 78 of file lpboost.h.

Referenced by LPBoost::LPBoost().

void train (   )  [virtual]
 

Train with preset data set and sample weight.

Reimplemented from Boosting.

Definition at line 66 of file lpboost.cpp.

References Boosting::grad_desc_view, Aggregating::lm_base, LearnModel::n_samples, LearnModel::ptd, LearnModel::ptw, LearnModel::set_dimensions(), and U.

The documentation for this class was generated from the following files:
Generated on Wed Nov 8 08:17:00 2006 for LEMGA by  doxygen 1.4.6