lemga::_gd_momentum

_gd_momentum Struct Template Reference

Gradient descent with momentum. More...

#include <optimize.h>

Inheritance diagram for _gd_momentum:

Inheritance graph
[legend]
Collaboration diagram for _gd_momentum:

Collaboration graph
[legend]
List of all members.

Public Member Functions

 _gd_momentum (LM *lm, const Step &lr, const Step &m)
const Dir & direction ()
 Search direction at w.

Public Attributes

Step momentum

Detailed Description

template<class LM, class Dir, class Step>
struct lemga::_gd_momentum< LM, Dir, Step >

Gradient descent with momentum.

With momentum, the weight change is accumulated:

\[ \Delta w\leftarrow-\eta\frac{\partial E}{\partial w}+\mu\Delta w. \]

When the derivative is roughly constant, this speeds up the training by using an effectively larger learning rate

\[ \Delta w \approx -\frac{\eta}{1-\mu}\frac{\partial E}{\partial w}. \]

When the derivative somehow ``fluctuates'', the momentum stablizes the optimization to some exent by effectively decreasing the learning rate.

Definition at line 169 of file optimize.h.

Constructor & Destructor Documentation

_gd_momentum (  LM *  lm,
const Step &  lr,
const Step &  m
)  [inline]
 

Definition at line 172 of file optimize.h.

Member Function Documentation

const Dir& direction (   )  [inline]
 

Search direction at w.

Reimplemented from _gradient_descent.

Definition at line 175 of file optimize.h.

Member Data Documentation

Step momentum
 

Definition at line 170 of file optimize.h.

The documentation for this struct was generated from the following file:
Generated on Wed Nov 8 08:17:12 2006 for LEMGA by  doxygen 1.4.6