extreme learning machine for multi-categories...
TRANSCRIPT
tu-logo
ur-logo
Outline
Extreme Learning Machine for Multi-Categories Classification Applications
Hai-Jun Rong1,2, Guang-Bin Huang1 and Yew-Soon Ong2
1School of Electrical and Electronic Engineering
2School of Computer EngineeringNanyang Technological University
Nanyang Avenue, Singapore 639798E-mail: {hjrong, egbhuang, asysong}@ntu.edu.sg
IEEE World Congress on Computational IntelligenceHong Kong, June 1-6 2008
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Outline
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Outline
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Outline
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Outline
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Outline
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with Additive Nodes
Figure 1: Feedforward Network Architecture: additive hiddennodes
Output of hidden nodes
G(ai , bi , x) = g(ai · x + bi ) (1)
ai : the weight vector connecting the i th hiddennode and the input nodes.bi : the threshold of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (2)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with Additive Nodes
Figure 1: Feedforward Network Architecture: additive hiddennodes
Output of hidden nodes
G(ai , bi , x) = g(ai · x + bi ) (1)
ai : the weight vector connecting the i th hiddennode and the input nodes.bi : the threshold of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (2)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with Additive Nodes
Figure 1: Feedforward Network Architecture: additive hiddennodes
Output of hidden nodes
G(ai , bi , x) = g(ai · x + bi ) (1)
ai : the weight vector connecting the i th hiddennode and the input nodes.bi : the threshold of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (2)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with RBF Nodes
Figure 2: Feedforward Network Architecture: RBF hiddennodes
Output of hidden nodes
G(ai , bi , x) = g (bi‖x − ai‖) (3)
ai : the center of the i th hidden node.bi : the impact factor of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (4)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with RBF Nodes
Figure 2: Feedforward Network Architecture: RBF hiddennodes
Output of hidden nodes
G(ai , bi , x) = g (bi‖x − ai‖) (3)
ai : the center of the i th hidden node.bi : the impact factor of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (4)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Feedforward Neural Networks with RBF Nodes
Figure 2: Feedforward Network Architecture: RBF hiddennodes
Output of hidden nodes
G(ai , bi , x) = g (bi‖x − ai‖) (3)
ai : the center of the i th hidden node.bi : the impact factor of the i th hidden node.
Output of SLFNs
fL(x) =LX
i=1
βi G(ai , bi , x) (4)
βi : the weight vector connecting the i th hiddennode and the output nodes.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Function Approximation of Neural Networks
Figure 3: Feedforward Network Architecture.
Learning Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , SLFNs with Lhidden nodes and activation functiong(x) are mathematically modeled as
fL(xj ) = oj , j = 1, · · · , N (5)
Cost function: E =PN
j=1
‚‚‚oj − tj‚‚‚
2.
The target is to minimize the costfunction E by adjusting the networkparameters: βi , ai , bi .
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Function Approximation of Neural Networks
Figure 3: Feedforward Network Architecture.
Learning Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , SLFNs with Lhidden nodes and activation functiong(x) are mathematically modeled as
fL(xj ) = oj , j = 1, · · · , N (5)
Cost function: E =PN
j=1
‚‚‚oj − tj‚‚‚
2.
The target is to minimize the costfunction E by adjusting the networkparameters: βi , ai , bi .
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Function Approximation of Neural Networks
Figure 3: Feedforward Network Architecture.
Learning Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , SLFNs with Lhidden nodes and activation functiong(x) are mathematically modeled as
fL(xj ) = oj , j = 1, · · · , N (5)
Cost function: E =PN
j=1
‚‚‚oj − tj‚‚‚
2.
The target is to minimize the costfunction E by adjusting the networkparameters: βi , ai , bi .
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Function Approximation of Neural Networks
Figure 3: Feedforward Network Architecture.
Learning Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , SLFNs with Lhidden nodes and activation functiong(x) are mathematically modeled as
fL(xj ) = oj , j = 1, · · · , N (5)
Cost function: E =PN
j=1
‚‚‚oj − tj‚‚‚
2.
The target is to minimize the costfunction E by adjusting the networkparameters: βi , ai , bi .
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Learning Algorithms of Neural Networks
Figure 4: Feedforward Network Architecture.
Learning Methods
Many learning methodsmainly based ongradient-descent/iterativeapproaches have beendeveloped over the pasttwo decades.
Back-Propagation (BP)and its variants are mostpopular.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Learning Algorithms of Neural Networks
Figure 4: Feedforward Network Architecture.
Learning Methods
Many learning methodsmainly based ongradient-descent/iterativeapproaches have beendeveloped over the pasttwo decades.
Back-Propagation (BP)and its variants are mostpopular.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Learning Algorithms of Neural Networks
Figure 4: Feedforward Network Architecture.
Learning Methods
Many learning methodsmainly based ongradient-descent/iterativeapproaches have beendeveloped over the pasttwo decades.
Back-Propagation (BP)and its variants are mostpopular.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Advantagnes and Disadvantages
PopularityWidely used in various applications: regression,classification, etc.
LimitationsUsually different learning algorithms used in differentSLFNs architectures.Some parameters have to be tuned mannually.Overfitting.Local minima.Time-consuming.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
SLFN ModelsLearning Methods
Advantagnes and Disadvantages
PopularityWidely used in various applications: regression,classification, etc.
LimitationsUsually different learning algorithms used in differentSLFNs architectures.Some parameters have to be tuned mannually.Overfitting.Local minima.Time-consuming.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Figure 5: Feedforward Network Architecture: any type ofG(ai , bi , x).
New Learning Theory
If continuous target function f (x) can beapproximated by SLFNs with adjustable hiddennodes then the hidden node parameters of suchSLFNs needn’t be tuned. Instead, all thesehidden node parameters can be randomlygenerated without the knowledge of the trainingdata. Given any nonconstant piecewisecontinuous function g, for any continuous targetfunction f and any randomly generatedsequence {(ai , bi )
Li=1},
limL→∞
‖f (x) − fL(x)‖ = 0
holds with probability one if βi is chosen tominimize the ‖f (x) − fL(x)‖, i = 1, · · · , L.
G.-B. Huang, et al., “Universal Approximation Using Incremental Constructive Feedforward Networks with Random
Hidden Nodes,” IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Convex Incremental Learning Machine,” Neurocomputing, vol. 70, pp. 3056-3062, 2007.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Figure 5: Feedforward Network Architecture: any type ofG(ai , bi , x).
New Learning Theory
If continuous target function f (x) can beapproximated by SLFNs with adjustable hiddennodes then the hidden node parameters of suchSLFNs needn’t be tuned. Instead, all thesehidden node parameters can be randomlygenerated without the knowledge of the trainingdata. Given any nonconstant piecewisecontinuous function g, for any continuous targetfunction f and any randomly generatedsequence {(ai , bi )
Li=1},
limL→∞
‖f (x) − fL(x)‖ = 0
holds with probability one if βi is chosen tominimize the ‖f (x) − fL(x)‖, i = 1, · · · , L.
G.-B. Huang, et al., “Universal Approximation Using Incremental Constructive Feedforward Networks with Random
Hidden Nodes,” IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Convex Incremental Learning Machine,” Neurocomputing, vol. 70, pp. 3056-3062, 2007.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Unified Learning Platform
Figure 6: Feedforward Network Architecture: any type ofG(ai , bi , x).
Mathematical Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , standard SLFNswith L hidden nodes and output functiong(x) are mathematically modeled as
LXi=1
βi G(ai , bi , xj ) = tj , j = 1, · · · , N
(6)
(ai , bi ): Hidden node parameters.
βi : the weight vector connecting the i thhidden node and the output node.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Unified Learning Platform
Figure 6: Feedforward Network Architecture: any type ofG(ai , bi , x).
Mathematical Model
For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm , standard SLFNswith L hidden nodes and output functiong(x) are mathematically modeled as
LXi=1
βi G(ai , bi , xj ) = tj , j = 1, · · · , N
(6)
(ai , bi ): Hidden node parameters.
βi : the weight vector connecting the i thhidden node and the output node.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Mathematical ModelPLi=1 βi G(ai , bi , xj ) = tj , j = 1, · · · , N is equivalent to Hβ = T, where
H(a1, · · · , aL, b1, · · · , bL, x1, · · · , xN )
=
2664G(a1, b1, x1) · · · G(aL, bL, x1)
.
.
. · · ·...
G(a1, b1, xN ) · · · G(aL, bL, xN )
3775N×L
(7)
β =
26664βT
1...
βLT
37775L×m
and T =
26664tT1...
tTN
37775N×m
(8)
H is called the hidden layer output matrix of the neural network; the i th column of H is the output of the i thhidden node with respect to inputs x1, x2, · · · , xN .
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Outline
1 Neural NetworksSingle-Hidden Layer Feedforward Networks (SLFNs)Conventional Learning Algorithms of SLFNs
2 Extreme Learning MachineUnified Learning PlatformELM Algorithm
3 ELM for Multi-Categories Classification Problems
4 Performance Evaluations
5 Summary
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set ℵ = {(xi , ti)|xi ∈ Rn, ti ∈ Rm, i = 1, · · · , N}, activationfunction g, and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β: β = H†T.
where H† is the Moore-Penrose generalized inverse of hidden layer outputmatrix H.
Source Codes of ELM
http://www.ntu.edu.sg/home/egbhuang/
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set ℵ = {(xi , ti)|xi ∈ Rn, ti ∈ Rm, i = 1, · · · , N}, activationfunction g, and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β: β = H†T.
where H† is the Moore-Penrose generalized inverse of hidden layer outputmatrix H.
Source Codes of ELM
http://www.ntu.edu.sg/home/egbhuang/
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set ℵ = {(xi , ti)|xi ∈ Rn, ti ∈ Rm, i = 1, · · · , N}, activationfunction g, and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β: β = H†T.
where H† is the Moore-Penrose generalized inverse of hidden layer outputmatrix H.
Source Codes of ELM
http://www.ntu.edu.sg/home/egbhuang/
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set ℵ = {(xi , ti)|xi ∈ Rn, ti ∈ Rm, i = 1, · · · , N}, activationfunction g, and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β: β = H†T.
where H† is the Moore-Penrose generalized inverse of hidden layer outputmatrix H.
Source Codes of ELM
http://www.ntu.edu.sg/home/egbhuang/
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
Extreme Learning Machine (ELM)
Three-Step Learning Model
Given a training set ℵ = {(xi , ti)|xi ∈ Rn, ti ∈ Rm, i = 1, · · · , N}, activationfunction g, and the number of hidden nodes L,
1 Assign randomly hidden node parameters (ai , bi), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β: β = H†T.
where H† is the Moore-Penrose generalized inverse of hidden layer outputmatrix H.
Source Codes of ELM
http://www.ntu.edu.sg/home/egbhuang/
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Unified Learning PlatformELM Algorithm
ELM Learning Algorithm
Salient Features
“Simple Math is Enough.” ELM is a simple tuning-free three-step algorithm.
The learning speed of ELM is extremely fast.
Unlike the conventional learning methods which MUST see the training data before generating the hiddennode parameters, ELM could generate the hidden node parameters before seeing the training data.
Unlike the traditional classic gradient-based learning algorithms which only work for differentiable activationfunctions, ELM works for all bounded nonconstant piecewise continuous activation functions includingnon-differential activation functions.
Unlike the traditional classic gradient-based learning algorithms facing several issues like local minima,improper learning rate and overfitting, etc, ELM tends to reach the solutions straightforward without suchtrivial issues.
The ELM learning algorithm looks much simpler than many learning algorithms: neural networks andsupport vector machines.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
ELM for Multi-Categories Classification Problems
Three basic methods
1 Single ELM classifier: m output nodes of ELM for m-class applications.We say x is in class l if output node l has the highest output value.
2 One-Against-All ELM (ELM-OAA): m-class classification problems areimplemented by m binary ELM classifiers, each of which is trainedindependently to classify one of the m pattern classes.
3 One-Against-One ELM (ELM-OAO): the m pattern classes are pairwisedecomposed into m(m − 1)/2 two different classes, and each of them istrained by one binary ELM classifier.
Exponential loss based decoding approach used in ELM-OAA and ELM-OAOE. L. Allwein, et al., “Reducing multiclass to binary: a unifying approach for margin classifiers,” Journal of Machine
Learning Research, vol. 1, pp. 113-141, 2001.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
ELM for Multi-Categories Classification Problems
Three basic methods
1 Single ELM classifier: m output nodes of ELM for m-class applications.We say x is in class l if output node l has the highest output value.
2 One-Against-All ELM (ELM-OAA): m-class classification problems areimplemented by m binary ELM classifiers, each of which is trainedindependently to classify one of the m pattern classes.
3 One-Against-One ELM (ELM-OAO): the m pattern classes are pairwisedecomposed into m(m − 1)/2 two different classes, and each of them istrained by one binary ELM classifier.
Exponential loss based decoding approach used in ELM-OAA and ELM-OAOE. L. Allwein, et al., “Reducing multiclass to binary: a unifying approach for margin classifiers,” Journal of Machine
Learning Research, vol. 1, pp. 113-141, 2001.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
ELM for Multi-Categories Classification Problems
Three basic methods
1 Single ELM classifier: m output nodes of ELM for m-class applications.We say x is in class l if output node l has the highest output value.
2 One-Against-All ELM (ELM-OAA): m-class classification problems areimplemented by m binary ELM classifiers, each of which is trainedindependently to classify one of the m pattern classes.
3 One-Against-One ELM (ELM-OAO): the m pattern classes are pairwisedecomposed into m(m − 1)/2 two different classes, and each of them istrained by one binary ELM classifier.
Exponential loss based decoding approach used in ELM-OAA and ELM-OAOE. L. Allwein, et al., “Reducing multiclass to binary: a unifying approach for margin classifiers,” Journal of Machine
Learning Research, vol. 1, pp. 113-141, 2001.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
ELM for Multi-Categories Classification Problems
Three basic methods
1 Single ELM classifier: m output nodes of ELM for m-class applications.We say x is in class l if output node l has the highest output value.
2 One-Against-All ELM (ELM-OAA): m-class classification problems areimplemented by m binary ELM classifiers, each of which is trainedindependently to classify one of the m pattern classes.
3 One-Against-One ELM (ELM-OAO): the m pattern classes are pairwisedecomposed into m(m − 1)/2 two different classes, and each of them istrained by one binary ELM classifier.
Exponential loss based decoding approach used in ELM-OAA and ELM-OAOE. L. Allwein, et al., “Reducing multiclass to binary: a unifying approach for margin classifiers,” Journal of Machine
Learning Research, vol. 1, pp. 113-141, 2001.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
ELM for Multi-Categories Classification Problems
Three basic methods
1 Single ELM classifier: m output nodes of ELM for m-class applications.We say x is in class l if output node l has the highest output value.
2 One-Against-All ELM (ELM-OAA): m-class classification problems areimplemented by m binary ELM classifiers, each of which is trainedindependently to classify one of the m pattern classes.
3 One-Against-One ELM (ELM-OAO): the m pattern classes are pairwisedecomposed into m(m − 1)/2 two different classes, and each of them istrained by one binary ELM classifier.
Exponential loss based decoding approach used in ELM-OAA and ELM-OAOE. L. Allwein, et al., “Reducing multiclass to binary: a unifying approach for margin classifiers,” Journal of Machine
Learning Research, vol. 1, pp. 113-141, 2001.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Table 1: Specification of Real-World Classification BenchmarkProblems
Type Datasets # Attributes # Classes # ObservationsTraining Testing
Type I Glass 9 6 110 104Vehicle 18 4 420 426Page Blocks 10 5 2,700 2,773Image Segmentation 19 7 1,100 1,210Satellite Image 36 6 4,400 2,035Shuttle 9 7 43,500 14,500DNA 180 3 1,000 1,186Optical Recognition of Handwritten Digits 64 10 3,823 1,797Pen-Based Recognition of Handwritten Digits 16 10 7,494 3,498
Type II Cancer 98 14 144 46Arrhythmia 279 16 300 152Letter Recognition 16 26 10,000 10,000
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Table 2: Comparison of network complexity of ELM-OAA, ELM andELM-OAO
Datasets ELM-OAA ELM ELM-OAOGlass 30 30 20Vehicle 90 110 50Page Blocks 130 160 50Image Segmentation 200 210 30Satellite Image 450 470 150Shuttle 260 340 80DNA 350 450 290Handwritten Optical 420 470 110
RecognitionHandwritten Pen-Based 650 690 190
RecognitionCancer 40 50 10Arrhythmia 50 70 20Letter Recognition 1100 1500 510
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Table 3: Comparison of testing accuracy of ELM-OAA, ELM andELM-OAO
Datasets ELM-OAA ELM ELM-OAOGlass 66.346 64.423 65.385Vehicle 79.513 79.489 81.378Page Blocks 95.873 95.761 95.397Image Segmentation 95.216 94.667 94.493Satellite Image 89.670 89.663 89.955Shuttle 99.667 99.715 99.581DNA 94.866 94.828 94.496Handwritten Optical 96.928 96.948 96.343
RecognitionHandwritten Pen-Based 98.294 98.302 97.819
RecognitionCancer 78.652 77.522 73.000Arrhythmia 65.441 65.395 61.770Letter Recognition 93.417 93.029 93.055
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
Table 4: Comparison of training and testing time (seconds) ofELM-OAA, ELM and ELM-OAO
Datasets ELM-OAA (s) ELM (s) ELM-OAO (s)Training Testing Training Testing Training Testing
Glass 0.0250 0.0617 0.0117 0.0773 0.0133 0.0219Vehicle 0.1508 0.0617 0.0680 0.0531 0.0547 0.0313Page Blocks 1.7125 0.5023 0.6641 0.2070 0.3422 0.2836Image Segmentation 3.3547 0.2953 0.6000 0.0906 0.1453 0.1102Satellite Image 43.838 1.6500 8.4320 0.2758 5.1039 1.0383Shuttle 151.88 7.4594 36.614 1.6516 30.395 4.8422DNA 7.0797 0.6227 5.0328 0.2359 4.7070 0.4578Handwritten Optical Recognition 39.002 2.0734 6.7414 0.3461 4.2766 2.5727Handwritten Pen-Based Recognition 140.17 5.3305 18.127 0.7742 17.923 7.3648Cancer 0.0491 0.0191 0.0072 0.0194 0.0469 0.0422Arrhythmia 0.1250 0.0586 0.0266 0.0273 0.1414 0.2281Letter Recognition 2295.6 73.006 76.066 1.9700 1429.5 279.57
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
Summary
ELM, ELM-OAO and ELM-OAA obtain similar testingaccuracies.ELM-OAO usually requires smaller number of hiddennodes than the single ELM classifier and ELM-OAA.The training time required by ELM-OAO is similar or lessthan ELM and ELM-OAA when the number of patternclasses is small (say, not larger than 10).However when the number of pattern classes is large (say,larger than 10), the training time cost by ELM-OAO is mostlikely higher than the single ELM classifier but still smallerthan ELM-OAA.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines
tu-logo
ur-logo
Neural NetworksELM
ELM for Multi-Categories Classification ProblemsPerformance Evaluations
Summary
References
References
G.-B. Huang, et al., “Universal Approximation Using Incremental Networks with Random HiddenComputational Nodes”, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, 2006.
G.-B. Huang, et al., “Extreme Learning Machine: Theory and Applications,” Neurocomputing, vol. 70, pp.489-501, 2006.
G.-B. Huang, et al., “Convex Incremental Extreme Learning Machine,” Neurocomputing, vol. 70, pp.3056-3062, 2007.
M.-B. Li, et al., “Fully complex extreme learning machine,” Neurocomputing, vol. 68, pp. 306-314, 2005.
N.-Y. Liang, et al., “A Fast and Accurate On-line Sequential Learning Algorithm for Feedforward Networks,”IEEE Transactions on Neural Networks, vol. 17, no. 6, pp. 1411-1423, 2006.
Q.-Y. Zhu, et al., “Evolutionary Extreme Learning Machine”, Pattern Recognition, vol. 38, no. 10, pp.1759-1763, 2005.
G.-B. Huang, et al., “Can Threshold Networks Be Trained Directly?” IEEE Transactions on Circuits andSystems II, vol. 53, no. 3, pp. 187-191, 2006.
G.-B. Huang, et al., “Real-Time Learning Capability of Neural Networks”, IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 863-878, 2006.
ELM Web Portal: www.ntu.edu.sg/home/egbhuang Extreme Learning Machines