signal decomposition methods with application to energy...

IntroductionNon-negative Matrix Factorization (NMF)

Sparse Component Analysis (SCA)Independent Component Analysis (ICA)

Application: Signal decomposition for Load disaggregationConclusion

Signal Decomposition Methods with Application toEnergy Disaggregation

Alireza Rahimpour

Supervisor: Dr. Hairong Qi

University of TennesseeDepartment of Electrical Engineering and Computer Science

June 23, 2015

Alireza rahimpour. UTK Signal decomposition methods




Contents

1 Introduction

2 Non-negative Matrix Factorization (NMF)

3 Sparse Component Analysis (SCA)

4 Independent Component Analysis (ICA)

5 Application: Signal decomposition for Load disaggregation

6 Conclusion


3/58




What is source separation?

Signal decomposition

Separation of a set of source signals from a set of mixed signals.

V = WH

(1)

V ∈ Rm×n: Observation matrixW ∈ Rm×r : Basis matrix, dictionary or source matrixH ∈ R r×n: Activation matrix

m: dimension of the observationr: number of sources

n: number of observations


4/58





One Example

Cocktail party


5/58





Magic?

V = WH

In fact, without some prior knowledge, it is not possible touniquely estimate the original source signals.

Need to impose constraints on the problem.


6/58





Three main families of signal decomposition methods

1 Non-negative Matrix Factorization (NMF):Both sources and mixtures are non-negative.

2 Sparse Component Analysis (SCA):Most of the entries of activation matrix are zero.

3 Independent Component Analysis (ICA):Sources are statistically independent and non-Gaussian.


7/58





Supervised vs. Unsupervised methods

V = WH

V: Observation matrixW: Basis matrix or dictionary

H: Activation matrix

Unsupervised:

Only observation matrix V is available and W and H are bothunknown : Blind source separation.

Supervised:

Observation matrix V and basis matrix W are known and goal is toestimate the activation matrix H.


8/58





History of Signal decomposition methods

Blind source separation (BSS) problem formulated around1982, by Hans, et al. for a biomedical problem :first papers in the mid of the 80’s.

Great interest from the community, mainly in France and laterin Europe and in Japan, and then in the USA.

In June 2009, 22000 scientific papers are recorded by GoogleScholar (Comon and Jutten, 2010)

People with different backgrounds: signal processing, statisticsand machine learning.

Until the end of the 90’s, BSS=ICA

First NMF methods in the mid of the 90’s but famouscontribution in 1999.

First SCA approaches around 2000 but massive interest since.


9/58




NMF IntroductionSolving NMFConstraints on NMFSupervised NMFApplications of NMF

Non-negative Matrix Factorization (NMF)


10/58





Non-negative Matrix Factorization

A matrix factorization where everything is non-negative.

V ∈ Rm×n+ , W ∈ Rm×r

+ , H ∈ R r×n+


11/58





Why Non-negativity?

Data is often non-negative by nature:

Pixel intensities

Amplitude spectra

Energy consumption

Occurrence counts

User scores

Stock market values

..., etc.


12/58





Why Non-negativity? cont.

NMF allows only additive but not subtractive combinationsduring the factorization:

Result in part-based representation of the data.

Such nature can discover the hidden components that havespecific structures and physical meanings.


13/58





Part-based representation of NMF


14/58





History of research on NMF:

NMF is more than 30-year old!

First work:

Positive Matrix Factorization-PMF (Paatero and Tapper, 1994)

Popularized by Lee and Seung (1999) for learning the parts ofobjects (Nature journal).

Since then, widely used in various research areas for diverseapplications.


15/58





Solving NMF

How do we solve for W and H, given a known V?Answer: Frame as optimization problem:

MinimizeW≥0,H≥0

D(V ||WH)

Two measure of divergence proposed by (Lee and Seung,1999) are:

Euclidean (EU) : DEU(V ‖WH) =∑

ij(Vij − (WH)ij)2

Kullback-Leibler (KL):

DKL(V ‖WH) =∑

ij(VijlogVij

(WH)ij− Vij + (WH)ij)


16/58





And many other divergence measures

Itakura-Saito (IS) divergence (Fvotte et al., 2009)

β- divergence (Eguchi and Kano., 2001)

α-divergence (Cichocki et al., 2006, 2008)

Bregman divergence (generalizes β-divergence) (Bregman, 1967; I.S. Dhillon and S. Sra, 2005)

Rnyis divergence (Devarajan and Ebrahimi, 2005)

γ-divergence (Fujisawa and Eguchi, 2008)

etc.

NMF divergence choice depends on the data and on theapplication.


17/58





Algorithms

There are several ways in which the W and H may be found:

Lee and Seung’s multiplicative update rule (Lee and Seung,1999) has been a popular method.

Some successful algorithms are based on AlternatingNon-negative Least Squares (ANLS) (Berry et al. ,06):

In each step of algorithm, first H is fixed and W found by anon-negative least squares solver, then W is fixed and H isfound analogously.

And many other. . .


18/58





Initialization

A good initialization of parameters (W and H) is important due tothe existence of local minima.

Random initializations:- initialize (nonnegative) parameters randomly several times;- keep the solution with the lowest final cost.

Strcutural data-driven initializations:

initialize W by clustering of data points V (Kim and Choi,2007);initialize W by singular value decomposition (SVD) of data

points V (Boutsidis and Gallopoulos, 2008);etc ...


19/58





Adding constraints to NMF

Problems:

- NMF solution is not unique.- Hence NMF is not guaranteed to extract latent components asdesired within a particular application.

Possible solution:

Given the application, impose some knowledge-based constraintson:W, H, or on both W and H.

- Appropriate constrains may lead to more suitable latentcomponents.


20/58





Adding constraints to NMF

Different types of constraints have been considered in previousworks:

Sparsity constraints: either on W or H (e.g., Hoyer, 2004;Eggert and Korner, 2004);


21/58





Adding constraints to NMF, Cont.

Group sparsity-inducing penalties in NMF.(A. Lefvre et al., 2011; Sun and Mazumder, 2013; El Badawyet al., 2014)


22/58





Adding constraints to NMF, Cont.

And many more possible constraints

Smoothness on activation matrix (Virtanen, 2007; Jia andQian, 2009; Essid and Fevotte, 2013);

Cross-modal correspondence constraints (Seichepine et al.,2013; Liu et al., 2013);

Minimum volume NMF (Miao and Qi, 2007);

Orthogonal NMF (Choi, 2008);

etc.


23/58





Supervised NMF

Assumption:

One has access to example signals at the training step

1 NMF uses these training data to train a set of basis vectorsfor each source (i = 1, . . . , k).

V traini = W train

i Htraini

2 The basis matrix (trained bases) W traini is used as a

representative model for the training data for each source i.

3 In the testing or separation stage, the trained basis matricesfor all sources are concatenated in one bases matrix:

W = [W1, . . . ,Wk ]


24/58





Supervised NMF, Cont.

In the testing stage, for a new test observation V the basesmatrix W is kept fixed and only the weights matrix H isupdated: H = argmin

H≥0||Vtest −WH||2

The estimated signal for each source is therefore found bymultiplying each source basis in the bases matrix with itscorresponding weights in the weights matrix:

Vi = Wi Hi


25/58





Applications

Computer vision and Image analysis:

Feature representation, clustering (Lee et al., 99)

Action recognition (Masurelle et al., 2014)Denoising and inpainting (Mairal et al., 2010)Face recognition (Soukup and Bajla, 2008)Tagging (Kalayeh et al., 2014)


26/58





Other applications

Acoustic signal processing:-Speech recognition (Cichocki et al., 04)-Signal enhancement/denoising: (Sun and Mazumder, 2013)-Compression (Ozerov et al., 2011b; Nikunen et al., 2011)

Medical Imaging : CT image reconstruction (Damon et al., 2013)

Text mining:-Document clustering (Xu et al., 03; Shahnaz et al., 06)-Topic detection and trend tracking(Berry et al., 05)

Social network: ( Chi et al., 07; Wang et al., 08)

Recommendation system:

-The Netflix Prize: predict user ratings for films (Zhang et al., 06)


27/58




SCA problem statementSolving the SCADictionary learningApplication

Sparse Component Analysis (SCA)


28/58





The Sparse Decomposition Problem:


29/58





Sparse Decomposition problem

1

Lp norm: ‖α‖p = (∑

i |αi |p)1/p

1Slide from (Marial, Sapiro 2010)


30/58





Lp norm

In general the lp norm is a popular measure of sparsity in the

mathematical analysis community.

For measuring sparsity, 0 < p < 1 is of the most interest but,unfortunately, it leads to a non-convex optimization problem.

-We usually choose p = 1 for obtaining the sparse solution.


31/58





Approaches

Relaxation methods:Smooth the L0, e.g., basic pursuit (BP, also referred to asLASSO), FOCUSS. (Chen,Donoho&Saunders:1995)

Greedy algorithm:Build the solution one non-zero element at a time, e.g.,mathching pursuit (MP), OMP (Mallat&Zhang:2003)


32/58





How to choose the Dictionary D?

Choose D from a known set of transforms (e.g., wavelets,DCT, etc.)But, this is not optimal.

Training (Learn from examples):Several dictionary learning approaches:

-K-SVD: (M. Aharon, M. Elad, 2006)

- Alternates between sparse coding of the examples based on the current

dictionary and an update process for the dictionary atoms so as to better

fit the data.


33/58





Application

Sparsity

Many natural signals are sparse or compressible in the sense thatthey have concise representation when expressed in the properbasis.

Transform sparsity of MR images.

The largest coefficients are preserved while all others set to zero.

- Transform-based compression: JPEG, JPEG-2000, and MPEG.


34/58





Application: Compressed sensing

Replace samples with few linear projections: y = φx

φ = Measurement or sensing basisY= Our measurementX=Original signal


35/58





Why Compressed sensing?

Reducing the number of measurments

The number of sensors may be limited.

The measurements may be extremely expensive: neutronscattering imaging.

The sensing process may be slow: MRI.


36/58




Problem statementPreprocessingSolving the ICAICA Application

Independent Component Analysis (ICA)


37/58





Independent Component Analysis

This method is the base of blind source separation.

X = ASX: Observation matrixA: Mixing matrixS: Source matrix

- All we observe is X and we must estimate both A and S using it.

Assumption:

1 The sources being considered are statistically independent.

2 The independent components have non-Gaussian distribution.


38/58





ICA preprocessing

Preprocessing

Preprocessing steps in order to simplify and reduce the complexityof the problem:

Centering

Whitening

Dimensionality reduction

Whitening and dimension reduction can be achieved with:-Principal Component Analysis (PCA)-Eigenvalue Decomposition (EVD)


39/58





Algorithms

ICA finds the independent components by:

Maximizing the statistical independence of the estimated components.

Many ways to define independence, and this choice governs the form of

the ICA algorithm.

Minimization of mutual information (Comon, 1994)

-Kullback-Leibler Divergence.

-Maximum entropy.

Maximization of non-Gaussianity (Bell and Sejnowski, 1995)

Non-Gaussianity measures:-Kurtosis : kurt(x) = 0 if x Gaussian

-Negentropy : (= 0 when Gaussian)


40/58





Application

Audio signal processing : (Cichocki & Amari 2002);

Image processing :(Cichocki & Amari 2002);

Data mining: (Lee et al. 2009);

Biomedical signal processing: (Azzerboni et al. 2004);

And many more.


41/58




What is Load disaggregation?Related worksProblem formulation

Application:Signal decomposition for Load disaggregation


42/58





Signal decomposition for Load disaggregation

Load disaggregation

The task of taking a whole-home energy signal from a singlemonitoring point and separating it into its component appliances.

Also called: Non-Intrusive Load Monitoring (NILM)Non-Intrusive: without use of additional measuring equipment inside the

consumer’s house.Alireza rahimpour. UTK Signal decomposition methods

43/58





Why does it matter?

Customers

- Understand their bill and plan their monthly budget.- Identify/repair/replace energy hogs.- Be able to make a financial decision for when to use an appliance.

Utilities

- Identify/verify appliances that could participate in DemandResponse.- Improve relationships with customers.- Understand customer behavior to improve capacity planning.


44/58





How much energy can we really save?

ACEEE analyzed results from 36 different studies between1995-2010 and found energy savings varied based on the datagiven to customers:


45/58





Goal?

Having device level energy information can cause users to conservesignificant amounts of energy.

But,current electricity meters only report whole-home data.So,

Developing algorithmic methods for disaggregation presents a key point

in energy conservation.Alireza rahimpour. UTK Signal decomposition methods

46/58





Commercial Load monitoring device:

Plugwise:


47/58





Related works

Academic work on energy disaggregation began with the work ofGeorge W. Hart, at MIT in the early 1980s.

Methods based on Classification:

These approaches look for some features in power signals, andcluster devices according to these features.1-signal analysis and feature extraction.2-model learning and classification.

Methods based on decomposition:

Sparse coding (Kolter, Andrew Y. Ng 2010, MIT, Stanford)


48/58





Cont.

In general, algorithms for energy disaggregation have receivedgrowing interest in recent years. (Kolter, Batra, and Ng 2010; Kimet al. 2011; Ziefman and Roth 2011)

Another challenge:

Existing energy disaggregation approaches virtually all usehigh-frequency sampling (e.g. per second or faster)

But,smart meters (communication-enabled power meters):limited to recording whole home energy usage at low frequencies.

-Currently installed in more than 32 million homes (Institute for Electric

Efficiency 2012)Alireza rahimpour. UTK Signal decomposition methods

49/58





NMF for Load disaggregation

X ∈ Rm×d : Aggregated energy consumption for all devices in agiven house.

Di ∈ Rm×n : The energy consumption of the ith appliance.Each column of Di : One day of power consumption.

d: Number of testing days (unforeseen)n: Number of training daysm: Number of samples in each column ( i.e., each day)k= number of all the devices at home.

Dictionary D:

Concatenation of all the basis vectors for all the devices.D ∈ Rm×T (D = [D1,D2, . . .Dk ] and T = n × k).


50/58





X = DA

A1:k = argminA1:k≥0

∥∥∥∥∥∥∥∥∥X − [D1 . . .Dk ]

A1

A2...Ak

∥∥∥∥∥∥∥∥∥

2

2

Ai (i = 1, . . . , k): the activation matrix for the ith device’s basematirx (Di ).

After calculating the activation coefficient for each device (Ai ), theestimated signal for the ith device would be:

Xi = Di Ai

Xi ∈ Rm×d , Di ∈ Rm×n and Ai ∈ Rn×d


51/58





NMF for LD, Cont.

Training stage:

Decomposition of the aggregated signal to each device’s powerconsumption at the same days as the training set (n).

Testing stage:

Estimating the power consumption of each device by having accessto the aggregated signal in a totally different random days (d) thanthe days in training set (n).

Using 80% of the data as training set (as basis in dictionary) and 20% of

the data for the testing set (aggregated signal).

Constrained NMF

What additional constraints can we define for this problem?


51/58





NMF for LD, Cont.

Training stage:

Decomposition of the aggregated signal to each device’s powerconsumption at the same days as the training set (n).

Testing stage:

Estimating the power consumption of each device by having accessto the aggregated signal in a totally different random days (d) thanthe days in training set (n).

Using 80% of the data as training set (as basis in dictionary) and 20% of

the data for the testing set (aggregated signal).

Constrained NMF

What additional constraints can we define for this problem?Alireza rahimpour. UTK Signal decomposition methods

52/58




SummaryAvailable softwares for signal decompositionReferencesThe End!

Conclusion


53/58





Summary:

We reviewed different works on signal decomposition based onNMF, ICA, SCA.

Choosing the right method highly depends on the structure of dataand intended application.

Adding prior knowledge to the problem as a constraint.

NMF is effective and has been gaining increasing popularity in thelast few years: significant number of publications on different topics.

Application of NMF method on Energy disaggregation.

Future:How to handle big data? e.g., Web-scale data mining.

Online signal decomposition: how to update the factorization when new data

comes in without recomputing from scratch.


54/58





Available softwares for different signal decompositionalgorithms:

SPAMS PACKAGE:http://spams-devel.gforge.inria.fr/downloads.html

SPArse Modeling Software is an optimization toolbox for solving various sparse estimation problems.Dictionary learning and matrix factorization (NMF, sparse PCA, ...)

libNMF : A Library for Nonnegative Matrix Factorization.http://www.univie.ac.at/rlcta/software/

Beta-NTF: Nonnegative Matrix and Tensor Factorization in Python.https://code.google.com/p/beta-ntf/

Sklearn.decomposition.NMF: Non-Negative matrix factorization.http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html

The NMF:DTU Toolbox-MATLAB.http://cogsys.imm.dtu.dk/toolbox/nmf/index.html

The FastICA package for MATLAB.http://research.ics.aalto.fi/ica/fastica/

Sparsity-based Blind Source Separation (BSS) method:http://www.ast.obsmip.fr/article715.html

Many softwares available in ICA Central, ICALab.


http://spams-devel.gforge.inria.fr/downloads.html

http://www.univie.ac.at/rlcta/software/

https://code.google.com/p/beta-ntf/

http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html

http://cogsys.imm.dtu.dk/toolbox/nmf/index.html

http://research.ics.aalto.fi/ica/fastica/

http://www.ast.obsmip.fr/article715.html

55/58





References

Vahid Abolghasemi, Saideh Ferdowsi, and Saeid Sanei. Sparse multichannel source separation using incoherentk-svd method. In Statistical Signal Processing Workshop (SSP), 2011 IEEE, pages 477480. IEEE, 2011.

A. Cichocki, S. Cruces, and S. Amari. Generalized Alpha-Beta Divergences and Their Application to RobustNonnegative Matrix Factorization. Entropy, 13:134170, 2011.

A. Lefvre, F. Bach, and C. Fvotte. Itakura-Saito nonnegative matrix factorization with group sparsity. In Proc.International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic, 2011.URL http://perso.telecom-paristech.fr/ fevotte/Proceedings/icassp11c.pdf.

L. Le Magoarou, A. Ozerov, and N. Q. K. Duong. Text-informed audio source separation using nonnegative matrixpartial co-factorization. In Proc IEEE. Int. Workshop on Machine Learning for Signal Processing (MLSP), 2013.

D. D. Lee and H. S. Seung. Learning the parts of objects with nonnegative matrix factorization. Nature, 401:788791, 1999.

D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In Advances in Neural andInformation Processing Systems 13, pages 556562, 2001.

H. Lee, Y.-D. Kim, A. Cichocki, and S. Choi. Nonnegative tensor factorization for continuous EEG classification.International Journal of Neural Systems, 17(4):305317, 2007. c

H. Lee, A. Cichocki, and S. Choi. Kernel nonnegative matrix factorization for spectral EEG feature extraction.


56/58





L.-H. Lim and P. Comon. Multiarray signal processing: Tensor decomposition meets compressed sensing.Compte-Rendus de lAcademie des Sciences, section Mecanique, 338(6):311320, June 2010.

A. Limem, G. Delmaire, M. Puigt, G. Roussel, and D. Courcot. Non-negative matrix factorization using weightedBeta divergence and equality constraints for industrial source apportionment. In 23rd IEEE International Workshopon Machine Learning for Signal Processing (MLSP 2013), Southampton, UK, September 2225 2013.

C.-J. Lin. On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization. IEEETransactions on Neural Networks, 18:15891596, 2007a.

C.-J. Lin. Projected Gradient Methods for Nonnegative Matrix Factorization. Neural Computation, 19:27562779,2007b.

H. A. Loeliger. An introduction to factor graphs. IEEE Signal ProcessingMagazine, 21:2841, 2004.

W. Lu, W. Sun, and H. Lu. Robust watermarking based on dwt and nonnegative matrix factorization. Computersand Electrical Engineering, 35(1):183188, 2009.

J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. TheJournal of Machine Learning Research, 11(10-60), 2010. URL http://dl.acm.org/citation.cfm?id=1756008.

A. Masurelle, S. Essid, and G. Richard. Gesture recognition using a NMF-based representation of motion-tracesextracted from depth silhouettes. In IEEE International Conference on Acoustics, Speech, and Signal Processing(ICASSP), Florence, Italy, 2014.

S. Meignier and T. Merlin. LIUM SPKDIARIZATION: AN OPEN SOURCE TOOLKIT FOR DIARIZATION. InCMU SPUD Workshop, Texas, USA, 2010. URL http://lium3.univ-lemans.fr/diarization/doku.php/welcome.

P. Melville and V. Sindhwai. Recommender systems. In C. Sammut and W. G., editors, Encyclopedia of MachineLearning. Springer-Verlag, 2010. URL http://dl.acm.org/citation.cfm?id=245121.


57/58





N. Mohammadiha, P. Smaragdis, and A. Leijon. Supervised and unsupervised speech enhancement using NMF.IEEE Transactions on Audio Speech and Language Processing, 21(10):21402151, Oct. 2013.

L. Miao and H. Qi. Endmember extraction from highly mixed data using minimum volume constrained nonnegativematrix factorization. IEEE Trans. on Geoscience and Remote Sensing, 47:765777, 2007.

V. Monga and M. K. Mihcak. Robust and secure image hashing via non-negative matrix factorizations. IEEETrans. on information Forensics and Security, 2(3):376390, Sep. 2007.

J Zico Kolter, Siddharth Batra, and Andrew Y Ng. Energy disaggregation via discriminative sparse coding. InAdvances in Neural Information Processing Systems, pages 11531161, 2010.

Michael Zeifman and Kurt Roth. Nonintrusive appliance load monitoring: Review and outlook. Consumer

Electronics, IEEE Transactions on, 57(1):7684, 2011.


58/58






signal decomposition methods with application to energy...

Documents