NIR

PreprocessingWith focus on NIR

Åsmund Rinnan , Frans van den Berg, Rasmus Bro, Søren Balling Engelsen, Lars Nørgaard , Jonas

Thygesen

NIR

Introduction

• How often is it used?• When is it used?• What methods are used?• Why is it used?

• NIR is a measurement technique “screaming” for preprocessing

Introduction

Techniques

Summary

NIR

IntroductionNIR

FructoseGlucose0

5075

1005025

Introduction

Techniques

Summary

NIR

IntroductionTechniques

• Correction of light scatter– MSC/ ISC– PMSC– EMSC/ EISC

• SNV• Derivation

– Norris&Williams– Savitsky&Golay

• Baseline-correction

• Normalization• Detrend• Use of reference

values– O-PLS– OSC– OS– SIS

Introduction

Techniques

Summary

NIR

IntroductionWhy

A “fat” baseline

Baseline with a slope/ curve

Nonlinearity

Introduction

Techniques

Summary

NIR

MSC

Reference

Raw

spe

ctru

m

ISC

• Offset• Normalization

Introduction

Techniques

Summary

1. P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500

NIR

MSC/ ISC• Offset• Normalization

Introduction

Techniques

Summary

NIR

EISC

Ghx Tknown f

exλxx knownorgnew fff

2orgorgorg xx1x cbaf

λλλλ log2 cbaf

1. H. Martens and E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635

2. H Martens, JP Nielsen, SB Engelsen (2003): Light scattering and light absorbance separated by extended multiplicative signal correction. The application to near-infrared transmission analysis of powder mixtures, Analytical chemistry, 75, 394-404

3. DK Pedersen, H Martens, JP Nielsen, SB Engelsen (2002): Near-Infrared Absorption and Scattering Separated by Extended Inverse Signal Correction (EISC): Analysis of Near-Infrared Transmittance Spectra of Single Wheat, Applied Spectroscopy, 56, 1206-1214

4. Thennadil, SN, Martin, EB (2005): Empirical preprocessing methods and their impact on NIR calibrations - a simulation study , Journal of Chemometrics, 19, 77-89

• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

EMSC(w/ 2. degree &

wavelength)• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

PxSC

X = EMEIMI

1. Isaksson, T.; Kowalski, B. R. (1993): Piece-wise multiplicative scatter correction applied to Near-Infrared diffuse transmittance data from meat products, Applied Spectroscopy, 47, 702-09

• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

PMSC(window size = 35)

• Offset• Normalization

Introduction

Techniques

Summary

NIR

SNV

x

orgx

orgnew

xx

1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

• Offset• Normalization

Introduction

Techniques

Summary

NIR

SNV vs MSC

b

a org

new

xx

MSC SNV

Introduction

Techniques

Summary

NIR

Detrend

n

i

iineworg axx

1

1

1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

• Offset• Baseline

Introduction

Techniques

Summary

NIR

DerivationSavitsky-Golay

i

i

i

i

fnew

mm

mm

mm

mm

d

x

...

1...1

...0...

1...1

...

!

,

C

CC

Co

ox

T

T

Tmfm,forg,

1. A Savitsky, MJE Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639 2. P.A. Gorry (1990): General Least-Squares smoothing and differentiation by the convolution (Savitsky-Golay) method, Analytical Chemistry, 62, 570-573

• Offset• Baseline

Introduction

Techniques

Summary

NIR

DerivationSavitsky-Golay

• Offset• Baseline

Introduction

Techniques

Summary

NIR

DerivationSavitsky-Golay

• Offset• Baseline

Introduction

Techniques

Summary

NIR

DerivationNorris-Williams

3

32

3

:

1

1,

,

2,1,1,2,,1,1,2,,

1,2,,1,1,,

1,,

,,

iiforg

fsmooth

forgforgforgforgfsmoothfsmoothfsmoothfnew

forgforgfsmoothfsmoothfnew

n

iifnew

fnewfnew

xx

xxxxxxxx

xxxxx

x

xxoptional

nd

st

1. Norris, KH (1983): in Food Research and Data Analysis (Eds: Martens H and Russwurm H Jr), Applied Science, London, 95-114 2. Norris KH, Williams PC (1984): Optimization of mathematical treatments of raw near-infrared signal in the measurement of protein in hard red spring wheat: I.

Influence of particle size, Cereal Chemistry, 61, 158-165

• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

DerivationNorris-Williams

• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

DerivationNorris-Williams

• Offset• Baseline• Normalization

Introduction

Techniques

Summary

NIR

Preprocessing

• PMSC: Sensitivite to window size• EMSC: Small effect on baseline

Detrend is better• MSC and SNV are practical the

same• Savitsky-Golay is more robust

than Norris-Williams

Introduction

Techniques

Summary

NIR

Preprocessing

Offset

Baselinje

Normalization

(P)MSC/ISC/SNV

x x

(P)EMSC/EISC x x

Detrend x x

Deriveration x x x

Introduction

Techniques

Summary

NIR

Thanks for the attention

www.models.life.ku.dk