Learning-Based Detection of Acne-like RegionsUsing Time-Lapse Features

Siddharth K. Madan and Kristin J. DanaDepartment of Electrical and Computer Engineering

Rutgers UniversityNJ, USA

O.CulaJohnson & Johnson Consumer and Personal Products Division

Springfield, USANJ, USA

Abstract—Objective evaluation of acne treatment requiresobserving test subjects for multiple months. To capture theappearance of acne lesions during the treatment period, asubject is photographed at imaging sessions separated by timeintervals of days or weeks. The efficacy of the treatmentmethod is evaluated by counting the number of acne lesionsin the acquired skin images. Traditionally, the counting of acnelesions has been done manually. However, manual counting isunreliable and time consuming; therefore in recent years therehas been an increasing interest in automatically detecting andcounting acne lesions using computer-based methods. In thispaper we model acne-like and non-acne regions using spatio-temporal features, and use a supervised learning approach tofind the separating hyperplane between the regions in the featurespace. The temporal component is an important feature becauseacne lesions change over time, while scars and other marksremain constant. Precise alignment is a challenge in computingmeaningful temporal features. The images must be aligned toa subpixel level, exceeding the requirements of typical facealignment algorithms. We have acquired and aligned a timeseries acne dataset by imaging a human subject with facial acneunder the same illumination and pose on 39 different days overa period of three months. The resulting time-lapse video of skinwith precision alignment is the first of its kind and impressivelydemonstrates the temporal evolution of acne lesions. We use thisregistered time-lapse set to train and test an acne lesion classifier.

I. INTRODUCTION

Treatment of acne lesions is an important and widelyresearched problem in the dermatology community. For everytreatment procedure, it is important to quantitatively assessthe efficacy. A popular method to assess the effectiveness ofan acne treatment is to count the number of lesions [3], [13],[16]. The subject is imaged at different imaging sessions, withintervals of days and weeks and the acquired images recordthe appearance and evolution of images during the treatmentprocess. The lesion counting step has traditionally been donemanually; but this method is unreliable and time consuming.For example, consider the skin region with acne lesions shownin Figure 1 (a). The acne lesions in the skin region have beenmanually marked in Figure 1 (b). It can be seen that countingsuch lesions in every skin region of the face in a clinical studyrequires a great amount of time and effort; therefore, recentlythere has been great interest in computer-based detection ofacne lesions. In previous work, multispectral imaging andlinear discriminant functions have been used to automatically

(a) (b)

Fig. 1. (a): Skin image with acne lesions. (b): Borders of acne lesions inthe skin image have been manually outlined. Counting such lesions in everyregion of the face image is time consuming and tedious.

detect acne lesions [5], but this work does not have a time-varying component. Gaussian mixture models have been useddetect and track these lesions over multiple time points [4];since the detection is within a single image frame, temporalfeatures are not used. In [15], skin images are registeredover multiple time points, but the alignment is done to assistmanual lesion counting. In this paper, we present a novelsupervised learning technique to automatically detect acne-likelesions. We model skin regions by a six dimensional vectorusing temporal and spatial features, and detect the separatingboundary between the patch images. The temporal componentis an important feature because acne lesions change overtime, while scars and other marks remain constant. Precisealignment of the image sequence is important for computingmeaningful temporal features. We train and test a classifier ona time series acne dataset obtained by imaging a human subjectwith facial acne under the same illumination and pose on 39different days over a period of three months. The images havebeen acquired under cross-polarized modality to eliminatesurface-layer reflectance (we discuss polarized imaging of skinin section II). The time-lapse images are registered beforeperforming any quantitative processing. All images in the time-lapse set are shown in appendix A. 1.

The rest of the paper is organized as follows: in section IIwe discuss polarized imaging of skin, in section III we discussthe acquisition and registration of the time-lapse images, insection IV we discuss the feature space representation andclassification of the patches, in section V we present theexperimental results, and in section VI we discuss the futuredirection of research.

1The registered and original time-lapse videos are available at http://www.ece.rutgers.edu/∼kdana/Research.html

978-1-4673-0372-9/11/$26.00 ©2011 IEEE

Incident LightSurface/SpecularReflection

Subsurface/DiffuseReflection

Surface Normal

Fig. 2. Interaction of light with the skin surface.

II. POLARIZED IMAGING OF SKIN

Skin is a multi-layered medium, with complex fine scalegeometry, an oily layer at the air-skin interface, and withlayers of different types of cells in the stratum corneum,epidermis, and dermis. Consequently, the way skin reflectsvisible light is defined by the reflection and inter-reflection ofincident light at the interfaces between layers with differentindices of refraction. A simplified skin reflectance model isthe surface-subsurface model [12] that describes reflectance asthe sum of surface and subsurface components. The surfacecomponent is the part of the incident light reflected off the skinsurface. This component arises due to the oily layer presenton normal facial skin. The subsurface component is the partof the incident light traveling through the stratum corneumand the epidermis. The subsurface component suffers multiplesubsurface scattering events before it exits the skin. Figure 2schematically illustrates the multi-layered structure of skin andits interaction with visible light. When skin is illuminated withlinearly polarized light, the surface reflection preserves thepolarization of the incident light, but the polarization status ofthe subsurface component gets randomized by the birefringentdermal collagen fibers [1]. The dichotomy in polarization ofthe reflected components makes linearly polarized imaging ofskin quite useful [14], [8]. The incident light can be polarizedby placing a linear polarizer in front of the light source.If a polarizer is placed in front of the sensor, the intensitymeasured by the sensor depends on the relative orientation ofthe polarizer in front of the light source and the polarizer infront of the sensor [7]. If the polarizer in front of the sensor isparallel to the polarizer in front of the light source, the entiresurface component is measured by the sensor, but only halfthe subsurface component reaches the sensor. If Is denotes thesurface component and Id denotes the subsurface component,the intensity Ip measured by the sensor is,

Ip = Is +1

2Id. (1)

We use the term parallel-polarized image for an image ac-quired with parallel polarizers in front of the light source andthe sensor. A parallel-polarized image enhances the surfacecomponent, and brings out surface features like raised bordersof lesions, pore structure, and wrinkles [9]. If the polarizer infront of the sensor is perpendicular to the polarizer in frontof the light source, the entire surface component gets blocked,

Light source

Camera

Chinrest

Fig. 3. Imaging equipment used to acquire the time-lapse images.

and the sensor measures only half the subsurface component.The intensity Ix measured by the sensor is,

Ix =1

2Id. (2)

We use the term cross-polarized image for an image acquiredwith perpendicular polarizers in front of the light source andthe sensor. The cross-polarized image brings out subsurfaceskin features like color variation due to melanin erythema[10]. The subsurface component penetrates 300 micrometersor more below the skin surface [7]; therefore the subsurfacecomponent offers a reasonable measurement of melanin andsuperficial blood vessels residing in the epidermis and theupper part of the dermis, respectively. When no polarizers areplaced in front of the light source and the sensor, the intensityIv measured by the sensor would be the sum of the surfaceand the subsurface components.

Iv = Is + Id. (3)

We use the term visual image for an image acquired withoutany polarizers in front of the sensor and the light source. In thispaper we have acquired images under cross-polarized modalityto eliminate the surface reflection and bring out skin features.

III. DATA ACQUISITION AND IMAGE REGISTRATION

Figure 3 shows the custom built imaging equipment usedto acquire the face images. The imaging equipment consistsof a light source, a sensor, and polarizer filters placed in frontof the light source and the sensor. The polarizer filters areorientated perpendicular to each other, to acquire the imagesunder cross-polarized modality. Using the imaging equipmentwe acquire 39 images of a human face with acne lesions overa three month period. Figure 4 shows sample images of theface from the database. Each image is 980×1504 in size.During the three month period acne lesions appear, evolve,and disappear. Note that the lesions are predominantly presentin the forehead and chin regions. We train the classifier onthe chin region and test on forehead region. The acquiredtime-lapse face images are misregistered. In order to performquantitative analysis we register all 39 time-lapse images usinga two step approach. In the first step, we use the Lucas-Kanade algorithm [2] to globally register the forehead regions.Figure 5 (a) and (b) show the 980×664 top part of two faceimages in the database. We globally register the top part of theface images using the Lucas-Kanade algorithm. In the secondstep, we extract the 476×232 skin regions from the globallyregistered face images, and register them using [11]. Figure

Fig. 4. Sample images of a person with acne lesions. Note that the profileof acne lesions changes significantly with time.

Fig. 5. Top part of two face images in the database.

6 (a) shows the forehead regions extracted from the top partof the face images. Figure 6 (b) shows the pixel locations ofthe corresponding feature points in the input skin regions, andFigure 6 (c) shows the pixel locations in the globally registeredimages. Note that in the globally registered images the pixellocations of corresponding feature points do not preciselyoverlap indicating residual misregistration. Figure 6 (d) showsthe pixel locations of the corresponding feature points afterremoving the residual misregistration. Note that in Figure 6(d) the pixel locations precisely overlap indicating accurateregistration. We have included time-lapse videos showing the39 registered time-lapse images, and the 39 misregistered time-lapse images. The resulting time-lapse video of skin, to thebest of our knowledge, is the first of its kind and impressivelydemonstrates the evolution of lesions with time.

IV. DETECTING ACNE-LIKE REGIONS IN SKIN IMAGES

We model acne-like regions using a six dimensional fea-ture vector representing temporal and spatial variations, andclassify using logistic regression [6]. Note that computingthe spatio-temporal features requires a precise point to point

(a)

(b)

(c) (d)

Fig. 6. (a): Two of the 39 skin images with acne lesions acquired duringthe three month clinical study. (b): Pixel locations of corresponding featurepoints in the input images. (c): Pixel locations of corresponding feature pointsin the globally registered images. (d): Pixel locations of corresponding featurepoints in the final precisely registered forehead regions.

(a)

(b)

(c)

Fig. 7. (a): Forehead region with acne-like regions marked in white and non-acne regions marked in black. (b): Acne-like regions marked in the foreheadregion. (c): Non-acne regions marked in the forehead region.

registration of the skin regions.Figure 7 shows an example forehead region in the dataset

and sample 30×30 acne-like and non-acne regions in theforehead region. The acne-like regions are marked in whiteand the non-acne regions are marked in black. The first ordertemporal difference image, Id(x, y), for a region extractedfrom a time-lapse image acquired at time t is defined as,

Id(x, y) = I ′(x, y) − I(x, y), (4)

where I ′(x, y) is the region image extracted from the time-lapse image acquired at time t + 1 and I(x, y) is the regionimage extracted from the time-lapse image acquired at time t.Figure 8 (a) and (b) show the first order temporal differenceimages for the acne-like regions and non-acne regions inFigure 7 (b) and (c); note that the temporal difference imagesfor acne-like regions have significantly higher intensity values.The temporal differences captures the variation with time

Fig. 8. (a): Difference images for acne-like regions shown in Figure 7 (b).(b): Difference images for non-acne regions shown in Figure 7 (c). Note thatthe intensity values are generally higher in the difference images for acne-likeregions.

TABLE ISPATIAL STANDARD DEVIATIONS

Red Green Blue7.06 11.63 9.048.33 12.18 9.945.71 9.85 7.294.88 8.96 6.083.30 4.19 4.084.63 4.55 4.273.54 4.05 4.143.85 6.90 5.83

of acne-like regions; therefore we select the mean intensityvalues, mR, mG, and mB in the red, green, and blue channelsof the first order temporal difference image as the first threecomponents of feature vector. The mean intensity value mR

in the red channel is defined as,

mR =1

N

∑x,y

IdR(x, y), (5)

where N is the number of pixels in the region and IdR(x, y)is the intensity of the red channel at pixel location(x, y) inthe first order temporal image. mG and mB are defined in asimilar way. We use spatial standard deviations σR, σG, andσB in the red, green, and blue channels to capture the intensityprofile in acne-like regions. The spatial standard deviation σRin the red channel is defined as,

σR =

√1

N

∑x,y

(IR(x, y) − IR)2, (6)

where N is the number of pixels in the region, IR(x, y) isthe intensity of the red channel at pixel location (x, y), andIR is the mean intensity of the red channel in the region.Spatial standard deviations σG and σB in the green and theblue channels are defined in a similar way. Table I shows thespatial standard deviations in the red, green, and blue channelsfor the patch images shown in Figure 7 (b) and 7 (c). The firstfour lines show the standard deviations for acne-like regionsand the last four lines show the standard deviations for non-acne regions. Note that the spatial standard deviations aregenerally higher in acne-like patch images, which indicates

(a) (b)

(c) (d)

(e)

Fig. 9. Training set feature vectors. In each plot two components of thefeature vectors are plotted together. (a,b): Temporal mean of red-green andgreen-blue channels. (c): Temporal mean of blue channel and spatial standarddeviation of red channel. (d,e): Spatial standard deviation of red-green andgreen-blue channels. White points correspond to acne-like regions and blackpoints correspond to non-acne regions.

that spatial standard deviation can be used to classify acne-like regions. We select the spatial standard deviations in thered, green, and blue channels as the last three componentsof the feature vector. The final feature vector x is defined asx = [mR,mG,mB , σR, σG, σB ]T .

Figure 9 shows the feature space representation of regionsextracted from the chin region. In Figure 9, each plot showstwo components of the six dimensional feature space. Thepoints corresponding to acne-like regions are shown in whiteand the points corresponding to non-acne regions are shownin black. Note that in the feature space, the acne-like re-gions are reasonably well separated from non-acne regionseven in the two dimensional projection, and the separatingboundary can be modeled using a hyperplane. We learn theseparating hyperplane between acne and non-acne regions inthe six dimensional feature space using the logistic regressionclassifier [6]. We train the classifier on 100 30×30 acne/non-acne patches from the chin region, and test on 250 30×30acne/non-acne patches from the forehead regions.

V. CLASSIFICATION RESULTS FOR ACNE-LIKE/NON-ACNEREGIONS

We have trained the logistic regression classifier on 100,50 acne-like and 50 non-acne, 30×30 patches extracted fromthe chin region, and tested on 250, 125 acne-like and 125non-acne, 30×30 patches extracted from the forehead region.Table II summarizes the classification results for the training

TABLE IITRAINING STAGE RESULTS

Total regions: 50 (acne-like) + 50 (non-acne) = 100Correctly classified regions: 97/100 = 97%Correctly classified acne-like regions: 49/50 = 98%Correctly classified non-acne regions: 48/50 = 96%

TABLE IIITESTING STAGE RESULTS

Total regions: 125 (acne-like) + 125 (non-acne) = 250Correctly classified regions: 223/250 = 89.2%Correctly classified acne-like regions: 113/125 = 90.4%Correctly classified non-acne regions: 110/125 = 88%

stage, and table III summarizes the classification results forthe testing stage. In the testing stage, we obtained an overall89.2% success rate in classifying acne-like/non-acne regions.The high success rate in the testing stage implies that localtemporal and spatial variations we can be used to successfullyclassify acne-like regions.

VI. CONCLUSION

In this paper we presented a novel algorithm to detect acnelesions using images acquired under cross-polarized modality.Automatic lesion detection enables computer assisted countingof acne lesions in skin images, which increases the accuracyand reduces the time and effort of the counting process. Infuture, we aim to extend the work to classify different typesof acne lesions: comodones, papules, pustules, infiltrates, andcysts. Every acne lesion type has a different life-cycle, andresponds differently to acne treatment. Assessing the responseof a treatment procedure to different types of acne lesionsrequires the counting of each type of acne lesion. We believethat our research would have significant impact in the studyof acne and the methods used to treat acne.

APPENDIXTIME-LAPSE VIDEOS OF SKIN IMAGES

Figure 10 and 11 show the 39 images used to generatethe time-lapse video. Videos showing the misregistered andthe registered set of time-lapse images are available at http://www.ece.rutgers.edu/∼kdana/Research.html. Figure 12 (a)shows the mean of all the images in the time-lapse set beforeregistration and Figure 12 (b) shows the mean of all the imagesin the time-lapse set after registration.

REFERENCES

[1] R. Anderson, “Polarized light examination and photography of the skin,”Archives of Dermatology, vol. 127, no. 7, pp. 1000–1005, 1991.

[2] S. Baker and I. Matthews, “Lucas–kanade 20 years on: A unifyingframework,” International Journal on Computer Vision, vol. 56, no. 3,pp. 221–255, 2004.

[3] J. Christiansen, P. Holm, and F. Reymann, “Treatment of acne vulgariswith the retinoic acid derivative ro 11-1430. a controlled clinical trialagainst retinoic acid,” Dermatologica, vol. 153, no. 3, pp. 172–176,1976.

[4] G. O. Cula, P. R. Bargo, and N. Kollias, “Imaging inflammatory acne:lesion detection and tracking,” in SPIE 7548, 75480I, 2010.

t2

t3

t1

t0

t20

t19

t18

Fig. 10. First 21 of the 39 time-lapse skin images with acne lesions.

t23

t24

t22

t21

t38

t37

t36

Fig. 11. Last 18 of the 39 time-lapse skin images with acne lesions.

[5] H. Fuji, T. Yanagisawa, M. Mitsui, Y. Murakami, M. Yamaguchi,N. Ohyama, T. Abe, I. Yokoi, Y. Matsuoka, and Y. Kubota, “Extractionof acne lesion in acne patients from multispectral images,” in 30thAnnual international IEEE EMBS conference, 2008, pp. 4078–4081.

[6] T. Hastie, R. Tibshirani, and J. Friedman, The elements of statisticallearning: data mining, inference,and prediction. Springer, 2001.

[7] S. Jacques, J. Ramella-Roman, and K. Lee, “Imaging skin pathologywith polarized light,” Journal of Biomedical Optics, vol. 7, no. 3, pp.329–340, 2002.

[8] S. L. Jacques, J. C. Ramella-Roman, and K. Lee, “Imaging superficialtissues with polarized light,” Lasers in Surgery and Medicine, vol. 26,no. 2, pp. 119–129, 2000.

[9] N. Kollias, Bioengineering of the Skin. CRC Press, 1997, ch. 7, pp.95–104.

[10] N. Kollias and G. N. Stamatas, “Optical non-invasive approaches fordiagnosis of skin diseases,” Journal of Investigative Dermatology, no. 7,

(a) (b)

Fig. 12. Mean of all the images in the time-lapse set before registration (a)and after registration (b). Notice that with registration the mean image showsthe aggregate set of lesions over time.

pp. 64–75, 2002.[11] S. Madan, K. J. Dana, and O. G. Cula, “Quasiconvex alignment of

multimodal skin images for quantitative dermatology,” in ComputerVision and Pattern Recognition Workshop, 2009, pp. 117–124.

[12] S. Nayar, X. Fang, and T. Boult, “Removal of specularities using colorand polarization,” in IEEE Conference on Computer Vision and PatternRecognition, 1993, pp. 583–590.

[13] S. Sigurdsson, P. A. Philipsen, L. K. Hansen, J. Larsen, M. Gniadecka,and H. C. Wulf, “Detection of skin cancer by classification of ramanspectra,” IEEE Transactions on Biomedical Engineering, vol. 51, no. 10,pp. 1784–1793, 2004.

[14] R. C. N. Studinski and I. A. Vitkin, “Methodology for examiningpolarized light interactions with tissues and tissue like media in theexact backscattering direction,” Journal of Biomedical Optics, vol. 5,no. 3, pp. 330–337, 2000.

[15] J. Witkowski and H. Simons, “Computer-assisted alignment and trackingof acne lesions indicate that most inflammatory lesions arise from come-dones and de novo,” Journal of the American Academy of Dermatology,vol. 58, no. 4, pp. 603–608, 2008.

[16] J. A. Witkowski and H. M. Simons, “Objective evaluation ofdemethylchlortetracycline hydrocloride in treatment of acne,” The jour-nal of the American Medical Association, vol. 196, no. 5, pp. 397–400,1966.