deep learning techniques to classify scanning electron … · 2018-11-21 · and where all proposal...
TRANSCRIPT
![Page 1: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/1.jpg)
NFFA-Europe has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654360
Deep Learning techniques to classify Scanning Electron Microscope (SEM) images at the nanoscalethe NFFA case study
S. Cozzini, R. Aversa, C. De Nobili, A. Chiusole, G.B Brandino
CNR – IOM / eXact lab srl
![Page 2: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/2.jpg)
Agenda
• Introduction: NFFA-EUROPE project
• Data Repository for NFFA-EUROPE project
• Classify Scanning Electron Microscope (SEM) images at the nanoscale.
• Conclusions & perspectives
![Page 3: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/3.jpg)
NFFA-Europe has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654360
EU funded project
it provides the widest range of tools for research at the nanoscaleFree transnational access to academia & industry
www.nffa.eu
![Page 4: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/4.jpg)
The consortium
20 partners of which 10 nanofoundries co-located with Analytical Large Scale facilities
Coordinated by CNR-IOM
![Page 5: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/5.jpg)
The offer
TA Transnational Access activities
Multidisciplinary research at the nanoscale performed at nano-laboratories and ALSFs
Integration of theory & numerical analysis with advanced characterization
NA Networking activities
Interface for different user communities
Industrial exploitation of experimental data
JRAJoint Research activities
Methods & tools at the frontier in nanoscience research
Improved infrastructures for academic & industrial projects
![Page 6: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/6.jpg)
NFFA Data management
JRA3:
e-Infrastructure for data and information management
A transversal activity devoted to the setup of the first Information and Data Repository Platform(IDRP) for Nano science
• Definition of new metadata standards for data sharing in nanoscience
• Automatic acquisition of key metadata and create a
data repository for future data access
Data infrastructure is complemented by Data Analysis Services.
![Page 7: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/7.jpg)
NFFA IDRP architecture
Nanoscience Foundries and Fine Analysis – EUROPE
26
further develop the lithography and pattern transfer technologies towards the controlled and reproducible
fabrication of sub-10 nm structures in surfaces over an area large enough to perform meaningful scientific
experiments and to further exploit possible application of these patterning technologies in several test areas: 1)
semiconductor quantum-dots for advanced IT technologies, 2) bioanalytic devices, where the specific interaction of
biomolecules with surface patterns matching their size could result in exciting new scientific insights, 3) patterned
surfaces serving as model catalysts in heterogeneous catalysis (see also JRA2). Users will immediately benefit from
the results.
PSI(coordinator, CH), PRUAB(ES), TUG(AT), DESY(DE), CNRS(FR), FORTH(EL)
WP8: Research on e-infrastructure for data and information management Effective archiving and providing
of open access to scientific data repositories are challenges that require novel data management methods. Suitable
data formats for interoperability are mandatory as well as an effective link to metadata. The previous work under
NFFA-Design Study identified the relevant information that is needed for data analysis and represents the
background of the present JRA that will develop and deploy the IDRP as a key item of the NFFA-EUROPE
infrastructure; it will be operated by users (those who generate the data) and it will be accessed by “third party
users” (those who access the NFFA-EUROPE repository under suitable rules and conditions). The whole
information and data management process handled within the NFFA-EUROPE infrastructure will then be done on
the IDRP. A new entry will be added to the IDRP when a new proposal is received, generating a work-place where
all relevant information on users, samples, previously acquired results, and related literature is uploaded or linked
and where all proposal evaluation steps and scheduling of access to the NFFA-EUROPE facilities are recorded.
Manual data entries by the users will be necessary, but limited to an overall effort lower than conventional log-
booking.
The access to the experiment work-place will be extended to the users and to their collaborators in e-access/remote
mode, so that remote data analysis or complementary work (simulations, calculations, other) can be performed in
an interactive mode in real time. The system will develop modules for remote-operation. All relevant data and
metadata of all the content of the proposal’s “work-place” will be archived in the open-access IDRP. The access
rules to the IDRP will imply some limitations, and will require proper acknowledgement of the source of the data
and metadata, including the scientists and institutions who did the work and who should agree on its further use by
the “third-party” users. IDRP will also provide on line data analysis services, adopting a similar approach as that
proposed also by the PaNDAS project which will be submitted to the H2020 INFRADEV-4 call. A collaboration
and an exchange of ideas and techniques is ongoing among the two proposals.
Easy data access from all facilities and via the NFFA portal for all NFFA users
![Page 8: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/8.jpg)
NFFA IDRP deployment
Nanoscience Foundries and Fine Analysis – EUROPE
26
further develop the lithography and pattern transfer technologies towards the controlled and reproducible
fabrication of sub-10 nm structures in surfaces over an area large enough to perform meaningful scientific
experiments and to further exploit possible application of these patterning technologies in several test areas: 1)
semiconductor quantum-dots for advanced IT technologies, 2) bioanalytic devices, where the specific interaction of
biomolecules with surface patterns matching their size could result in exciting new scientific insights, 3) patterned
surfaces serving as model catalysts in heterogeneous catalysis (see also JRA2). Users will immediately benefit from
the results.
PSI(coordinator, CH), PRUAB(ES), TUG(AT), DESY(DE), CNRS(FR), FORTH(EL)
WP8: Research on e-infrastructure for data and information management Effective archiving and providing
of open access to scientific data repositories are challenges that require novel data management methods. Suitable
data formats for interoperability are mandatory as well as an effective link to metadata. The previous work under
NFFA-Design Study identified the relevant information that is needed for data analysis and represents the
background of the present JRA that will develop and deploy the IDRP as a key item of the NFFA-EUROPE
infrastructure; it will be operated by users (those who generate the data) and it will be accessed by “third party
users” (those who access the NFFA-EUROPE repository under suitable rules and conditions). The whole
information and data management process handled within the NFFA-EUROPE infrastructure will then be done on
the IDRP. A new entry will be added to the IDRP when a new proposal is received, generating a work-place where
all relevant information on users, samples, previously acquired results, and related literature is uploaded or linked
and where all proposal evaluation steps and scheduling of access to the NFFA-EUROPE facilities are recorded.
Manual data entries by the users will be necessary, but limited to an overall effort lower than conventional log-
booking.
The access to the experiment work-place will be extended to the users and to their collaborators in e-access/remote
mode, so that remote data analysis or complementary work (simulations, calculations, other) can be performed in
an interactive mode in real time. The system will develop modules for remote-operation. All relevant data and
metadata of all the content of the proposal’s “work-place” will be archived in the open-access IDRP. The access
rules to the IDRP will imply some limitations, and will require proper acknowledgement of the source of the data
and metadata, including the scientists and institutions who did the work and who should agree on its further use by
the “third-party” users. IDRP will also provide on line data analysis services, adopting a similar approach as that
proposed also by the PaNDAS project which will be submitted to the H2020 INFRADEV-4 call. A collaboration
and an exchange of ideas and techniques is ongoing among the two proposals.
IDRP
KIT-DM@CNR
B2SHARE EUDAT
SERVICE
Materialcloud@EPFL
![Page 9: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/9.jpg)
A case study: classifying SEM images by Neural network
![Page 10: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/10.jpg)
Our Issue: SEM images
• One SEM Available at CNR-IOM Trieste with 150,000 images NOT classified
• 10 SEM across European partners: the work can be exported to a sizeable part of the community
![Page 11: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/11.jpg)
Sharing images is nice..
• A couple of million nano images can be of some help for some nanoscience..
But before doing that we need to start classifying them…
![Page 12: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/12.jpg)
SEM images classification steps
• STEP1: Classify images (scientific skills)
• STEP2: Train a neural network (deep learning)
• STEP 3: Use the network as classifier (inference)• Semi - Automatic tool for SEM users
• Massive process of all the images
• Specific task in nano science: wires alignment
![Page 13: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/13.jpg)
We created and manually annotated the first dataset of classified SEM images (18,577 images).
Aversa et al., in preparation
Step1: classify images..
![Page 14: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/14.jpg)
Step 2: train the network !
![Page 15: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/15.jpg)
Step2 : the tools/infrastructure…
![Page 16: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/16.jpg)
Step2: The deep Learning network..
• Models:• AlexNet
• Inception-v3/v4
• Densenet
• Deep learning techniques:• Training from scratch
• Transfer learning (Feature extraction, Fine Tuning)
• Deep learning frameworks:• TensorFlow
• Neon/Nervana
![Page 17: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/17.jpg)
Glossary
• Supervised learning: labelled examples
• Transfer learning: applying knowledge of a trained network to a new domain
• Check point: set of parameters saved at a certain point of the training
• Feature Extraction: previous layers frozen to the check point + last layer(s) randomly initialized
• Train from scratch: all the parameters of all the layers are randomly initialized
• Fine tuning: all the parameters are initialized to the last check point and are allowed to vary
![Page 18: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/18.jpg)
Feature extraction:ImageNet Checkpoint
![Page 19: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/19.jpg)
Training from Scratch: alexnet
![Page 20: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/20.jpg)
Training from Scratch: inc-v4
![Page 21: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/21.jpg)
Training from Scratch: inc-v3
![Page 22: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/22.jpg)
Which is the best ?
![Page 23: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/23.jpg)
What about batch size ?
![Page 24: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/24.jpg)
Densenet vs Inception..
![Page 25: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/25.jpg)
Densenet vs Inception
![Page 26: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/26.jpg)
Step3: Data Analysis services:
![Page 27: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/27.jpg)
sem-classifier.nffa.ue
![Page 28: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/28.jpg)
Data analysis service
• A nanoscience task: mutually coherent alignment of nanowires
• Alignments score comes by ML classification
![Page 29: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/29.jpg)
Conclusions&perspective
• A distributed Data management infrastructure for NFFA-EUROPE up&running
• We applied the deep learning technique to train an automatic image classification engine for Nano images to provide Data Analysis services on the top of the infrastructure
• A full automated procedure has been then setup to automatically annotate SEM images once user load them on the KTDM&IDRP..
• Metadata generated automatically thanks to ML
• A pandora box was open for many different interesting nanoscience problems to be solved with the help of deep learning techniques to complement NFFA-EUROPE IDRP with data analysis services.
![Page 30: Deep Learning techniques to classify Scanning Electron … · 2018-11-21 · and where all proposal evaluation steps and scheduling of access to the NFFA -EUROPE facilities are recorded](https://reader033.vdocuments.net/reader033/viewer/2022042204/5ea573efaa049671962bbec4/html5/thumbnails/30.jpg)
References
• R. Aversa, “Scientific Image Processing within the NFFA-EUROPE Project”, MHPC thesis, 16-12-2016
• C. De Nobili, “Deep Learning for Nanoscience Scanning Electron Microscope Image Recognition”, MHPC thesis, 18-12-2017
• H.M. Modarres, R. Aversa, S. Cozzini, et al., “Neural Network for Nanoscience Scanning Electron Microscope Image Recognition”, Scientific Reports 7, 13282(2017)
• R. Aversa, S. Cozzini, “The first annotated set of Scanning Electroscope Microscopy images”, in preparation
• Web classifier: sem-classifier.nffa.eu