DeCoRe project-team (ADM) 2016-2019Deep Convolutional and Recurrent networks for image, speech & text

I Success of deep learning in vision, speech, NLP, . . .I Many processing layers from raw input signal upwardsI All parameters trained jointly in end-to-end mannerI Why now: Data, computation, training algorithms

I Areas of research in DeCoreI Multi-modal data: modeling relation images and textI Visual recognition: many classes, incremental learningI Time series with multiple resolutions and missing dataI Automating network architecture designI Processing non-regular data: 3D shapes, molecular graphs, etc.

I Who?I Organizers: L. Besacier, D. Pellerin, G. Quenot, J. VerbeekI Labs: GIPSA, LIG, LJKI Teams: AGPIG, AMA, GETALP, MRIM, SigmaPhy, THOTH

1 / 2

PhD theses funded by DeCoRe

I Anuvabh Dutt: Object recognition andlocalization

I Supervision: D. Pellerin & G. QuenotI Hierarchical concept recognitionI Network architecture evolution for

incremental learningI Concept-based multimedia retrieval

I Maha Elbayad: Natural language imagedescription

I Supervision: L. Besacier & J. VerbeekI CNN-RNN design from pixels to wordsI Increase diversity in generated captionsI Visual attention for compositionality

Search in videos: ”Guitar

and Hand”

Output: “A young boy is

holding a cell phone.” 2 / 2