arX

iv:1

208.

5365

v1 [

cs.C

V]

27 A

ug 2

012

1

A Missing and Found Recognition Systemfor Hajj and Umrah

Salah A. Aly

Center of Research Excellence in Hajj and Umrah (HajjCore),College of Computers and Information Systems,

Umm Al-Qura University, Makkah, KSAsalahaly@uqu.edu.sa

Abstract—This note describes an integrated recognition systemfor identifying missing and found objects as well as missing, dead,and found people during Hajj and Umrah seasons in the two Holycities of Makkah and Madina in the Kingdom of Saudi Arabia.It is assumed that the total estimated number of pilgrims willreach 20 millions during the next decade. The ultimate goal of thissystem is to integrate facial recognition and object identificationsolutions into the Hajj and Umrah rituals. The missing and foundcomputerized system is part of the CrowdSensing system for Hajjand Umrah crowd estimation, management and safety.1

I. I NTRODUCTION

According to the Royal Embassy of Saudi Arabia, morethan 10 million pilgrims (foreign and local) arrived to theHoly city of Makkah to perform Hajj and Umrah in 2011,among them, five million pilgrims performed Umrah duringRamadan and three million pilgrims performed Hajj. The totalestimated number is expected to reach 20 million pilgrimsper year during the next decade. Due to the huge crowd thatoccurs in the two Holy cities of Makkah and Madinah, severalquestions arise such as: How to identify missing, dead, andfound people, and how to collect and distribute missing andfound objects?

One of the most crucial steps in the CrowdSensing system isto detect individuals’ faces in an image and run a facial recog-nition algorithm through a database of registered pilgrims[4].Therefore, the proposed approach is more focused on devel-oping a pattern detection algorithm that does not depend onthe three-dimensional complex data of faces, but it dependsonthe general outer Silhouette that is almost shared between allfaces. The system goal is to develop a computational modelfor face detection and recognition that is reasonably simpleand acceptably accurate under various conditions of facialexpressions, and various background environments.

The collected data regarding missing or found objects andpeople will be available throughout all year, and can beaccessed online immediately from a PC or mobile device.The online missing and found database will be supported bya smart search engine for various search criterias. Also, thesystem can be deployed for border protection against illegalimmigrants.

1———————————————————————————Thanks to HajjCoRE, Center of Research Excellence in Hajj and Umrah

at UQU, an agency for supporting this work.

Fig. 1: A proposed recognition system for missing and foundobjects during Hajj and Umrah seasons.

In this note, we describe a CrowdSensing system with twofolds [1], [2]:

• Identifying missing, dead, and found people during over-crowds using facial detection and recognition,

• Collecting missing and found objects using a web-portalsolution.

II. A W EB-PORTAL SYSTEM FORM ISSING AND FOUND

ITEMS

During overcrowds in the two Holy cities during Hajj andUmrah, thousands of items are found and lost when movingaround and getting-in sacred places. The recognition systemconsists of the following parts, see Fig. 1:

1) Boxes to collect found objects or personal items inMakkah and Madina. The Boxes can be placed in themain intersection roads or near-by the two Holy Mosques.

2) A touch screen system to be located in streets and publicplaces to access the web-portals.

3) A PC, IPAD, or mobile device with Android system toaccess the web-portal.

4) A main server to control and manage the system.5) A database engine to support all missing and found

objects or personal items.6) Administrator and alerting system to take actions in cases

of emergences.

2

Fig. 2: MFHajj portal interface website developed by theCrowdsensing.net team. The system consists of data collectors(mobiles, cameras, PCs), main server, search engine, and analerting system. The system is used to recognize missing andfound people during Hajj and Umrah seasons

We describe a MFHajj example as follows. Assume anobject (watch or mobile device) is found. A description ofthis object along with a captured image will be entered intothe web-portal via a mobile device supporting Android or byusing one of the touch screens, which are located in the streets.Immediately, the information about this found object will beavailable online in the web-portal system, a search engine andobject identification is supported in this system. The objectowner can claim at any time, an evidence is given.

III. A S MART SYSTEM FORRECOGNIZING M ISSING,DEAD, AND FOUND PEOPLE

During the overcrowds in Makkah and Madina, thousandsof pilgrims are missing from their group members or can befound without any IDs. Furthermore, the majority of them donot know their location or the correct direction to their hotelrooms. The system for identifying missing, dead, and foundpeople consists of several parts as described below [2], seeFig. 2:

1) A computer server to process the user’s queries and to runthe developed facial detection and recognition algorithms.The server will run a web-portal solution.

2) A database search engine with all pilgrims’ images [3].Such images can be easily captured at the KSA bordersor in airport’s entrance. Images of the local pilgrims canbe collected when their Hajj permit cards are given.

3) A mobile device supporting Android system that can sendand receive online images, videos, and texts.

4) A touch screen with webcam, which can be located inthe main intersection roads or public places. Such a touchscreen can be similar to ATM machines.

5) Administrator and alerting system to take actions in casesof emergences.

The HUDA contains images taken during the 2011-2012Hajj and Umrah seasons of a large number of pilgrims (various

Fig. 3: MFHajj portal interface website developed by Crowd-sensing.net team. The system consists of data collectors (mo-biles, cameras, PCs), main server, search engine, and analerting system.

races). It contains at least six images for each individual,in a varied range of poses, facial expressions (open andclosed eyes, smiling/ not smiling), facial details (glasses/ noglasses), in different lighting conditions, and against randombackgrounds. All images are in full-color JPG format, seeFig. 5.

For example, if a person is found alive or dead, his/herimage can be taken by a mobile device supporting AndroidSystem. After that a query will be sent to the computerserver to run the facial recognition algorithm and searchingin the pilgrims database, it will be conduced to watch hisface, then extract his/her identity. Thereafter, a reply messageimmediately will be sent to the mobile device.

A. Android and Software Portal Application

We developed a software portal application to be installedin all mobile devices supporting Android System. When theapplication runs in a user’s mobile, an image will be takenand sent to a computer server for analysis. The server willconsult the database system and will run facial recognitionand detection algorithms. Our developed software will use animage search engine as shown in Fig. 3. Also, an alertingsystem is used for an authority assistance. The software andportal applications are available in our project website shownin [1].

B. Hajj and Umrah Face Recognition Dataset

The HUDA contains images taken during the 2011-2012Hajj and Umrah seasons of a large number of pilgrims (variedraces and appearances) [1], [3]. It contains at least 3 images foreach individual in a varied range of poses, facial expressions(open / closed eyes, smiling / not smiling) facial details(glasses / no glasses), in different lighting conditions, andagainst random backgrounds. All the images are in full-colorJPG format.

3

Fig. 4: Samples of Hajj and Umrah Dataset. One pilgrim imageis stored in the MFHajj database and can be used in the searchengine. Another different pilgrim image is taken to performmissing and found search process.

IV. CROWDSENSING SYSTEM

The CrowdSensing system is established to support theexisting efforts to manage the overcrowds and solve themissing and found problem during Hajj and Umrah seasonsin the two Holy cities of Makkah an Madina. The goal of thisCrowdSensing system is to deploy techniques from ComputerVision and Image Processing to develop a portal website forHajj and Umrah missing and found people [1]. The systemrequires all pilgrims to register their personal data when theyplan to perform Hajj or Umrah. Such info can be collectedeasily when pilgrims cross the KSA borders.

One application of the proposed dataset is the CrowdSens-ing System, which consists of three main components, seeFig. 3:

1) A database of all individuals who arrive to the kingdom toperform the holy rituals. This database contains all theirpersonal information along with a personal photo, and itcan be updated via our web portal.

2) Advanced monitoring cameras scattered around the GrandMosque in Makkah, airports, hospitals, and all areas ofinterest.

3) Our proposed face detection & recognition algorithm isto be used for acquiring faces from images captured bythe monitoring cameras and use them to identify missingand found individuals [1].

There are several advantages of CrowdSensing system, someof which:

1) Estimating and counting crowds.2) Identifying missing, dead, and found people via face

detection and recongition.3) Identifying missing and found items.4) Tracking pedestrians, and compute flow and density of

crowds.5) Simulating crowds and movements.6) Recognizing empty spaces inside and outside El-Harram.7) Recognizing human event activities and event classifica-

tions.8) Collecting data and gathering information.

V. D ISCUSSION ANDCONCLUSION

In this note, we presented a computerized recognitionsystem to solve the missing and found problem in Makkahand Madina during Hajj and Umrah seasons. A web-portalprototype is developed and has been tested in 2011 Hajjseason [2].

ACKNOWLEDGMENTS

This research is funded by a grant number 11-nan1707-10from the Long-Term National Plan for Science, Technologyand Innovation (LT-NPSTI), the King Abdulaziz City forScience and Technology (KACST)- Kingdom of Saudi Arabia.We thank the Science and Technology Unit at Umm A-QuraUniversity for their continued logistics support.

REFERENCES

[1] www.crowdsensing.net. Copyright, 2011.[2] www.mfhajj.com. InCrowdSensing Project, Copyright, 2011.[3] S. A. Aly. Hajj and umrah dataset for face recognition anddetection.

May 2012. arXiv:1205.4463v1.[4] Hossam Zawbaa and Salah A. Aly. Hajj and umrah event recognition

datasets. Copyright Hajjcore, Umm Al-Qura University, 2012. Available:arXiv:1205.2345v1 [cs.CV].

1 2 3 4 5 6 7

1

2

3

4

5

6

7

8

9

10

11

Fig. 5: A sample of Hajj and Umrah Dataset.