Jan Sedmidubsky September 23, 2014Face Retrieval for Security Applications
Dr. Jan Sedmidubsky
Prof. Pavel Zezula
Face Retrieval
for Security Applications
Laboratory of Data Intensive Systems and Applications
http://disa.fi.muni.cz/
Faculty of Informatics, Masaryk University
Brno, Czech Republic
1/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications 2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
Extraction of
face features
<14.4, 8.2, -8.4, 704.0, …>
<12.6, 9.7, -6.1, 654.0, …>
<17.9, 12.1, -9.1, 692.0, …>
<8.8, 7.7, -3.5, 570.8, …>
2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
Similarity
Search
Technology
Extraction of
face features
<14.4, 8.2, -8.4, 704.0, …>
<12.6, 9.7, -6.1, 654.0, …>
<17.9, 12.1, -9.1, 692.0, …>
<8.8, 7.7, -3.5, 570.8, …>
Indexing
face features
2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
Similarity
Search
Technology
Extraction of
face features
<14.4, 8.2, -8.4, 704.0, …>
<12.6, 9.7, -6.1, 654.0, …>
<17.9, 12.1, -9.1, 692.0, …>
<8.8, 7.7, -3.5, 570.8, …>
Indexing
face features
<13.9, 9.5, -6.0, 712.1, …>
2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
Similarity
Search
Technology
Extraction of
face features
<14.4, 8.2, -8.4, 704.0, …>
<12.6, 9.7, -6.1, 654.0, …>
<17.9, 12.1, -9.1, 692.0, …>
<8.8, 7.7, -3.5, 570.8, …>
Indexing
face features
<13.9, 9.5, -6.0, 712.1, …>
Similarity search
2/15
Jan Sedmidubsky
Face Recognition Technology
September 23, 2014Face Retrieval for Security Applications
Detection
of faces
Similarity
Search
Technology
Extraction of
face features
<14.4, 8.2, -8.4, 704.0, …>
<12.6, 9.7, -6.1, 654.0, …>
<17.9, 12.1, -9.1, 692.0, …>
<8.8, 7.7, -3.5, 570.8, …>
Indexing
face features
<13.9, 9.5, -6.0, 712.1, …>
Similarity search
Visualization of the
most similar faces
0.12 0.17 0.18 0.23
2/15
Jan Sedmidubsky
Face Recognition Technology:
Components
• Components of general face recognition technology
1) Face detection function
• Localizes bounding boxes of faces within a static image
2) Feature extraction function
• Extracts a characteristic feature of a face image
3) Face recognition function
• Computes similarity score between features of two faces
September 23, 2014Face Retrieval for Security Applications
<13.9, 9.5, -6.0, 712.1, …>
<13.9, 9.5, -6.0, 712.1, …>
0.12
<15.2, 8.8, -5.9, 707.8, …>
(3)
(1)
(2)
3/15
Jan Sedmidubsky
Our Face Recognition Technology
• Our objective – to effectively and efficiently
retrieve the most similar faces to a query face(s)
• Improvement of detection/recognition components:
1) Face detection function – combining multiple state-ofthe-art
approaches for more effective detection
2) Feature extraction func. – aggregating multiple features
3) Face recognition function
• Combining multiple approaches for more effective recognition
• Multi-face query evaluation for more effective retrieval
• Pre-selection of candidate set and its re-ranking for more
efficient retrieval
September 23, 2014Face Retrieval for Security Applications 4/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications 5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications 5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications 5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications 5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Aggregation
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications
Union aggregation
recall: 100%
precision: 60%
5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Aggregation
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications
Union aggregation
recall: 100%
precision: 60%
Intersection aggregation
recall: 33%
precision: 100%
5/15
Jan Sedmidubsky
• Aggregation of existing detection approaches to
increase recall/precision
Aggregation
Our Face Recognition Technology:
Face Detection
September 23, 2014Face Retrieval for Security Applications
Union aggregation
recall: 100%
precision: 60%
Intersection aggregation
recall: 33%
precision: 100%
2/3 aggregation
recall: 66%
precision: 100%
5/15
Jan Sedmidubsky
Our Face Recognition Technology:
Face Detection (cont.)
• Current implementation:
– Aggregation of at least 2 approaches out of 3
• OpenCV – open source from Computer vision library
• FaceSDK – commercial software from Luxand
• VeriLook SDK – commercial software from Neurotechnology
September 23, 2014Face Retrieval for Security Applications
Low-quality face images
# of faces: 1000
High-quality face images
# of faces: 10,000
Recall Precision Recall Precision
OpenCV 55 89 92 86
FaceSDK 63 83 95 94
VeriLook SDK 73 84 100 96
Our aggregation 62 98 97 100
6/15
Jan Sedmidubsky
• Utilization of existing extraction approaches to
characterize a face by a set of different features
• Current version integrates three approaches of
MPEG-7, FaceSDK and VeriLook SDK features
Our Face Recognition Technology:
Feature Extraction
September 23, 2014Face Retrieval for Security Applications
Extraction
MPEG-7 feature
0.25kB float vector: <13.9, 9.5, -6.0, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
Cameron Diaz
always
present
present if
extraction is
successful
present if a user
annotates the
face
7/15
Jan Sedmidubsky
• Similarity of two faces is computed as aggregation
of normalized inner similarities between features
Our Face Recognition Technology:
Face Recognition
September 23, 2014Face Retrieval for Security Applications
MPEG-7 feature
0.25kB float vector: <13.9, 9.5, -6.0, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
Cameron Diaz
MPEG-7 feature
0.25kB float vector: <15.1, 8.6, -5.4, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
8/15
Jan Sedmidubsky
• Similarity of two faces is computed as aggregation
of normalized inner similarities between features
Our Face Recognition Technology:
Face Recognition
September 23, 2014Face Retrieval for Security Applications
MPEG-7 feature
0.25kB float vector: <13.9, 9.5, -6.0, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
Cameron Diaz
MPEG-7 feature
0.25kB float vector: <15.1, 8.6, -5.4, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
118.6
0.06
2543
Inner
similarities
8/15
Jan Sedmidubsky
• Similarity of two faces is computed as aggregation
of normalized inner similarities between features
Our Face Recognition Technology:
Face Recognition
September 23, 2014Face Retrieval for Security Applications
MPEG-7 feature
0.25kB float vector: <13.9, 9.5, -6.0, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
Cameron Diaz
MPEG-7 feature
0.25kB float vector: <15.1, 8.6, -5.4, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
118.6
0.06
2543
Inner
similarities
0.12
0.11
0.06
Normalization
into [0, 1]
8/15
Jan Sedmidubsky
• Similarity of two faces is computed as aggregation
of normalized inner similarities between features
Our Face Recognition Technology:
Face Recognition
September 23, 2014Face Retrieval for Security Applications
MPEG-7 feature
0.25kB float vector: <13.9, 9.5, -6.0, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
Cameron Diaz
MPEG-7 feature
0.25kB float vector: <15.1, 8.6, -5.4, …>
FaceSDK feature
16kB of binary data
VeriLook SDK feature
32kB of binary data
0.06
Aggregation
118.6
0.06
2543
Inner
similarities
0.12
0.11
0.06
Normalization
into [0, 1]
8/15
Jan Sedmidubsky
• Inner similarities are computed by integrated
recognition approaches themselves
• Each inner similarity is then normalized according
to specifically learned normalization function
– Normalization func. learned from training data (example)
• Aggregation is minimum of normalized similarities
Our Face Recognition Technology:
Face Recognition (cont.)
September 23, 2014Face Retrieval for Security Applications 9/15
Normalized value =
100% - <precision of inner similarity>
Example:
inner similarity = 118.6  precision = 88% 
normalized value = 100% - 88% = 12%  0.12
80
88
96
104
112
120
128
136
144
152
160
168
176
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Inner similarity of MPEG-7
Precision(%)
Jan Sedmidubsky
• Aggregation approach increases the quality of face
recognition in terms of recall and precision
• Face images of high/low quality have a distance
between eyes >100/<30 pixels
Our Face Recognition Technology:
Face Recognition (cont.)
September 23, 2014Face Retrieval for Security Applications
Low-quality face images
# of faces: 1000
High-quality face images
# of faces: 10,000
Recall
precision=85%
Recall
precision=95%
Recall
precision=85%
Recall
precision=95%
MPEG-7 24 14 8 3
FaceSDK 23 16 14 0
VeriLook SDK 12 11 53 51
Our aggregation 31 24 54 51
10/15
Jan Sedmidubsky
• Multi-face query:
– Query composed of a number of reference faces
– Relevance feedback on 1M dataset:
• Manual selection of positive (correct) retrieval results
• Iterative search where positive results represent query faces
• 1st iteration: precision=6%, 5th iteration: precision=30% (k=100)
Our Face Recognition Technology:
Face Recognition (cont.)
September 23, 2014Face Retrieval for Security Applications
query
query
1st iteration
2nd iteration
11/15
Jan Sedmidubsky
• Preprocessing (indexing) phase:
– All faces are indexed according to the MPEG-7 feature
• M-index (Novak, 2012) structure is utilized
• Indexing of 1M faces takes ~10minutes
• Efficient (scalable) retrieval of 100 faces:
– Candidate set of 10,000 the most similar faces is efficiently
retrieved by M-index (only MPEG-7 feature is utilized)
• Retrieval out of 1M faces takes ~0.15s (~10s without M-index)
– Retrieved 10,000 faces are sorted (re-ranked) according to
our aggregation approach
– Top-ranked 100 faces are returned
Our Face Recognition Technology:
Face Recognition (cont.)
September 23, 2014Face Retrieval for Security Applications 12/15
Jan Sedmidubsky
• Preprocessing (indexing) phase:
– All faces are indexed according to the MPEG-7 feature
• M-index (Novak, 2012) structure is utilized
• Indexing of 1M faces takes ~10minutes
• Efficient (scalable) retrieval of 100 faces:
– Candidate set of 10,000 the most similar faces is efficiently
retrieved by M-index (only MPEG-7 feature is utilized)
• Retrieval out of 1M faces takes ~0.15s (~10s without M-index)
– Retrieved 10,000 faces are sorted (re-ranked) according to
our aggregation approach
– Top-ranked 100 faces are returned
Our Face Recognition Technology:
Face Recognition (cont.)
September 23, 2014Face Retrieval for Security Applications
Increases
effectiveness
Increases
efficiency
12/15
Jan Sedmidubsky
Our Face Recognition Technology:
Summary
• Advantages against other face recognition systems:
– Support for multi-face query evaluation
– Aggregation of multiple existing approaches for more
accurate detection and recognition
• Significant improve in the recognition accuracy (naive approach:
6% accuracy, our approach: 40% accuracy)
– Tradeoff between the recognition accuracy and
performance can be controlled by a user
• Online search within millions of faces
September 23, 2014Face Retrieval for Security Applications 13/15
Jan Sedmidubsky
Our Face Recognition Technology:
Demonstration Application
• Demonstrated by an online web application:
– Implemented within Java + Apache Tomcat
– Commercial licenses of FaceSDK and VeriLook SDK
– 1 million database of extracted faces
– http://disa.fi.muni.cz/FaceMatch/
September 23, 2014Face Retrieval for Security Applications 14/15
Jan Sedmidubsky
Questions?
Thank you for your attention.
Try our online web application:
http://disa.fi.muni.cz/FaceMatch/
September 23, 2014Face Retrieval for Security Applications 15/15