10/26/2015
1
PA198
Augmented Reality Interfaces
Lecture 4
Augmented Reality Tracking
Fotis Liarokapis
liarokap@fi.muni.cz
26th October 2015
What is Tracking?
• Tracking works out where we are standing and
looking
– So that graphics can be draw in the right place
• Continually locating the users viewpoint
– Position (x, y, z)
– Orientation (yaw, pitch, roll)
Ideal Tracking
• Not easy
• Ideal scenario:
– Accuracy
– Precision
– Low‐latency
– Agile
– Robust
Tracking for AR
• More difficult than VR
• Tracking systems used for AR environments
must satisfy three basic requirements:
– The tracker must provide high accuracy when
calculating the pose
– The latency between the graphics system and the
tracker must be very low
– The tracker’s range of operation must be wide
enough to cover the needs of the application
Tracking Requirements
• Augmented Reality Information Display
– World Stabilized
– Body Stabilized
– Head Stabilized
Head Stabilized Body Stabilized World Stabilized
Increasing Tracking
Requirements
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Tracking Technologies
• Active
– Mechanical, Magnetic, Ultrasonic
– GPS, Wifi, cell location
• Passive
– Inertial sensors (compass, accelerometer, gyro)
– Computer Vision
– Marker based, natural feature tracking
• Hybrid Tracking
– Combined sensors (i.e. vision + inertial)
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
10/26/2015
2
AR Tracking Taxonomy
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Tracking Categories
• Three types of tracking:
– Sensors
– Visual
– Hybrid
Sensor Tracking
Tracking Principles
• The six main principles of tracking operation
include:
– Time of flight (TOF)
– Spatial scan
– Mechanical linkages
– Inertial sensing
– Phase difference sensing
– Direct-field sensing
Tracking Types
Billinghurst, M., Clark, A. Lee, G. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction, Vol. 8, No. 2-3 2014
Mechanical Trackers
• Mechanical tracking devices are widely used nowadays
due to the low cost of production
• The lack of certain hardware devices (i.e.
transmitter/receiver) makes mechanical trackers much
less sensitive to their immediate environment than
other types of trackers
– i.e. Electromagnetic trackers
• Two different types of mechanical devices are currently
used in the industry/research including
– The arm
– Force sensing ball
10/26/2015
3
Mechanical Trackers .
• Idea: mechanical arms with joint sensors
– Advantages: high accuracy, haptic feedback
– Disadvantages: cumbersome, expensive
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Mechanical Trackers ..
• The ‘arm’ or ‘boom’ sensing device takes
measurements in rotation using either a
potentiometer or optical encoders
• The device measures the forces exerted and it
is therefore applied in force-sensing joysticks
Mechanical Trackers ...
BOOM 3C BOOM HF
Electromagnetic Trackers
• Electromagnetic trackers are comprised of two
simple electronic systems:
– A transmitter
– A receiver
• Usually, their main function is to detect the
generated variations of the received signal
• In other words, the position and orientation of
the transmitters can be calculated
Tracking Error
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Electromagnetic Trackers .
Polhemus FASTRACK electromagnetic tracker
10/26/2015
4
Optical Trackers
• Optical trackers have the ability to operate over
large areas in indoor or outdoor environments
• However, the implementations of optical
tracking systems are diverse using
– Infra-red LEDs, photodiodes, lasers, video cameras,
web-cameras
– Combinations of these
Optical Trackers .
• The creation and maintenance of
a corresponding virtual line of
sight is essential for the
operation of any optical tracking
system
• They function by placing the light
sources or fiducials on the object
to be tracked and then
determine the position of the
object using light detectors
Cheap Optical Trackers Optical Tracker
• Idea
– Image Processing and
Computer Vision
• Specialized
– Infrared, Retro-Reflective,
Stereoscopic
• Monocular Based Vision
Tracking
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Outside-In v.s. Inside-Out Tracking
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Optical Tracking Technologies
• Scalable active trackers
– InterSense IS-900, 3rd Tech HiBall
• Passive optical computer vision
– Line of sight, may require landmarks
– Can be brittle
• Computer vision is
computationally-intensive 3rd Tech, Inc.
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
5
HiBall Tracking System (3rd Tech)
• Inside-Out Tracker
– $50K USD
• Scalable over large area
– Fast update (2000Hz)
– Latency Less than 1 ms
• Accurate
– Position 0.4mm RMS
– Orientation 0.02° RMS
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Motion Tracking
• Very popular for games
• Need markers
• Can track whole body
• Expensive solution
• Visit HCI Lab!!
VRECKO
• Modular framework for experiments in VR
Motion
Capture
Polygonal mesh
editing
Geometrical
sculpture
Physical
simulations
27
http://vrecko.cz/
VRECKO Video
28
http://vrecko.cz/research/vekva/freehand-painting/
Acoustic Trackers
• Acoustic tracking systems make use of ultrasonic
signals to avoid interference with the detectable
spectrum of human users
– Based on TOF measurement
• This method measures the time needed for the
sound to reach the receivers and then the
distance is calculated based on the speed of
sound in the air, producing absolute position and
orientation values
Acoustic Trackers .
• Since TOF can only measure distance, to achieve
3D tracking a combination of transmitter and
receiver is required
– For 3 DOF one transmitter and one receiver is
required
• For 6 DOF tracking 3 transmitters and 3 receivers
are necessary
Acoustic Magic
10/26/2015
6
Acoustic Trackers Pros and Cons
• Pros: Small, Cheap
• Cons: 3DOF, Line of Sight, Low resolution,
Affected Environment Condition (pressure,
temperature)
Ultrasonic
Logitech
IS600
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Inertial Trackers
• Inertial trackers measure changes in rotation regarding
one, two or even three axes by using a device called a
gyroscope
• Gyroscopes can maintain spinning on a particular axis
while in motion based on the laws of conservation of
angular momentum
– When an external force is applied the reaction is a motion
perpendicular to the axis of rotation
• Common applications for gyroscopes include
– Direction measurements for submarines, ships and pedestrian
navigation
Inertial Trackers .
• Their main advantage is that they do not use
receivers or transmitters avoiding
communication errors
• The main disadvantage is that they provide only
rotational information (3 DOF)
– Therefore it is more difficult for them to interface
when compared to other tracking systems (6 DOF)
Inertial Trackers ..
Inertia Cube from InterSense
Wii Remote
GPS Trackers
• GPS is a technology widely used for outdoor
tracking
• The most important categories include
– Standard GPS
– Differential GPS
– Real-time kinematic GPS
GPS Trackers .
• Standard GPS is a satellite based positioning
system that utilizes a total of 29 satellites
– This will change with Galileo!
• The position of the user is determined by
processing radio signals from the satellites
• In theory, GPS systems can estimate the user’s
position, by calculating the arrival time of at
least three satellite signals
10/26/2015
7
GPS Trackers .
• Satellites send position +
time
• GPS Receiver positioning
– 4 satellites need to be visible
– Differential time of arrival
– Triangulation
• Accuracy
– 5-30m+, blocked by weather,
buildings etc
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
GPS Trackers ..
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Problems with GPS
• Takes time to get satellite fix
– Satellites moving around
• Earths atmosphere affects signal
– Assumes consistent speed (the speed of light)
– Delay depends where you are on Earth
– Weather effects
• Signal reflection
– Multi-path reflection off buildings
– Signal blocking
– Trees, buildings, mountains
• Satellites send out bad data
– Misreport their own position
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Differential GPS
• Uses emitting
ground stations
that refine the
resolution
• Accurate to <
5cm close to
base station
– 22m/100 km
• Expensive
– $20-40,000 USD
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Differential GPS .
• The mobile GPS receiver
monitors signals from a
fixed radio transmitter and
another GPS receiver
• To refine the resolution
the transmitter sends the
corrected co-ordinates
– Based on the difference
between the known and
the computed positions
Assisted-GPS (A-GPS)
• Use external location server to send GPS signal
– GPS receivers on cell towers, etc
– Sends precise satellite position (Ephemeris)
• Speeds up GPS Tracking
– Makes it faster to search for satellites
– Provides navigation data (don’t decode on phone)
• Other benefits
– Provides support for indoor positioning
– Can use cheaper GPS hardware
– Uses less battery power on device
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
8
Assisted-GPS (A-GPS)
http://gps-response.com/61-a-gps.html
Cell Tower Triangulation
• Calculate phone position
from signal strength
– < 50 m in cities
– > 1 km in rural
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
WiFi Positioning
• Estimate location based
on WiFi access points
– Use known locations of
WiFi access points
• Triangulate through
signal strength
– i.e. PlaceEngine
• Accuracy
– 5 to 100m
• Depending on WiFi density
PlaceEngine
• Enables a device equipped with Wi-Fi such as a
laptop PC or smart phone to determine its
current location
– Can be used in conjunction with web sites that
provide local area information to gain easy access to
nearby services
– Client software for PC and mobiles
– Free of charge
http://www.placeengine.com/en
Indoor WiFi Location Sensing
• Indoor Location
– Asset is people tracking
• Aeroscout
– http://aeroscout.com/
– WiFi + RFID
• Ekahau
– http://www.ekahau.com/
– WiFi + LED tracking
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Integrated Systems
• Combine different
systems
– GPS, Cell tower, WiFi
signals
• Database of known
locations
– 700 million Wi-Fi access
points and cellular towers
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
9
Skyhook
• Skyhook Context Accelerator’s features and
analytics allows you to create place-specific
experiences
• Combines Wi-Fi with GPS, Cell Towers, IP
address and device sensors to give you the
fastest, most accurate positioning for any
device on any OS
http://www.skyhookwireless.com/
Skyhook Video
http://www.skyhookwireless.com/
Comparative Accuracies
• Study testing iPhone 3GS cf. low cost GPS
• A-GPS
– 8 m error
• WiFi
– 74 m error
• Cell Tower Positioning
– 600 m error
Accuracy of iPhone Locations: A Comparison of Assisted GPS, WiFi, and Cellular Positioning, Transactions in GIS, Volume 13 Issue 1, 5 - 25
Visual Tracking
Visual Tracking
• Tracking in AR is usually performed using
computer vision algorithms
– Still experimental
– Less expensive in terms of cost
– More computing power is required
– Works reasonably good for indoor environments
– Problems with outdoor environments
Visual Tracking .
• Establishes correspondences between the
video feed and 3D positions in space
(u, v) ↔ (x, y, z)
• 6‐DOF Position can be calculated from these
correspondences
10/26/2015
10
Visual Tracking ..
• Lots of tools:
– Three‐point‐pose
– RANSAC
– N‐point‐pose
– Iterative nonlinear optimisation
– Robust M‐estimation
– etc
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Approaches to Visual Tracking
• Use a marker
– Corners of square give easy correspondences
• Use a known textured object
– Coordinates of texture features are known
• Learn an unknown environment on‐line
– Coordinates of scene are computed on‐the‐fly
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Marker‐based Tracking
• Distinctive shapes which can be found using
elementary image processing operations
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Marker‐based Tracking .
• Has been done for more than
10 years
• A square marker provides 4
corners
– Enough for pose estimation!
• Several open source solutions
exist
• Fairly simple to implement
– Standard computer vision
methods
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Marker‐based Tracking ..
• Best suited for tangible manipulation of virtual
elements and untrained users
• Unsuitable for uncontrolled environments
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Tracking Challenges in ARToolKit
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
11
Limitations of ARToolKit
• Partial occlusions cause tracking failure
• Affected by lighting and shadows
• Tracking range depends on marker size
• Performance depends on number of markers
– i.e. artTag, ARToolKitPlus
• Pose accuracy depends on distance to marker
• Pose accuracy depends on angle to marker
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Different Approaches
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Other Marker Tracking Libraries
• arTag
– http://www.artag.net/
• ARToolKitPlus [Discontinued]
– http://studierstube.icg.tu-
graz.ac.at/handheld_ar/artoolkitplus.php
• stbTracker
– http://studierstube.icg.tu-
graz.ac.at/handheld_ar/stbtracker.php
• MXRToolKit
– http://sourceforge.net/projects/mxrtoolkit/
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Known‐Template Tracking
• Exploits advances in image processing
• Rapid feature extraction and invariant
descriptor matching
• Distinctive points of a textured object are
matched to the image
• Must be known in advance!
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Natural Feature Tracking
• Tracking from features of the surrounding
environment
– Corners, edges, blobs, ...
• Generally more difficult than marker tracking
– Markers are designed for their purpose
– The natural environment is not…
• Less well-established methods
• Usually much slower than marker tracking
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Natural Feature Tracking .
• Use Natural Cues of Real Elements
– Curves
– Edges
– Lines
– Surface Texture
– Interest Points
• Model or Model-Free
• No visual pollution
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
12
Curve Based Tracking
• Track curved features like the arches of the
bridge
– 1998
www.loria.fr/~petitjea/papers/mva99.pdf
Edge Based Tracking
• RAPiD [Drummond et al. 02]
– Initialization, Control Points, Pose Prediction
(Global Method)
https://vimeo.com/91642721
Line Based Tracking
• Visual Servoing [Comport et al. 2004]
https://www.youtube.com/watch?v=_Dln257k2Sc
Region-based Approach
• On initialization the user
selects a plane of interest
• The rectifying Homography
and rectified template
image are retained
Template
Input Image at time t
H
Birkbeck, N. Registration for Augmented Reality, 2006
Region-based Approach .
• When new image
arrives, use image
intensities to
refine the
Homography
Birkbeck, N. Registration for Augmented Reality, 2006
Template
Image at time t Image at time t+1
HH
Dense Reconstruction
• Allows occlusion and interaction between
physical and real world
– Newcombe & Davison 2010
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
10/26/2015
13
Marker vs. Natural Feature Tracking
• Marker tracking
– + Can require no image database to be stored
– + Markers can be an eye-catcher
– + Tracking is less demanding
– - The environment must be instrumented with markers
– - Markers usually work only when fully in view
• Natural feature tracking
– - A database of keypoints must be stored/downloaded
– + Natural feature targets might catch the attention less
– + Natural feature targets are potentially everywhere
– + Natural feature targets work also if partially in view
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Some Algorithms for
Visual Tracking
Random Forest
• Multi-classifier based on Randomized Trees
• Firstly introduced in 1997 handwritten
recognition (Amit, Y.,German, D.)
• Developed by Leo Breiman (Medical Data
Analisys)
• Applied to tracking by detection (LePetit06)
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest - Main Features
• Fast training step and execution
• Good precision
• Random selection of the independent variables
– Known as features
• Random selection of examples
• Easy to implement and parallelizable
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest - Classifier Training
• N Binary-Trees are
grown
• Pixel intensity tests
are executed in
any non-terminal
node
• Pixels can be
selected at
random
• Posterior
distributions are
stored in leave
nodes
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest - Example
Classification
• Every example is
dropped down
the trees
• The example
traverse the tree
towards the leaf
nodes
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Pixels to be tested
10/26/2015
14
Random Forest - Combine Results
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
T1 T2 Tn
Random Forest
  

T
k
n
kitlabelclass tcYPExample
1_ |maxarg
FERNS
• Introduced in 2007 (Mustafa Özuysal)
• Multi-classifier
• Applied to 3D keypoint recognition
• Successfully applied to image
recognition/retrieval (Zisserman07)
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Original FERNS
• A classic Naïve Bayesian framework provides a
simple and fast method
• FERNS are non-hierarchical structures used to
classify the patches
– Each one consists of a small set of binary tests
– Returns the probability that a patch belongs to any one
of the classes that have been learned during training
• Combined in a Naive Bayesian way
• Using randomized trees the classifier is trained
– By synthesizing many views of the keypoints extracted
from a training image as they would appear under
different perspective or scale
Ozuysal, M., Fua, P., Lepetit, V. Fast Keypoint Recognition in Ten Lines of Code, Proc. CVPR, IEEE Computer Society, 2007. (DOI: 10.1109/CVPR.2007.383123)
FERNS - Main Features
• Non hierarchical structure
• Semi Naive-Bayes Combination Strategy
• Random selection of the independent variables
– Known as features
• Random selection of examples
• Easy to implement and parallelizable
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
FERNS Classifier Training
x x x
3
2 Possible Outputs
0 7
.
.
.
 ik cCFP |
Posterior Distributions
(Look-up Tables)
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
FERNS Classifier Training .
1 1 0
0 0 0
0 1 06
0 0 1 1
2
3
Class 1
Class 2
Class 1
Class 2
.
.
.
.
.
.
.
.
.
.
Fern 1
Fern 2
Fern n
Posterior Distributions
(Look-up Tables)
.
.
.
.
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
10/26/2015
15
Example Classification
   
M
ikflabelclass cCFPExample
1
_ |maxarg
2 6 1
Posterior Distributions
(Look-up Tables)
Fern 1 Fern 2 Fern 3
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest vs FERNS
• Rotation Range
– 20 Trees, 15 depth
– 225 different classes
– 400 images per class
80
85
90
95
100
0 PI/2 PI 3PI/2 2PI
%ClassificationRate
Rotation Range
FERNS Random Forest
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest vs FERNS .
• Scale Range
– 20 trees, 15 depth
– 225 different classes
– 400 images per class
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
0
10
20
30
40
50
60
70
80
90
100
0,8-1,0 0,5-1.0 1,0-1,2 1,0-1,5 0,8-1,2 0,5-1,5
%ClassificationRate
Scale Range
FERNS Random Forest
Random Forest vs FERNS ..
• Size of the training set
– 20 trees, 15 depth
– 225 different classes
– [0.5-1.5] scale range
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
45
50
55
60
65
70
75
80
350 550 750 1000 1300 1500 1800 2100 2800 3800 4550
%ClassificationRate
Training Set Size
FERNS Random Forest
Random Forest vs FERNS …
• Number of different classes
– 20 trees, 15 depth
– [0.8-1.2] scale range
– 1500 Training images per class
80
85
90
95
100
200 325 425 525 625 725
%ClassificationRate
Number of Classes
FERNS Random Forest
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest vs FERNS ….
• Training time
– 20 trees, 15 depth
– 225 different classes
– [0.5-1.5] scale range
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
0
50
100
150
200
250
300
350 550 750 1000 1500 1800 2500 2800 3800
TrainingoTime(s)
10/26/2015
16
FERNS Video
https://www.youtube.com/watch?v=OxibDXlWIoA
Random Forest - FERNS
• Tracking of Planar Surfaces
• The Classifiers are applied for interest point
(feature) matching
• Matched Points are used during camera pose
estimation Process
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest - FERNS .
• Building the training set
– Frontal view of the object
to be detected
– Feature Point extraction
FAST (Rosten06) and YAPE
(CvLab)
– Sub-images (patches) are
generated for each class
Classes to Be recognized by the Classifier
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Random Forest - FERNS ..
• Building the training set
– Generate random affine transformations
– Generate new examples of each class
…..
Random Affine
transformation
s
Training Set (examples)
Barandiaran, I., Cottez, C., et al. Comparative Evaluation of Random Forest and Fern classifiers for Real-Time Feature Matching, VICOM Tech, 2008
Unknown Environments
• SLAM: Simultaneous
Localization and Mapping
– Introduced to computer vision
by Davison et al 2003
• Structure of world computed
from image
• (x,y) observed directly – z
computed from parallax
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Parallel Tracking and Mapping (PTAM)
• Splits tracking and mapping into separate tasks
– Map built in background from keyframes
• Draws from photogrammetry
– Bundle adjustment and epipolar geometry
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
10/26/2015
17
PTAM Video
Hybrid Tracking
Hybrid AR Systems
• Currently, there is no perfect tracking technology and
the existing ones have some advantages as well as
limitations
• To overcome the limitations of each tracking technology
hybrid systems can be employed for combining
hardware devices to achieve better results that
otherwise could not be handled
• Hybrid systems can be successfully combine vision
techniques with haptic devices to improve the overall
tracking efficiency as well as increasing the capabilities
of the system
Sensor Tracking
• Used by many “AR browsers”
• GPS, Compass, Accelerometer, Gyroscope
• Not sufficient alone (drift, interference)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Outdoor Hybrid Tracking
• Combines
– computer vision
• Natural feature tracking
– Inertial gyroscope sensors
• Both correct for each other
– Inertial gyro - provides frame to frame prediction
of camera orientation
– Computer vision - correct for gyro drift
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Combining Sensors and Vision
• Sensors
– Produce noisy output (= jittering augmentations)
– Are not sufficiently accurate (= wrongly placed
augmentations)
– Gives us first information on where we are in the world,
and what we are looking at
• Vision
– Is more accurate (= stable and correct augmentations)
– Requires choosing the correct keypoint database to track
from
– Requires registering our local coordinate frame (online
generated model) to the global one (world)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
18
Hand Tracking / Interaction
• Real‐time unassisted monocular hand tracking
is still unsolved
http://www.raeng.org.uk/publications/other/georg-klein-presentation-frontiers-of-engineering
Outdoor AR Tracking System
• You, Neumann, Azuma outdoor AR system
(1999)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Robust Outdoor Tracking
• Hybrid Tracking
– Computer Vision, GPS, inertial
• Outdoors
– Reitmayer & Drummond (Univ. Cambridge)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Handheld Display
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Hybrid Tracking Indoors Video
https://www.youtube.com/watch?v=4H8JtLr0Mk8
Hybrid Tracking Outdoors Video
https://www.youtube.com/watch?v=GQ-Delh4DQA
10/26/2015
19
AR Registration
What is Registration?
• Registration is the accurate positioning of
virtual information into the real environment
• Mis-registration:
– Breaks the illusion that the two coexist
– Prevents acceptance of many serious applications
Chroma Key vs Z-Key
Blue
Background
Chroma Key Method
Virtual Object
Switch by
Chroma key
Video Camera
Virtual Object
Pixel-by-pixel
Merging device
Z-Key Method
Switch by
Z-key
Spatial Registration
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Sources of Registration Errors
• Static errors
– Optical distortions
– Mechanical misalignments
– Tracker errors
– Incorrect viewing parameters
• Dynamic errors
– System delays (largest source of error)
• 1 ms delay = 1/3 mm registration error
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Reducing Static Errors
• Distortion compensation
• Manual adjustments
• View-based or direct measurements
• Camera calibration
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
10/26/2015
20
View Based Calibration (Azuma 94)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Dynamic Errors
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Reducing Dynamic Errors
• Reduce system lag
– Faster components/system modules
• Reduce apparent lag
– Image deflection
– Image warping
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Reducing System Lag
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Reducing Apparent Lag
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
More on Reducing Dynamic Errors
• Match input streams
– Delay video of real world
to match system lag
• Predictive Tracking
– Inertial sensors helpful
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Azuma / Bishop 1994
10/26/2015
21
Predictive Tracking (Azuma 94)
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Conclusions
• Environment sensing
– Effective AR needs:
• User’s position & orientation
• Position of objects
• Depth map
• Video tracking
– Place markers in environment
– Hybrid systems (video + magnetic sensors, video +
accelerometer) have better performance
http://www-inst.eecs.berkeley.edu/~ee225b/sp11/lectures/Recent%20Advances%20in%20Augmented%20Reality%20PPT.pdf
Conclusions .
• Outdoor & mobile AR apps
– Not practical to cover environment
with markers
– Compass/gyroscopetracker
(orientation)
– GPS (position)
• Unprepared environments
– Rely on tracking visible natural
features
– Track horizon silhouette (given
database of environment)
http://www-inst.eecs.berkeley.edu/~ee225b/sp11/lectures/Recent%20Advances%20in%20Augmented%20Reality%20PPT.pdf
Questions