24/02/2020
1
PA201
Virtual Environments
Lecture 2
Virtual Reality Tracking
Fotis Liarokapis
liarokap@fi.muni.cz
24th February 2020
Motivation
• Different ways of interaction in VR
– More natural
– Higher level of immersion
– Task performance
– Control navigation
– Control interaction
• How do we track the objects/users?
• How do we determine what the user sees?
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Requirements
• Signaling
– Button presses
– Other user input
• Pose calculation
– Position (x, y, z)
– Orientation (yaw, pitch, roll)
http://www.tmt.unze.ba/zbornik/TMT2010/091-TMT10-029.pdf
DOF
• Degrees of freedom (DOF) are the set of independent
displacements and/or rotations that specify completely
the position and orientation of the body or system
• 1 DOF system
– Volume control, speed control, sliding along a straight line
etc
• 2 DOF system
– Mouse, joystick, drawing tablet etc
• 3 DOF system
– Position or orientation
• 6 DOF system
– Position and orientation
More DOFs
• More DOFs (more moving
parts)
– Human hand has 22
possible movements, so
tracking it completely would
be 22 DOF
• Position, orientation, joints
– Humans total over 100 DOF
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Top view of simplified human hand
kinematic model (22 DOFs)
What to Track
• Head pose
• Hand pose
• Other body part(s)
• Other objects
• Trackers can do:
– Only position
– Only orientation
– Both position and
orientation
24/02/2020
2
What to Track .
• If tracking of other body parts beside the head
is required, their respective positions and
orientations would be measured using special
purpose tracking probes attached to the parts
• The body parts' locations could then be
represented with respect to the head
coordinate system
https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf
Applications of Tracking
• Used in VR systems
– Used in CAVEs
– Not needed for VR training environments (e.g. flight
simulators)
• Essential for AR systems
• Beneficial for geo-location
– Positioning (e.g. “you are here”)
– Tracking (e.g. “where is my smartphone?”)
– Navigation (e.g. how to get from point A to point B)
– Geo-fencing (e.g. turn off alarms in cinema)
http://www.cs.tut.fi/courses/SGN-5406/lectures/VR7-sensors.pdf
Tracking Properties
Evaluation Criteria
• Data returned
• Spatial distortion (accuracy)
• Resolution
• Jitter (precision)
• Drift
• Lag
• Update Rate
• Range
• Interference and noise
• Mass, Inertia and Encumbrance
• Number of Tracked Points Durability
• Wireless
• Price
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Tracker Accuracy
• Difference between an object’s
pose and the reported pose
– Separate for position and
rotation
– Not the same thing as resolution
• Resolution is the minimum change
that the sensor can detect
• Typically degraded with
distance from the origin of the
reference system of
coordinates
– Operating range
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Tracker Jitter
• The change in tracker output
when the tracked object is
stationary
• A tracker with no jitter gives
constant value as output if
the object is stationary
• Should be minimized
• Unwanted effects in graphics
– Tremor
– Jumpy virtual objects
– Can be filtered -> latency
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
24/02/2020
3
Latency
• What makes up latency (or Lag)?
– Acquisition
– Transmission
– Filtering
• Latency = t1 - t0
– where:
• t0 time when sensor is at point p
• t1 time when sensor reports p
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Tracker Drift and Latency
• Drift: the steady increase in tracker error with
time
– As time passes, the tracker inaccuracy grows
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Update Rate
• Number of tracker position/orientation
samples per second
– High update rate != accuracy
– Poor use of update information may result in
more inaccuracy
– Communication pathways and data packet size are
important
– Typically 30-240 datasets/s
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Multiplexing Effect
• If a tracker measures several objects, sampling
rate can suffer
– Multiplexing effect
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Tracker update rate versus number of tracked objects
Range
• Position and orientation range could be
different
• Sensitivity not uniform across all axis
• Working volume
– What is the shape?
– Accuracy decreases with distance
– Range is inversely related to accuracy
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Interference and Noise
• Interference - external phenomenon that
degrades system’s performance
• Each type of tracker has different causes of
interference/noise
– Occlusion
– Metal
– Noise
– Environmental
• i.e door slamming, air conditioner, etc
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
24/02/2020
4
Principles
Tracking Principles
• The six main principles of tracking operation
include:
– Time of flight (TOF)
– Spatial scan
– Inertial sensing
– Mechanical linkages
– Phase difference sensing
– Direct-field sensing
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Time of flight (TOF)
• TOF systems rely on the
measure of distance between
features attached on one side
to a reference and on the other
side to a moving target
• These distances are determined
by measuring the time of
propagation of pulsed signals
between pairs of points
– Under the assumption that the
speed of propagation of the
signals is constant
Principle of a TOF tracking system
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Spatial Scan
• Spatial scan trackers compute the position and
the orientation of a target
– The optical sensors are typically cameras
• The principle is based on either:
– The analysis of 2D projections of image features
– On the determination of sweep-beam angles
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Spatial Scan Configurations
• Outside-in systems employ video cameras that
are placed on the reference and record features
of the target
– This technique is widely employed
• In an inside-out configuration the sensor is on
the target and the emitters are on the reference
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Outside-In vs. Inside-Out Tracking
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
24/02/2020
5
Outside-In v.s. Inside-Out Tracking .
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Inertial Sensing
• The principle of inertial sensing is based on
the attempt to conserve either
– A given axis of rotation
• Mechanical gyroscopes
– A position
• Accelerometers
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Intersense
• InertiaCube 3
– Orientation, inertial sensors
– 2000 €, wireless 3000 €
• InterTrax2
– Discontinued
• IS-900
– 6 DOF, inertial sensors + ultrasound
– Very accurate and fast
– 9 - 43,000 €
• IS-1200
– For mobile augmented reality
– InertiaCam looks for fiducials
– Supports over 30,000 fiducials
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
InertiaCube4
• Sourceless 3DOF tracking with full
360° range
• Accuracy of 1° yaw, 0.25° pitch &
roll
• 200 Hz update rate
• 2000° maximum angular rate
• Adjustable output filters and
rotational sensitivity
• Prize: $995
http://www.intersense.com/pages/18/234
Mechanical Gyroscope
• Mechanical gyroscopes
are based on the
principle of
conservation of angular
momentum
– The wheel is mounted
on a frame so that the
external moments are
minimized
– This allows the target
to turn around the
wheel without
experiencing a change
in the direction of its
axis
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Structure of a mechanical gyroscope
Accelerometer
• An accelerometer measures the
linear acceleration of an object to
which it is attached
• A single-degree-of-freedom
device which has
– Some kind of mass
– A spring like supporting system
– A frame structure with damping
properties
• It may rely for example on a mass
mounted on a piezo-electric
crystal and attached to the target
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Structure of an accelerometer
24/02/2020
6
Mechanical Linkages
• Two types have been used
• One is an assembly of
mechanical parts that can each
rotate providing the user with
multiple rotation capabilities
– The orientation of the linkages is
computed from the various
linkages angles measured with
incremental encoders or
potentiometers
– This type of tracking system uses
mechanical linkages between the
reference and the target
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Structure of a typical mechanically
linked tracking system
Mechanical Linkages .
• Other types of mechanical
linkages are wires that are rolled
on coils
– A spring system ensures that the
wires are tensed in order to
measure the distance accurately
• The degrees of freedom sensed
by mechanical linkage trackers
are dependent upon the
constitution of the tracker
mechanical structure
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Phase Difference
• Measures the relative phase of an incoming
signal from a target and a comparison signal of
the same frequency located on the reference
– As in the TOF approach, the system is equipped with
three emitters on the target and three receivers on
the reference
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Direct-Field
• Two different types:
– Magnetic field
• i.e. Magnetometers
– Gravitational field
• i.e. inclinometer
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Magnetometers
• Magnetometers measure the orientation of an
object with respect to the magnetic field of
the earth
– Magnetic field sensors include fluxgate, Hall
effect, magneto-resistive, and magneto-inductive
sensors
– The most relevant magnetometers use magnetoinductive
sensors
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Inclinometers
• An inclinometer operates on
the principle of a pendulum
• Common implementations use
electrolytic or capacitive
sensing of fluids
– A simple implementation may
measure the relative level of
fluids in two branches of a tube
to compute inclination
– More advanced
implementations include an
optical inclinometer
Principle of operation and structure
of an optical inclinometer
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
24/02/2020
7
Hybrid Approaches
• Hybrid technology refers to the combination of
various technologies and systems based on different
principles of operation
– i.e. TOF measurements versus phase difference
measurements
• Pros:
– Both relative and absolute measurements
– Make exhaustive measurements
• Cons:
– Complex
– Cost
Rolland, J.P., Baillot, Y., Goon, A.A. A survey of tracking technology for virtual environments, Fundamentals of wearable computers and augmented reality,
2001. (https://pdfs.semanticscholar.org/ce53/48128f94f3383bdc4eb15fb4eaf3721d521f.pdf)
Video Tracking
Vision based tracking
• Video is processed (computer vision)
– Very heavy calculation
– Markers, chroma-key helps a lot
– OpenCV, https://opencv.org/
• Image filtering, contour finding
• Feature extraction, object recognition
• Recognition of forms
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Feature-based Object Detection
• In feature-based object detection, standardization of
image features and registration (alignment) of
reference points are important
• Shape-based approaches
– Hard due to:
• The difficulty of segmenting objects of interest in the images
• Many objects with occlusions and shading
– Images may need to be filtered depending on the
application
• Different object types such as persons, flowers, and airplanes
may require different algorithms
• For more complex scenes, noise removal and transformations
invariant to scale and rotation may be needed
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Feature-based Object Detection .
• Color-based approaches
– Not always useful but easy to be acquired
– Low cost
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Template-based object detection
• Object detection matches features between the
template and the image sequence under analysis
– If a template describing a specific object is available
– Object detection with an exact match is computationally
expensive
– Quality of matching depends on the details and the
degree of precision provided by the object template
• Two types exist:
– Fixed template matching
– Deformable template matching
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
24/02/2020
8
Fixed template matching
• Useful when object shapes do not change with
respect to the viewing angle of the camera
• Two major techniques
– Image subtraction
• The template position is determined from minimizing the
distance function between the template and various
positions in the image
– Correlation
• Matching by correlation utilizes the position of the
normalized cross-correlation peak between a template and
an image to locate the best match
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Deformable Template Matching
• Deformable template matching approaches are
more suitable for cases where objects vary due
to rigid and non-rigid deformations
– These variations can be caused by either the
deformation of the object per se or just by different
object pose relative to the camera
– Because of the deformable nature of objects in most
video, deformable models are more appealing in
tracking tasks
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Motion Detection
• Motion detection complicates the object
detection problem by adding object’s
temporal change requirements,
– However it also provides another information
source for detection and tracking
• A large variety of motion detection algorithms
have been proposed
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Object Tracking using Motion
• Motion detection provides useful information for
object tracking
– Tracking requires extra segmentation of the
corresponding motion parameters
• Two categories:
– Motion-based
– Model-based
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Object Tracking using Motion .
• Motion-based
– Rely on robust methods for grouping visual motion
consistencies over time
– Fast but have considerable difficulties in dealing with
non-rigid movements and objects
• Model-based
– Explore the usage of high-level semantics and knowledge
of the objects
– More reliable compared to the motion-based
– Suffer from high computational costs for complex models
• Due to the need for coping with scaling, translation, rotation,
and deformation of the objects
http://medianet.kent.edu/surveys/IAD01F-objdetection/index.html
Face Recognition
• Easy way is to compare a selection of facial
features from the image and a face database
• Typically used in security systems and can be
compared to other biometrics such as
fingerprint or eye iris recognition systems
– Recently, popular as a commercial identification
and marketing tool
• Face Recognition
– http://www.face-rec.org/
24/02/2020
9
Camera Tracking
• A single camera can do 2D
tracking
– Typically used in second
person VR
• Participants watch themselves
in the virtual world
• The video source is used both
for tracking the user and adding
the user’s image into the virtual
world
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Camera Tracking .
• Usually multiple cameras needed for 3D tracking
– Several dots are needed to acquire the orientation
– The positions of the light dots in the image give the
directions of the objects as seen from the camera
– The operation range can be very large
– Requires calculation, can be expensive and slow
– Time-of-flight cameras send IR
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Videometric Tracking
• Inside-looking-out
• Video camera on the tracked object
• The camera watches surroundings
• VR system analyzes images
– Try to locate landmarks
– Derive camera’s relative position to the landmarks
• Often used in augmented reality systems
– Can operate in a large area, also outdoors
• i.e. a camera on the HMD enables to determine the corners of the room
• Computational resources needed to do the image analysis
becomes a consideration
• A single camera can determine its own 6-DOF position
– Multiple reference points needed
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Videometric Tracking .
• The location of the landmarks must me known
or be discernible from other data
– Otherwise calculations get very complex and
probability for errors grow
– Landmark color and shape different from
surroundings
• Reference points
– Using LED lights at certain spots
– Using special easily distinguishable patterns
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Videometric Tracking ..
• Usage of LED lights
• Infrared sensitive cameras
• Activating the lights as patterns
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Tracker Types
24/02/2020
10
Mechanical Trackers
• Mechanical tracking devices are widely used
nowadays
• The lack of certain hardware devices (i.e.
transmitter/receiver) makes mechanical trackers
much less sensitive to their immediate environment
than other types of trackers
– i.e. Electromagnetic trackers
• Two different types of mechanical devices are
currently used in the industry/research including
– The arm
– Force sensing ball
Mechanical Trackers .
• Idea: mechanical arms with joint sensors
– Advantages: high accuracy, haptic feedback
– Disadvantages: cumbersome, expensive
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Mechanical Trackers ..
• The ‘arm’ or ‘boom’ sensing
device takes measurements
in rotation using either a
potentiometer or optical
encoders
• The device measures the
forces exerted and it is
therefore applied in forcesensing
joysticks
BOOM 3C
Mechanical Trackers - BOOM
• A BOOM display is an HMD
(head mounted display)
mounted on the end of a
mechanical arm
– The system detects the position
and orientation by this arm
– Advantage is very high update
rate
– Disadvantage is limited range of
user´s motion BOOM HF
Mechanical Trackers - BOOM Tabletop
• A tabletop model
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Mechanical Trackers - Motion Capture
Suit
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
24/02/2020
11
Electromagnetic Trackers (EM)
• Electromagnetic trackers are
comprised of two simple
electronic systems:
– A transmitter
– A receiver
• Usually, their main function
is to detect the generated
variations of the received
signal
• In other words, the position
and orientation of the
transmitters can be
calculated
Polhemus FASTRACK electromagnetic tracker
EM Characteristics
• Data returned: 6 DOF
• Spatial distortion – 0.6 mm, 0.025°
• Resolution – 0.00508 mm, 0.025° / inch from receiver
• Jitter (precision) – mm to cm
• Drift - none
• Lag – reported 4 ms
• Update Rate - 120 Hz
• Range - 5 ft
• Number of Tracked Points – 16 (divides update rate)
• Wireless - yes
• Interference and noise – metal, earth
• Mass, Inertia and Encumbrance - minimal
• Durability - high
• Price - $4000+
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
EM Advantages
• Measure position and orientation in 3D space
• Does not require direct line of sight
• Low encumbrance
• Cost
• Good performance close to emitter
• Lag
• Can be built ‘into’ devices
• Earth magnetic field good for 3DOF
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
EM Disadvantages
• Accuracy affected by
– DC: Ferrous metal and electromagnetic fields.
– AC: Metal and electromagnetic fields
• Operate on only one side of the source
– i.e. the working hemisphere
• Low range (effectively 5’ – does not scale well)
• Calibration
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
EM Applications
• Ascension Flock of Birds
• Polhemus Fastrak
• Extremely popular
• Good for many applications
– CAVEs (remove metal)
– HMDs
– Projection displays
– Fishtank
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Tracking Error
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
24/02/2020
12
Optical Trackers
• Optical trackers have the ability to operate over
large areas in indoor or outdoor environments
• However, the implementations of optical
tracking systems are diverse using
– Infra-red LEDs, photodiodes, lasers, video cameras,
web-cameras
– Combinations of these
Optical Trackers .
• The creation and maintenance of
a corresponding virtual line of
sight is essential for the
operation of any optical tracking
system
• They function by placing the light
sources or fiducials on the object
to be tracked and then
determine the position of the
object using light detectors
Cheap Optical Trackers Optical Tracker
• Idea
– Image Processing and
Computer Vision
• Specialized
– Infrared, Retro-Reflective,
Stereoscopic
• Monocular Based Vision
Tracking
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Optical Tracking Technologies
• Scalable active trackers
– InterSense IS-900, 3rd Tech HiBall
• Passive optical computer vision
– Line of sight, may require landmarks
– Can be brittle
• Computer vision is
computationally-intensive 3rd Tech, Inc.
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
HiBall Tracking System (3rd Tech)
• Inside-Out Tracker
– $50K USD
• Scalable over large area
– Fast update (2000Hz)
– Latency Less than 1 ms
• Accurate
– Position 0.4mm RMS
– Orientation 0.02° RMS
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
24/02/2020
13
Motion Tracking
• Very popular for VR
• Need markers
• Can track whole body
• Expensive solution
• A feasible tool for creating animation
• Visit HCI Lab!!
Motion Tracking .
• Can be based on any tracking
method
– Often cameras are used
– Reflectors or LEDs used
– Facial 1-2 cameras
– Full body 4-6 cameras
• Often not real time (Movies)
• The movements are recorded
and animation then rendered
WorldViz PPT
• Freedom of movement in VR
– Track multiple users over large
areas while maintaining
precision and accuracy
– Quickly configure and calibrate
with user-friendly, streamlined
user interface
– Real-time sub-millimeter
accuracy
– Integrate with various other
real-time rendering software
https://www.worldviz.com/virtual-reality-motion-tracking/
OptiTrack
• High-speed, low latency
• Head, controller & body
tracking
• Track any object type using
a single infrared LED marker
configuration
– Allows for identically
manufactured HMDs,
weapons, controllers, etc to
be tracked simultaneously
http://optitrack.com/motion-capture-virtual-reality/
VRECKO
• Modular framework for experiments in VR
Motion
Capture
Polygonal mesh
editing
Geometrical
sculpture
Physical
simulations
77
http://vrecko.cz/
VRECKO Video
78
http://vrecko.cz/research/vekva/freehand-painting/
24/02/2020
14
Valve’s Lighthouse Tracking
• The main idea behind tracking is flooding a room
with non-visible light, Lighthouse functions as a
reference point for any positional tracking device
(like a VR headset or a game controller) to figure
out where it is in real 3D space
http://gizmodo.com/this-is-how-valve-s-amazing-lighthouse-tracking-technol-1705356768
Valve’s Lighthouse Tracking .
• Valve’s Lighthouse boxes don’t have any cameras
• They just fire light out (sixty times every second)
into the world to help ships (or VR headsets)
navigate on their own
• That light comes from a whole bunch of
stationary LEDs, plus a pair of active laser
emitters that spin like crazy
http://gizmodo.com/this-is-how-valve-s-amazing-lighthouse-tracking-technol-1705356768
Valve’s Lighthouse Tracking ..
• The receiver (VR headset or controller) is covered
with little photosensors that detect the flashes and
the laser beams
• When a flash occurs, the headset simply starts
counting (like a stopwatch) until it “sees” which one
of its photosensors gets hit by a laser beam
• It uses the relationship between where that
photosensor exists on the headset, and when the
beam hit the photosensor, to mathematically
calculate its exact position relative to the base
stations in the room
http://gizmodo.com/this-is-how-valve-s-amazing-lighthouse-tracking-technol-1705356768
Acoustic Trackers
• Acoustic tracking systems make use of ultrasonic
signals to avoid interference with the detectable
spectrum of human users
– Based on TOF measurement
• Measures the time needed for the sound to reach the
receivers and then the distance is calculated based on the
speed of sound in the air, producing absolute position and
orientation values
Acoustic Trackers .
• Since TOF can only measure distance, to achieve
3D tracking a combination of transmitter and
receiver is required
– For 3 DOF one transmitter and one receiver is
required
• For 6 DOF tracking 3 transmitters and 3 receivers
are necessary
Acoustic Magic
Acoustic Trackers Pros and Cons
• Pros: Small, Cheap
• Cons: 3DOF, Line of Sight, Low resolution,
Affected Environment Condition (pressure,
temperature)
Ultrasonic
Logitech
IS600
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
24/02/2020
15
Ultrasound Head Tracker
• Transmitter, receiver and
electronics unit
• Transmitter is a set of three
ultrasonic speakers 30cm from
each other
– Rigid and fixed triangular frame
• Receiver is a set of three
microphones
– Placed at the top of the HMD
– May be part of 3D mice, stereo
glasses, or other interface
devices
• Range typically about 1.5 m
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Ultrasound head tracker architecture
Ultrasound Extended Range
• Certain applications
require larger user
motion volume than a
transmitter can cover
• Several transmitters
multiplexed with one
receiver
• Only one transmitter
works at a time
• The computer switches
the transmitters
http://www.fkm.utm.my/~kasim/eng_design/vr/VR.Tracking.pdf
Other Trackers
Gloves
• Allow natural interaction with virtual world
– Grab
– Gestures
• Two types
– Force feedback
– Pinch gloves
Angle Measurement
• Measurement of the bend
of various joints in the
user’s body
• Used for:
– Reconstruction of the
position of various body
parts (hand, torso).
– Measurement of the motion
of the human body
(medical)
– Gestural Interfaces
• Sign language
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Angle Measurement Technology
• Optical Sensors
– Emitter and receiver on ends of
sensor
– As sensor is bent, the amount of
light from emitter to receiver is
attenuated
– Attenuation is determined by
bend angle
– Examples: Flexible hollow tubes,
optical fibers
– VPL Data Glove
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
24/02/2020
16
Angle Measurement Technology .
• Strain Sensors
– Measure the mechanical
strain as the sensor is bent
– May be mechanical or
electrical in nature.
– P5 Glove $25 (!)
– Cyberglove (Virtual
Technologies)
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Joints and Cyberglove Sensors
Interphalangeal
Joint (IP)
Metacarpophalangeal
Joint (MCP)
Thumb Rotation
Sensor
Proximal Interphalangeal
Joint (PIP)
Metacarpophalangeal
Joint (MCP)
Abduction Sensors
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Angle Measurement Technology ..
• Exoskeletal Structures
– Sensors mimic joint structure
– Potentiometers or optical encoders in joints report
bend
– Exos Dexterous Hand Master
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
Exoskeleton & Angle Sensors
• Analogous
• Tethered
• No identification problem
• Real-time
• No range limit
• Rigid body approximation
Pinch Gloves
• Have sensor contacts on the
ends of each finger
• Good as interfaces
• Good at grabbing
• No feedback
Monkey
• High accuracy
• High data rate
• Not realistic motion
• No paid actor
http://www.cise.ufl.edu/~lok/teaching/ve-s09/tracking.ppt
24/02/2020
17
Videos
• Dexmo exoskeleton glove lets you feel objects in
virtual reality
– https://www.youtube.com/watch?v=nin1_0EHU5g
• Stelarc: EXOSKELETON
– https://www.youtube.com/watch?v=vj0JUfmxkv4
• Paralysed walk again thanks to revolutionary
robotic exoskeleton that responds to brain waves
– https://www.youtube.com/watch?v=KZuXDSZ6JSA
Outdoor Tracking
GPS Trackers
• GPS is a technology widely used for outdoor
tracking
• The most important categories include
– Standard GPS
– Differential GPS
– Real-time kinematic GPS
GPS Trackers .
• Standard GPS is a satellite based positioning
system that utilizes a total of 29 satellites
– This will change with Galileo!
• The position of the user is determined by
processing radio signals from the satellites
• In theory, GPS systems can estimate the user’s
position, by calculating the arrival time of at
least three satellite signals
GPS Trackers .
• Satellites send position +
time
• GPS Receiver positioning
– 4 satellites need to be visible
– Differential time of arrival
– Triangulation
• Accuracy
– 5-30m+, blocked by weather,
buildings etc
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
GPS Trackers ..
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
24/02/2020
18
Problems with GPS
• Takes time to get satellite fix
– Satellites moving around
• Earths atmosphere affects signal
– Assumes consistent speed (the speed of light)
– Delay depends where you are on Earth
– Weather effects
• Signal reflection
– Multi-path reflection off buildings
– Signal blocking
– Trees, buildings, mountains
• Satellites send out bad data
– Misreport their own position
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Differential GPS
• Uses emitting
ground stations
that refine the
resolution
• Accurate to <
5cm close to
base station
– 22m/100 km
• Expensive
– $20-40,000 USD
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Differential GPS .
• The mobile GPS receiver
monitors signals from a
fixed radio transmitter and
another GPS receiver
• To refine the resolution
the transmitter sends the
corrected co-ordinates
– Based on the difference
between the known and
the computed positions
Assisted-GPS (A-GPS)
• Use external location server to send GPS signal
– GPS receivers on cell towers, etc
– Sends precise satellite position (Ephemeris)
• Speeds up GPS Tracking
– Makes it faster to search for satellites
– Provides navigation data (don’t decode on phone)
• Other benefits
– Provides support for indoor positioning
– Can use cheaper GPS hardware
– Uses less battery power on device
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
Assisted-GPS (A-GPS)
http://gps-response.com/61-a-gps.html
Cell Tower Triangulation
• Calculate phone position
from signal strength
– < 50 m in cities
– > 1 km in rural
Billinghurst, M. COSC 426: Augmented Reality, July 26th 2013.
24/02/2020
19
WiFi Positioning
• Estimate location based
on WiFi access points
– Use known locations of
WiFi access points
• Triangulate through
signal strength
– i.e. PlaceEngine
• Accuracy
– 5 to 100m
• Depending on WiFi density
Eye Tracking
Eye Tracking
• Eye trackers measure rotations of the eye in one
of several ways, but principally they fall into
three categories:
– VOG: Measurement of the movement of an object
(i.e. special contact lens) attached to the eye
– Optical tracking without direct contact to the eye
– EOG: Measurement of electric potentials using
electrodes placed around the eyes
https://en.wikipedia.org/wiki/Eye_tracking
Optical Tracking
• Light is reflected from the eye
and sensed
– Video camera or some other
specially designed optical sensor
• The information is then analyzed
to extract eye rotation from
changes in reflections
• To track over time video-based
eye trackers typically use
– The corneal reflection
• First Purkinje image
– The center of the pupil as features
https://en.wikipedia.org/wiki/Eye_tracking
Diagram of light and four Purkinje images
Video Oculography (VOG)
• VOG is a non-invasive, videobased
method of measuring
horizontal, vertical and
torsional position
components of the
movements of both eyes
– Using a head-mounted mask
that is equipped with small
cameras
– Usually employed for medical
purposes
https://en.wikipedia.org/wiki/Video-oculography
VOG examination in progress
VOG Operating Principles
• Iris tracking and highquality
video imaging
• Senses 3D linear
acceleration and 3D
rotational velocity
• Horizontal, vertical and
torsional eye movements
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
www.smivision.com/en/gaze-and-eye-tracking-systems/products/3d-vog.html
24/02/2020
20
VOG Performance
• Resolution
– Horizontal : 0.05°
– Vertical: 0.05°
– Torsional: 0.1°
• Head motion recording
– 3D rotational velocity [°/s]
– 3D linear acceleration [m/s2]
www.smivision.com/en/gaze-and-eye-tracking-
systems/products/3d-vog.html
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
Tradeoffs
• Highly accurate torsional measurement
• Permits comparison of slow phase velocity
(SPV) and head rotation velocity
• Useful for research and diagnosis
• Not practical for standard point-of-regard
research
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
Pupil - Corneal Reflection
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
drivingtraffic.com/wp-content/uploads/2010/08/eye.png
Pupil-Corneal Reflection
Operating Principles
http://www.ime.usp.br/~hitoshi/framerate/node2.html
www.archimuse.com/mw2010/papers/milekic/milekic.Fig1.jpg
Bright versus Dark Pupil
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
Pupil - Corneal Reflection Tradeoffs
• Ambient lighting
• Eye color
• Eyelashes
• Makeup
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
Electro-oculography (EOG)
www.virtualworldlets.net/Shop/ProductsDisplay/VRInterface.php?ID=90
www.adinstruments.com/solutions/images/eog_human.jpg
24/02/2020
21
EOG Operating Principles
• Permanent potential
difference between the
cornea and the fundus of
0.4 -1.0 mV
– Small voltages can be
recorded from the region
around the eyes which
vary as the eye position
varies
http://www.liv.ac.uk/~pcknox/teaching/Eymovs/emeth.htm
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
EOG Performance
• Accuracy : ± 2°
• Maximum rotation: ± 70°
• Linearity decreases
progressively for angles
> 30°
• Signal magnitude range:
5 – 20 µV/°
http://www.bem.fi/book/28/28.htm
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
EOG Tradeoffs
• Inexpensive
• Simple operation
• Need for frequent calibration and recalibration
– Corneoretinal potential can vary diurnally
– Affected by light and fatigue
– Drifting- electrode slipping, change in skin resistance
– Noise from other electrical devices, face muscles
– Blinking
Marchak, F.M., Eye Movement Recording, Veridical Research and Design Corporation, Society for Psychophysiological Research, 14 Sep 2011.
SMI Eye Tracking HMD for HTC Vive
• SMI’s eye tracking HMD based on HTC Vive
– Tracking : 250 Hz binocular
– Trackable field of view: Full field of view 110°
– Accuracy: 0.2°
• Data
– Inter-pupillary distance (IPD, pupil to pupil distance)
– Point of regard on display (left and right eye,
binocular)
– Gaze base point
– Gaze vectors (left and right eye, binocular)
https://www.smivision.com/eye-tracking/product/eye-tracking-htc-vive/
Eye Tracking Videos
• The Eye Tribe Tracker
– https://www.youtube.com/watch?v=XE0aANnzrL8
• GearVR eye tracking demo - using eyes for input
– https://www.youtube.com/watch?v=Wz61h6PrSsQ
• FOVE - Eye tracking use case
– https://www.youtube.com/watch?v=cmUYRkc6dPY
• Tobii Eye Tracker = Higher Gamer Skill Level
Increase
– https://www.youtube.com/watch?v=Kmil1Z84vnI
Questions