Surveillance and
Monitoring Systems based
on Complex Event
Processing
Tomáš Pitner
Masaryk University, Czech Republic
BIS 2012, Vilnius, Lithuania, May 23rd
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 2
Surveillance and Monitoring Systems
! The monitoring system continuously
! collects data from the monitored environment,
! evaluates this data and events that are essential
! takes the user's attention to important events,
! or is able to respond automatically.
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 3
Surveillance and Monitoring Systems
! TARGET
! Used to monitor the operation of various objects
! SCOPE
! From a single application, computer, or other device
! Upto large infrastructures
! USERS
! They help routine operators as well as strategic
management
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 4
Surveillance and Monitoring Systems
! PURPOSE
! Check functionality - detect faults
! Monitor reliability – discover, prevent outages
! Protect - find external and internal threats
! Explore what could not be modelled - unusual behavior
! Save by discovering frauds - even when not obvious
! Optimize - tune the operation
! Measure performance – real-time KPIs
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 5
Surveillance and Monitoring Systems
! BENEFITS
! See the current situation immediately
! Aggregated and visualized form
! Measure KPI in real-time
! Standard behavior profiles (devices, systems, people)
! Ensure business rules compliance
! Guarranty SLAs
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 6
Surveillance and Monitoring Systems
! IMPROVE THE SYSTEMS
! Shorten the (incident, event) detection time
! Reduce the time to discover the cause
! Make the intervention more effective
! Reduce staff costs, require less qualified personnel
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 7
Maturity Levels • Auto-adaptation
Feedback – users, environment
Level 4 -
Adaptivity
• Pro-active SMS
• Early warning
Level 3 –
Forecasting
• Advances SMS
• Discover important situation events based
on events, rules, context, and history
Level 2 – Detection
and Evaluation
• Baseline SMS
• Get interesting events and present
them to people
Level1 –
Filtering and Routing
Events
• Log storage
• Collecting, cleaning, storing
Level 0 –
Event Collection
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 8
Technology Requirements
! Various domains and data origin
! Measurements
! Logs
! Lower-level systems
! Large data volumes in realtime
! 10k+ events per sec
! Context, correlation, adaptation
! How to achieve it ?
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 9
Complex Event Processing Technology
! Real-time (or near RT) fast data flows
! Behaviour pattern detection
! Fraud detection, smart-grids, logistics, telco
CEP engine
Many
simple
events
RT aggregation
correlation
Complex
events
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 10
CEP-based Surveillance and Monitoring Center
Monitoring
Systems
Databases Logs
Sensor
measurements
Enterprise Service & Event Bus
CEP
Engine
Event
Store
Real-time Dashboards
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 11
Further Outline of the Talk
! Domains
! Technology
! Applications
! Partners
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 12
Application Domains
! Computer network security
! Facility management
! Fraud detection in large enterprises / sales networks
! Computing resources, Clouds
! Complex technological blocks
! Industrial production
! Smart grids, power networks
! Precise agriculture
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 13
Computer network security
! Based on Netflow monitoring
! Complement to host-based approach
! Instead of restrictive policies
! The only way in open, research institutions
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 14
Case of Masaryk University
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 15
Hierarchical nature
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 16
CEP-based Analysis and Other Processing
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 17
Facility management
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 18
Facility management
! WHY SO INTERESTING (in general and for us)?
! Masaryk University
! 9 Faculties, 40000+ students, 4500+ staff members
! 250+ buildings (150 own), 20500+ rooms
! 350000+ m2
! Campus of Masaryk University, the largest in CZ
! Technological equipment in newer buildings
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 19
Facility management @Campus of Masaryk Uni
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 20
Facility management / What is observed?
! 30+ buildings at Campus, 100000 m2
! Heating, cooling, air conditioning, moisturizing
! Security: fire detectors, access control, cameras
! Other: audiovisual equipment, lighting, power supply,
waste management
! Building monitoring system
! 100k records / day
! Alerts, Visualization
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 21
Fraud detection
! Applied for fuel-fraud detection at a gas-station network
! Many events from various domains
! Accounting/billing
! Fuel level in tanks
! Volumes sold
! Volumes supplied
! Saves dozens of M CZK annually for a moderate-size chain
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 22
Computing resources, Clouds
! All old-hat problems (grids) and many new ones
! Multi-tenancy
! IaaS (machine provider)
! PaaS (platform provider)
! SaaS (app provider)
! Make it simple, easy to integrate
! Secure (the players should be isolated)
! Keep overhead low
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 23
Industrial Production
! Monitoring enhancement for production information
system PHARIS
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 24
Industrial Production – What is monitored?
! Machines
! Number active/working, cycles done
! Operators, logins
! Operations performed on machines
! Reactions to events
! Derive the machine cycle profiles
! Detect faults, anomalities
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 25
Energy Production and Distribution, Smart-Grids
! Monitoring large smart-meter networks
! Monitoring and controlling alternative energy sources
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 26
Example in Smart-Grids
! Large smart-meter infrastructure
! milions of smart-meters in CZ
! What we know?
! Status (consumption) every 15 min
! Outages, failures at SM or communication
! Switch-off
! Unauthorized manipulation
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 27
Example in Smart-Grids: The Size
! Magnitude of up to 10 M smart-meters
! 10 TB of raw data
! 15 mins time frames important for some processing
! Legacy analytical apps
! New apps not just for smart-grids but also for smaller
infrastructures/suppliers
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 28
Rounding up...
Achieving higher SMS levels:
Level 0 – collecting data
Level 1 – basic patterns
… all for most legacy SMS … but we need:
Level 2 – advanced
Level 3 – predictive
Level 4 – adaptive
Monitoring Systems based on Complex Event Processing / Faculty of Informatics - LaSArIS 29
Rounding up...
! CEP-based monitoring
! Large-size, large data volume apps
! CEP allows down-sizing, supports hierarchical structure
! Recursive processing (low- and high-levels together)
! Multi-domain nature
! Context-aware monitoring
! Identification of common patterns
Thank you for your attention!
Questions? At any time to
tomp@fi.muni.cz
Tomáš Pitner
Masaryk University, Czech Republic