Y Soft AIVA system is using software robots for automated testing of web applications. These software robots utilize computer vision, image segmentation and classification to analyze content of web pages visually. The benefits of the visual location approach over the locator/component based approach lie in increased reliability and true black box testing without having to interfere with application internals.

The visual approach has its own challenges though. The software robot needs to be reliable able to infer web page structure and semantics purely from the captured image(s). There are helper guidelines, user provided hints or metadata available. To make matters more complex, a single element that needs to be located and interacted with can exist in multiple instances in a single web page. Consider a "street address" field in an order form of a typical e-shop. This field exists both as part of the billing and shipping addresses.

The AIVA system implements computer vision how to turn the web page into hierarchical hybrid model, which combines structure and semantics. This model relies on image segmentation. The goal of this student thesis is to explore an alternative to the current segmentation approach using: deep reinforcement learning (DRL) and computer vision. For comparison, provide also an implementation using the YOLOv8 model.

Image segmentation models are usually trained on diverse datasets and mostly not optimized for web pages or graphical user interfaces in general. Student shall implement an experimental tool that segments an image into a hierarchy of sections. Considering our previous example, this experimental implementation would identify an e-mail address field within the billing address section and distinguish it from a different address field within the shipping address section. 

This experimental tool is sufficient to work on simple examples only. It does not have to take text or complex raster images into account. The main objective is to evaluate segmentation within hierarchies of shapes.

The thesis should answer the following questions:
1. Which of these three approaches seems to be the most suitable approach for creating a hybrid structural/semantic model for a web page in AIVA?
2. Are there any optimization methods that can be used to further improve the reliability of this approach?
3. What data (in volume and quality) should be gathered to create a sufficiently complex data set for successful use of this approach in the AIVA system?