Διαχείρηση διαδικτυακού φυλλομετρητή με χρήση πολυτροπικών συσκευών και παρουσίαση του πλαισίου C.O.A.L.S.

Abstract

-

Humans interact with each other and with their environment in everyday life by utilizing the five basic senses as input modalities while using for output, modalities like sounds, gestures, gaze, facial and body expressions. In the other hand, when talking about human computer interaction, the existence of a difficulty of interaction between both sides, is always present. Thus through Human Computer Interaction science (HCI), this kind of interaction tends to be more user friendly and operative through intuitive actions related to the frequent and natural human behavior. Multimodal interaction provides an alternative means of communication with an application, which is different enough from the traditional WIMP style. Despite the great amount of devices in existence, most applications make use of a limited set of modalities, most notably speech and touch. This thesis describes the procedure to interact with Google Chrome browser through browser extensions, while performing hand and finger gestures and using them as multimodal input to the system from the Leap and the Kinect sensors. In addition, it is introduced the interactive multimodal framework C.O.AL.S., which enables the deployment of a vast variety of modalities. More specifically by fusing - combining different modalities from different sources or from a single source, in order to achieve the interpretation and extraction of semantic meaning from them which can vary according to the context, the task, the user and the time and finally to give the user the correct feedback according to the information gathered from the previous procedure.