Concept
XR-SENSE will research and develop next-generation methods for multi-sensory 3D scene reconstruction, intelligent data compression, and interactive rendering, seamlessly blending high-fidelity visuals with realistic haptic feedback. By integrating heterogeneous sensor data, real-time geometric tracking, and cross-modality coding, the project will establish a robust framework for immersive and inclusive XR experiences.
The rapid evolution of spatial computing and the widespread adoption of mixed realities have fundamentally transformed how we interact with digital content. However, today’s XR experiences remain overwhelmingly confined to the audio-visual domain, neglecting the sense of touch which is vital for natural human communication, spatial awareness, and professional precision. Furthermore, existing immersive applications often suffer from high bandwidth demands, physical hardware limitations, and a lack of inclusive design for diverse user needs.
The vision of XR-SENSE is to break these boundaries by pioneering an AI-driven, human-centered visuo-haptic medium. By transforming XR from something we merely see and hear into something we can actively feel and physically interact with, XR-SENSE will unlock unprecedented capabilities in autonomous mobility, remote medicine, and collaborative digital broadcasting.
Objectives & Ambition
/
01
Research and develop the intelligent processing unit of multimodal visual/geometry and haptic data.
Research and develop a novel haptic media authoring tool (HaptiForge). This tool will allow modeling and editing of both direct/physical haptic interactions and symbolic/tactile cues, enabling richer and more flexible multimodal XR experiences.
/
02
Research and development of an intelligent coding unit and an interactive rendering unit.
Develop scalable multimodal coding, transmission, and rendering solutions, based on saliency-driven multiscale LoD outputs of xSense, leveraging the use of learning-based representations. Research and develop the advanced xStream interaction module enabling novel forms of affective, tactile interaction through new wearable haptic interfaces, thus boosting immersion and presence.
/
03
Research on the interdisciplinary foundations of perception, cognition, and interaction in shared visuo-haptic XR environments.
Define innovative and meaningful quantitative and machine-readable priors for the optimal development of the rendering module. Special focus on the use of haptic rendering tools to convey emotions and on collaborative affective visuo-haptic interaction.
/
04
Distributed XR-space with advanced communication and edge intelligence.
The XR-space platform will support multi-human to multi-object interactions in challenging use cases that can be easily used by everyone. XR-SENSE will also challenge the potential of cutting-edge communications technology. Additionally, distributed processing will be supported and the potential of federated learning will be investigated for specific processes, where sharing domain knowledge in real time could improve performance.
/
05
Showcase and evaluate XR-SENSE in three pilots, strategically selected to cover rapidly advancing sectors.
Connected and autonomous vehicles: i) Immersive XR infotainment, co-presence of remotely located users in interactive XR, ii) Collaborative awareness and XR rendering of potentially occluded objects and information of interest
Remote collaborative surgery: i) Remote collaborative surgery through reconstruction of surgical room and remote surgeon, ii) Advanced superimposition and haptic interaction of 3D tomographic images, iii) Affective multimodal interactions between patients and their remotely located family members.
Bidirectional immersive XR broadcasting: i) Immersive live broadcasting of cultural and educational events; ii) Social telepresence and collaborative broadcasting.
/
06
Develop strong liaison with standardization activities.
Assure that developments within the project will comply with existing and emerging standards. Contribute to standardization activities XR-SENSE partners already lead/participate in and join new. Boost dissemination and exploitation to enhance sustainability and uptake of the results. Promote concertation activities with other complementary EU projects.
Three Pillar Modules
xSense
Intelligent Multimodal Processing & Reconstruction
- Focuses on spatiotemporal processing (denoising, completion, and automated co-registration) of Point Clouds (PCs), Meshes, and Radiance Fields (RFs).
- Extracts the “haptic medium” implicitly from captured geometries using Machine Learning to estimate material stiffness.
- Introduces HaptiForge, a semi-automatic authoring tool for modeling direct physical interactions and symbolic/tactile cues (like emotional touch).
xStream
Adaptive Coding & Interactive Rendering
- Develops learning-based compression algorithms for joint visuo-haptic and cross-modality coding.
- Utilizes an adaptive AI agent to optimize streaming and rendering based on cognitive, perceptual, and spatial saliency priors.
- Integrates cutting-edge wearable haptic interfaces (armbands, rings, thimbles) developed by partners like UNISI and bHaptics to provide tactile, thermal, and skin-stretch feedback.
xHuman
Human-Centered & Inclusive Interaction
- Integrates Social Sciences and Humanities (SSH) into the technical pipeline by design.
- Tracks physiological signals (heart rate variability, skin conductance, eye-tracking) to minimize the Prediction Error (PE) that causes cybersickness and reduces immersion.
- Translates qualitative human factors into machine-readable priors to guide the xStream AI optimization agent.
Consortium
Coordinator
The University of Patras was founded in the city of Patras in 1964 and started to operate in academic year 1966-1967. In June 2013 the University of Western Greece was integrated into the University of Patras. In 2019 the majority of the departments of the Technological Educational Institute of Western Greece were also integrated into the University of Patras.
Participants
Project Structure
The project is divided into six Work Packages (WP):
- Perform the administrative, scientific, technical, financial and innovation management of the project.
- Coordinate the contacts with the EC.
- Ensure that the deliverables are submitted on time. Complete the project
within the required time frame. - Ensure the communication flow and information circulation among the partners.
- Monitor project activities in line with its objectives and ensure their quality.
- Ensure that legal issues, patents & IPR are dealt according to contract & Consortium Agreement
- Perform data and ethics management.
- Kick-off project activities with fundamental elements like a review of best practices, requirements, methodology and architecture.
- A SoA analysis and selection of best practices that will be used.
- The description of the architecture, use cases and the system’s specifications.
- The methodology of the UCD.
- The study and specifications of the users and system-level requirements.
- The definition of the used specific KPI metrics
- Develop advanced haptic interfaces and representation methods, establishing the basis of a new visuo-haptic medium
- Develop the xSense module for multimodal acquisition, reconstruction, and spatiotemporal/latent processing
- Develop the xStream module for joint coding, compression, and adaptive rendering of visual and haptic modalities
- Develop HaptiForge, an innovative haptic media authoring tool for modeling, editing, and symbolic/physical integration of haptic content
- Research and develop the xHuman module, integrating perceptual, cognitive, affective, and social interaction factors into XR-SENSE.
- Deliver quantified, machine-readable models of SSH factors that guide and optimize xSense (processing) and xStream (coding and rendering).
- Develop and provide access to the necessary communication and cloud/edge infrastructure for scalable multimodal XR interaction.
- Enable layered local, distributed, or federated processing according to requirements and conditions.
- Implement distributed shared memory storage and secure, privacy-preserving data management.
- Integrate all the sub-systems and components, implemented in previous tasks, into the final XR-SENSE prototype.
- Ground the pilots’ specifications and assessment protocols.
- Set the ground for small-scale trials and quantification campaigns.
- Implement the final use cases and pilots.
- Design the validation and evaluation methods.
- Raise public awareness of project developments among key user groups, the scientific community, XR and EU stakeholders and the general public.
- Facilitate sharing of knowledge inside the Consortium.
- Provide market analysis, business plan and an exploitation strategy for the project outcomes.
- Investigate and contribute to relevant standards.
- Collaborate with related European initiatives.