11:00-11:10 Jiri Matas

Opening

11:10-11:45 Dimosthenis Karatzas (Univ. Autónoma de Barcelona, ES)

Trustworthy End-to-End Document Understanding slides - Automatic document processing enables the vast majority of daily interactions with and between institutions. From a research viewpoint, document understanding is a multimodal endeavour combining the visual analysis of document images with language processing. Document Visual Question Answering (DocVQA) has quickly reshaped the state of the art, converting document understanding into a key benchmark for all modern multimodal VLMs. However, it is easy to show that such models tend to memorize information from their training set and often hallucinate responses drawing information from their training data, a fact that raises concerns about the treatment of sensitive information in the training set. In this talk I will give an overview of the research that has taken place in the context of the European Lighthouse on Safe and Secure AI, aiming to address mainly privacy concerns, but also adversarial robustness and explainability, in the domain of document understanding. [abstract]

11:45-12:20 Marc Masana (TU Graz, AT)

Towards More Efficient and Adaptive Continual Learning Models slides - TBF [abstract]

12:20-13:30 Lunch break

13:30-14:05 Tatiana Tommasi (Politecnico di Torino, IT)

Efficient Learning, Unlearning, and Transferring for Vision and Language Models slides - One of the key challenges for AI models in open-world scenarios is handling novelty and adapting to a large variety of conditions. Large-scale, or so-called foundation models, come close to addressing these challenges by being trained on massive amounts of data and a wide range of tasks, capturing extensive knowledge variability. However, several open questions remain: What is the most effective way to leverage these models (zero-shot/fine-tuning)? Can they be safely pruned before being transferred to downstream tasks? What are the most efficient fine-tuning strategies when aiming to merge multiple specialized models for multi-task capabilities, or when the goal is to remove knowledge as in task negation or unlearning? In this talk, I will explore these questions and present recent findings on pruning and editing for vision and language models. [abstract]

14:05-14:40 Eric Brachmann (Niantic, DE)

Pushing the boundaries of structure-from-motion with machine learning slides - We are currently witnessing a remarkable renaissance of interest in structure-from-motion (SfM). Of course, SfM was never gone. Rather, solutions based on feature-matching and traditional multi-view geometry matured to a state about 10 years ago that turned them into reliable off-the-shelf components for various tasks in 3D vision. Still, traditional SfM approaches are most reliable when certain conditions are met. For example, reconstructing very few or a huge amount of images can be challenging. The talk will investigate how learning-based formulations of SfM can address these challenges. We will focus on scene coordinate regression, an implicit scene representation, that naturally avoids the explosion of complexity inherent to image-to-image matching when the number of images is large. The talk culminates in the presentation of ACEZero, a self-supervised scene coordinate regression pipeline, that is able to reconstruct 10.000 images in reasonable time. [abstract]

14:40-15:30 Coffee break

15:30-16:05 Pascal Mettes (University of Amsterdam, NL)

Hyperbolic Deep Learning slides - From linear layers and convolutions to self-attention, we are building all our advances in deep learning on a Euclidean foundation. But is Euclidean geometry the right choice? In this talk, I will dive into hyperbolic geometry for deep learning. I will discuss what hyperbolic geometry is, outline the inherent limitations of Euclidean geometry for deep learning, and discuss the potential of hyperbolic deep learning. I will outline recent works demonstrating the power of hyperbolic learning for hierarchical recognition, uncertainty, generalization, and especially for vision-language models. Lastly, I will show our ongoing efforts towards fully hyperbolic networks and how to get started in this field with our new hyperbolic learning library. [abstract]

16:05-16:40 Vladan Stojnic (CTU in Prague, CZ)

Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation slides - We propose a training-free method for open-vocabulary semantic segmentation using Vision-and-Language Models (VLMs). Our approach enhances the initial per-patch predictions of VLMs through label propagation, which jointly optimizes predictions by incorporating patch-to-patch relationships. Since VLMs are primarily optimized for cross-modal alignment and not for intra-modal similarity, we use a Vision Model (VM) that is observed to better captures these relationships. We address resolution limitations inherent to patch-based encoders by applying label propagation at the pixel level as a refinement step, significantly improving segmentation accuracy near class boundaries. Our method called LPOSS+, performs inference over the entire image, avoiding window-based processing and thereby capturing contextual interactions across the full image. LPOSS+ achieves state-of-the-art performance across a diverse set of datasets. [abstract]

11:00-11:10: Jiri Matas

Opening

11:10-11:45: Vincent Leroy (Naver Labs Europe, FR)

From CroCo to MASt3R: A Paradigm Change in 3D Vision? slides - In the rapidly evolving field of 3D vision, a multitude of heterogeneous downstream tasks coexist, such as visual localization, depth estimation, 3d reconstruction, etc. The current paradigm is to develop a dedicated method to solve each task, thereby largely ignoring their potential inter-connections and synergies. Developing unified models able to handle multiple 3D geometric downstream tasks remains a challenge. This presentation introduces three interconnected advancements in this respect: CroCo,  DUSt3R and MASt3R. CroCo, a self-supervised pre-training framework, utilizes a pretext task to lay foundations for DUSt3R/MASt3R, a unified foundational model for geometric 3D vision. Cross-view completion, or CroCo in short, first serves to learn robust representations of 3D geometry from pairs of images depicting the same scene from different viewpoints. By masking parts of an image and predicting these given another viewpoint of the scene, CroCo effectively captures priors about spatial relationships and geometric information, setting a strong foundation for downstream 3D vision tasks. Then, building on the robust pre-trained models provided by CroCo, DUSt3R introduces a novel approach for Dense Unconstrained Stereo 3D Reconstruction. This method revolutionizes traditional multi-view stereo reconstruction by regressing pointmaps that encode scene geometry without requiring calibrated or posed cameras. DUSt3R simplifies the complex pipeline of traditional 3D reconstruction methods, significantly reducing computational overhead and enhancing performance across various benchmarks. MASt3R further extends DUSt3R by adding the ability to establish accurate pixel correspondences. The journey from CroCo to MASt3R exemplify a significant paradigm shift in 3D vision technologies. This presentation will delve into the methodologies, innovations, and synergistic integration of these frameworks, demonstrating their impact on the field and potential future directions. The discussion aims to highlight how these advancements unify and streamline the processing of 3D visual data, offering new perspectives and capabilities in robotic navigation, cultural heritage preservation, and beyond.  [abstract]

11:45-12:20: Johan Edstedt (Linköping University, SE)

Why Image Matching is Still not Solved, and How to Solve it slides - Image matching, and structure-from-motion in general, has been unique among computer vision tasks in its ability to resist learning-based approaches. In this task, I will discuss some of the major challenges involved for learning-based image matchers, and how to solve them. [abstract]

12:20-13:40: Lunch break

13:40-14:15: Elisa Ricci (University of Trento, IT)

Beyond Zero-Shot Generalization: Strengthening Vision-Language Models with Test-Time Adaptation - Groundbreaking achievements in Vision-Language pretraining have increased the interest in crafting Vision-Language Models (VLMs) that can understand visual content alongside natural language, enabling a new definition of zero-shot classification. Despite huge pretraining databases, VLMs still face limitations, suffering from performance degradation in case of large train-test dissimilarity and requiring the design of highly generalizing textual templates. Test-Time Adaptation (TTA) can effectively improve the robustness of VLMs by adapting a given model to online inputs. In this talk, I will present an overview of a few recent works from my research group on TTA and other strategies aimed at improving the robustness of VLMs across various downstream tasks. [abstract]

14:15-14:50: Jan Šochman (CTU in Prague, CZ)

Pixel-Level Out-of-Distribution Detection slides - We propose a dense image prediction out-of-distribution detection algorithm, called PixOOD, which does not require training on samples of anomalous data and is not designed for a specific application which avoids traditional training biases. In order to model the complex intra-class variability of the in-distribution data at the pixel level, we propose an online data condensation algorithm which is more robust than standard K-means and is easily trainable through SGD. We evaluate PixOOD on a wide range of problems. It achieved state-of-the-art results on four out of seven datasets, while being competitive on the rest. [abstract]

14:50-15:35: Coffee break

15:35-16:10: Efstratios Gavves (University of Amsterdam, NL)

Grounding Foundation Models in Reality with Physics- and Causality-informed World Models - Foundation Models have taken the AI world by storm with their ability to generalize to novel, complex tasks beyond the scope of their training data. Whether they achieve this through true generalization or by ‘remembering’ vast training sets is debatable, but their performance is undeniably superior to previous approaches. With this success, the critical questions now are: What are the limitations of current large-scale learning? What frontiers remain? And what are the implications of these developments? In this talk, I argue that Robot Learning represents one of the most exciting frontiers, with Robot General Intelligence standing as the quintessential goal for AI. However, current approaches to Robot Learning face fundamental limitations that prevent them from achieving this ideal. I will present my recent work on a novel paradigm that incorporates explicit physical and causal priors into world models, enabling real robots to perform one-shot policy learning. This approach allows robots to learn new tasks from as little as one demonstration, pushing the boundaries of what’s possible in real-world robot learning. [abstract]

16:10-16:45: Radu Timofte (University of Würzburg, DE)

Advances in Image Restoration and Enhancement slides - This talk will provide an overview of the recent research conducted at the Computer Vision Lab, University of Wurzburg, focusing on the advances in image restoration, enhancement and manipulation, as key components in Computer Vision and AI. In particular, we will discuss topics such as restoration, raw processing, learning ISPs, quality enhancement, domain translation and synthesis, multimodality, efficiency and mobile AI. A special attention will be given to the complexity of the solutions and their potential for real-world applications. [abstract]

11:00-11:10: Jiri Matas

Opening

11:10-11:45: Rafal Mantiuk (University of Cambridge, UK)

How should we measure progress in computer vision and computational photography? slides - The improvemen - When working in the field of autonomous driving applications and mobility in general, there is a significant challenge to transfer results from the more controlled research environment into real-world scenarios. Deep learning approaches showcase impressive capabilities and advancements for particular benchmarks, but we tempt to hit a wall when rying to deal with the diverse and unpredictable conditions in real life. For one thing Autonomous vehicles encounter a wide array of scenarios, from varied weather conditions to complex traffic situations, changes in scenery etc. Furthermore, for autonomous systems to be safe, it needs to provide some 3D awareness, requiring accurate distance estimation. And last but not least, there is the expectation of the industry to reduce annotation costs, and the models to be applicable for various mobility setups and configurations, even in circumstances where the setup conditions are unknown. In this talk we will discuss some of the pragmatic approaches that we use to tackle some of those problems. We focus on the important task of vehicle and pedestrian detection. We disentangle the calibration aspects, image space segmentation and detection, and 3D monocular uplifting. And we introduce a concept of "camera independence" to make sure the applicability of the model can extend the domains it has been trained for. We illustrate with a variety of examples. briefly introduce a family of visual difference predictors (VDPs), which are image and video metrics that are derived from psychophysical models of human vision and offer additional capabilities lacking in popular metrics. [abstract]

11:45-12:20: Gul Varol (Ecole des Ponts ParisTech, FR)

Audio Description Generation for Movies using Vision-Language Models slides Audio Description (AD) is the narration to describe visual elements in movies or TV series, originally to assist visually impaired audiences. Creating high-quality AD annotations demands significant time and effort, posing a challenging task even for humans. This is due to the necessity for long-term memory, detailed visual understanding, and accurate character recognition. This talk will introduce a series of works, named AutoAD, that leverages video-language models to achieve the automatic generation of AD. [abstract]

12:20-13:30: Lunch break

13:30-14:05: Fatma Guney (Koc University, TR)

Learning Object-Centric Representations for Self-Driving slides - The choice of representation plays a key role in self-driving. Bird’s eye view (BEV) representations have shown remarkable performance in recent years. In this talk, I will introduce learned object-centric representations in BEV to distill a complex scene into more actionable information for self-driving. We first learn to place objects into slots with a slot attention model on BEV sequences. Based on these object-centric representations, we then train a transformer to learn to drive as well as reason about the future of other vehicles. We found that object-centric slot representations outperform both scene-level and object-level approaches that use the exact attributes of objects. Slot representations naturally incorporate information about objects from their spatial and temporal context such as position, heading, and speed without explicitly providing it. Our model with slots achieves increased coverage of the provided routes and, consequently, a higher driving score, with a lower variance across multiple runs, affirming slots as a reliable alternative in object-centric approaches. Additionally, we validate our model’s performance as a world model through forecasting experiments, demonstrating its capability to accurately predict future slot representations. [abstract]

14:05-14:40: Bill Psomas (NTUA, GR)

Leveraging Attention in Masked Image Modeling and Pooling slides - This talk explores innovative methodologies that leverage attention mechanisms to refine the process of masked image modeling (MIM) and enhance spatial pooling strategies in both vision transformers and convolutional networks. The first segment of the talk delves into MIM, pivoting away from the conventional random masking strategies towards an informed masking technique guided by attention—called AttMask. AttMask makes use of the inherent self-attention maps generated by transformer encoders to identify and mask the most salient image tokens. Through this, AttMask facilitates a more challenging pretext task in MIM, prompting the network to hone in on the most discriminative visual cues. The second part discusses the evolution of spatial pooling in vision models. By extracting a more distilled and spatially aware representation at the end of the network, SimPool is introduced, a versatile attention-based pooling mechanism. SimPool transcends the boundaries of supervised and self-supervised learning, providing high-quality attention maps that accurately outline object boundaries. This "universal" pooling illustrates that quality attention maps in vision transformers can be achieved without the need for self-supervised specificity or architectural modifications [abstract]

14:40-15:20: Coffee break

15:20-15:55: Tomas Hodan (Meta, CH)

Object pose estimation for augmented reality slides - I will motivate applications of object pose estimation for extended (virtual, augmented, mixed) reality, introduce devices and data we use at Meta Reality Labs, and describe our recent works. I will primarily focus on FoundPose, our recent method for unseen object pose estimation based on DINOv2. Besides, I will give an update on the BOP bechmark and introduce HOT3D, our new dataset for 3D tracking of hands and objects that will be publicly released in May. [abstract]

15:55-16:30: Jakub Paplham (CTU in Prague, CZ)

A Call to Reflect on Evaluation Practices for Age Estimation: Comparative Analysis of the State-of-the-Art and a Unified Benchmark. - Comparing different age estimation methods poses a challenge due to the unreliability of published results stemming from inconsistencies in the benchmarking process. Previous studies have reported continuous performance improvements over the past decade using specialized methods; however, our findings challenge these claims. This paper identifies two trivial, yet persistent issues with the currently used evaluation protocol and describes how to resolve them. We describe our evaluation protocol in detail and provide specific examples of how the protocol should be used. We utilize the protocol to offer an extensive comparative analysis for state-of-the-art facial age estimation methods. Surprisingly, we find that the performance differences between the methods are negligible compared to the effect of other factors, such as facial alignment, facial coverage, image resolution, model architecture, or the amount of data used for pretraining. We use the gained insights to propose using FaRL as the backbone model and demonstrate its efficiency. The results emphasize the importance of consistent data preprocessing practices for reliable and meaningful comparisons. We make our source code public. [abstract]

10:55-11:00: Jiri Matas

Opening

11:00-11:35: Marc Proesmans (KU Leuven, BE)

In search for camera and data independence for mobility applications - When working in the field of autonomous driving applications and mobility in general, there is a significant challenge to transfer results from the more controlled research environment into real-world scenarios. Deep learning approaches showcase impressive capabilities and advancements for particular benchmarks, but we tempt to hit a wall when rying to deal with the diverse and unpredictable conditions in real life. For one thing Autonomous vehicles encounter a wide array of scenarios, from varied weather conditions to complex traffic situations, changes in scenery etc. Furthermore, for autonomous systems to be safe, it needs to provide some 3D awareness, requiring accurate distance estimation. And last but not least, there is the expectation of the industry to reduce annotation costs, and the models to be applicable for various mobility setups and configurations, even in circumstances where the setup conditions are unknown. In this talk we will discuss some of the pragmatic approaches that we use to tackle some of those problems. We focus on the important task of vehicle and pedestrian detection. We disentangle the calibration aspects, image space segmentation and detection, and 3D monocular uplifting. And we introduce a concept of "camera independence" to make sure the applicability of the model can extend the domains it has been trained for. We illustrate with a variety of examples.[abstract]

11:35-12:10: Oriane Siméoni (VALEO.AI, FR)

Object localization (almost) for free harnessing self-supervised features. slides - The localization of objects in images is at the heart of many perception systems today. However, training object detectors requires large and expensive campaigns of annotation for a finite and pre-defined vocabulary. Instead, being able to discover objects in images without knowing in advance what objects populate the dataset is an exciting prospect. In this talk we will discuss solutions to exploit self-supervised pre-trained features to perform class-agnostic object localization with zero supervision, and such without requiring object proposals or the exploration of image collection. We will also explore means to unite the class-agnostic object localizations with open-vocabulary features in order to perform open-vocabulary semantic segmentation.[abstract]

12:10-13:10: Lunch break

13:10-13:45: Federico Tombari (Google/TU Munich, DE/CH)

3D scene understanding for real world applications - 3D scene understanding is a key enabler for applications in augmented reality, autonomous driving and robotics. The ability for 3D scene understanding methods to work in real world scenarios and to generalize across various domains is of fundamental importance for their applicability. In this talk we will look at two recent trends that aim for this goal. The first one is open vocabulary 3D scene understanding: we will introduce the core ideas and illustrate some of the main works in the literature for open vocabulary semantic segmentation from explicit and implicit 3D representations. The second one is using scene graphs as underlying representation for 3D scenes. Also in this case we will briefly introduce and discuss the main concepts, methodologies and datasets for 3D scene graph inference from 3D and RGB data.[abstract]

13:45-14:20: Lea Müller (Max Planck Institute, DE)

3D Human Mesh Reconstruction for Poses with Contact - Humans are social creatures and interaction plays an important role in our everyday life. The field of computer vision should be able to reconstruct such interaction in 3D from an image. However, this is a challenging task that requires tailored datasets and reconstruction methods. We will introduce BUDDI, a generative model that learns the joint distribution of 3D poses and shapes during interactions. Besides sampling new meshes from BUDDI, we can use this model as a prior during optimisation. To do this, we draw inspiration from the 3D content creation community and propose using a score distillation loss when fitting SMPL meshes to images. We evaluate our method of various benchmark and demonstrate that we can reconstruct more realistic human interactions from images such as hugs, people posing for pictures, or shaking hands.[abstract]

14:20-14:50: Coffee break

14:50-15:25: Tomas Jakab (University of Oxford, UK)

Learning Articulated 3D Animals in the Wild. slides - Reconstructing articulated objects, such as animals, from a single image has numerous practical applications, from robotics to 3D virtual asset creation. Predicting the 3D shape, articulation, viewpoint, and texture from a single image necessitates prior knowledge of the possible shapes and appearances of the object. Typically, learning this requires multi-view images or 3D scans. While such data are available for some types of objects like humans, it is unlikely that this approach will scale to the long tail of objects that appear in the world, including countless different animals. In this talk, we will explore how we can learn this 3D prior using only 2D internet images, which are available in abundance.[abstract]

15:25-16:00: Jonas Kulhanek (CIIRC - CTU in Prague, CZ)

Learning 3D scene representations for novel view synthesis. slides - Novel view synthesis and 3D scene representation are central topics of 3D vision, with many applications spanning areas such as AR/VR, robotics, and gaming. Volumetric representations, such as Neural radiation fields (NeRFs), became a dominant direction in the field, providing photorealistic rendering results at real-time rendering speeds. A popular scene representation used by NeRFs is to combine a uniform, voxel-based subdivision of the scene with an MLP. However, these representations do not use geometry priors (even though point clouds are often available), which hurts the learning time or the memory requirements. An alternative is to use an adaptive representation based on tetrahedra obtained by Delaunay triangulation instead of uniform subdivisions or point-based representations. Such a representation can enable efficient training and match the state-of-the-art results. This approach elegantly combines concepts from 3D geometry processing, triangle-based rendering, and modern neural radiance fields, and compared to voxel-based representations, it can provide more detail around parts of the scene likely to be close to the surface.[abstract]