In everyday situations, we often speak while moving (e.g., walking together, cooking, knitting), and we move while speaking (e.g. head, arms and hands, posture). Breathing mediates between speech and motion, and is a source for these activities: It provides the brain and muscles with the required oxygen, and the expiratory airflow is needed for the production of sound. Breathing is also a pacemaker for the speech flow: inhalation pauses are coordinated with prosody and syntax. Following recent theories on situated and embodied cognition and language, Salammbo’s originality will be to consider limb motion as a common context for spoken language, and breathing as a mediator between limbs and spoken language. It will adopt an interdisciplinary approach integrating linguistics, movement science and psychology. The first aim is to create a novel multimodal corpus, with simultaneous recordings of limb motion, respiratory, articulatory and acoustic data using advanced technology. A cross-linguistic longitudinal approach will be adopted. Native speakers of French and German will read and retell stories including novel words on three different days. Idiosyncratic properties of the speakers known to influence breathing and limb movements, i.e. physical fitness and body shape, will be taken into account as determinant factors of speech production in the context of body motion. To further assess the speech-breathing-limb link, speech tasks will be performed in different movement conditions with no motion, free hand motion, and rhythmic motions of the legs or hands. Based on this corpus, the link between spoken language, breathing and limb motion will be addressed in four working packages analyzing: a) the impact of idiosyncratic physical properties on body motion, respiration and different linguistic levels ranging from phonetics to syntax; b) the impact of body motions on speech planning, prosodic and segmental properties; c) the coordination between speech, breathing and limb motion using sophisticated time series analyses of synchronizations; and d) the role of limb movements for short- and long-term learning of novel information and vocabulary. The researchers involved in the two teams have multi-disciplinary profiles and complement each other with expertise in language and cognitive sciences, speech production, multi-modal communication, motor control and learning, engineering and signal processing. Different steps are proposed to spread the findings to different scientific and clinical communities and to a broader public. The results of the project will indeed have an impact on fundamental research with a deeper understanding of spoken language in the context of body motions, but will also be useful for applied research in language and speech education and therapy.

Perceiving is essentially a decision process, whereby our brain has to deal with multiple uncertain and often ambiguous elements from the sensory input to make sense of the world. This is particularly evident with multistable, ambiguous visual stimuli, where multiple interpretations (percepts) are possible for a single physical stimulus. In this condition, human observers spontaneously alternate between the possible percepts at unpredictable times. This phenomenon has intrigued brain scientists and philosophers for decades, who have addressed questions like “What pushes our brain to switch to a new percept?”, “What factors, in the observer’s experience, conscious will, or learnt behaviors, can help or interfere with perceptual multistability?”. Another important property of human visual processing is that it is strongly non uniform, meaning that only the central part of the image projected to the eye, reaching the fovea, is processed with high resolution. Therefore, a reliable analysis of the sensory input requires a sequential sampling of information (e.g. collecting different viewpoints and local detailed observations), which is achieved with incessant eye movements: we call this active vision. Vision-3E builds on the assumption that visual perception results from a dynamic functional loop, which consists of three major building blocks : Expectation about the physical world (through our internal beliefs), Exploration of the sensory evidence, and Exploitation (monitoring) of the combined information resulting from expectation and exploration, to build or update a stable percept of the world. Rather than a mere feed-forward construct we assume fully recurrent connections, since for instance percept-related activity projects back onto the early sensory processing levels, leading in turn to a more focused filtering of the sensory evidence. We aim at collecting behavioural and physiological evidence about these three key functions— expectation, exploration, and exploitation, in order to better understand and model their interaction. Importantly we also make the assumption that eye movements participate actively to the functional closed-loop, contributing to the sensory exploration and to the filtering of information leading to stabilise the percept (in the exploitation phase). Hence we will use multiple techniques (visual psychophysics, eye tracking, EEG, fMRI-guided neuro-stimulation) with innovative experimental designs, as well as computational modelling, to achieve a comprehensive understanding of active visual decision-making. We will focus on the particular framework of multistable perception, but we will exploit the consortium synergy to be able to generalize results and interpretations across different visual ambiguous stimuli and different experimental manipulations. The originality of Vision-3E is three-fold: First, as already mentioned, the role of eye movements for perceptual decisions will be thoroughly addressed (including with specific interventional experiments selectively perturbing them). Second, we will integrate a dynamic model of the sequential decisions leading to percept reversals in the standard theoretical stationary framework of multistability. Third, we will extract (offline, then online) oculomotor and EEG markers that will be directly fed into the model, and eventually test its predictive power (offline and online) about forthcoming percept reversals. The online closed-loop stimulation, involving fast computation on the EEG-model-TMS chain is the most ambitious and slightly risky development of the project. Feasibility and success of the workplan are granted by the strong collaborative attitude and complementary expertise of the consortium members, across five labs and three sites, all provided with state of the art equipment.

Remote human interaction and human-machine interaction require reliable speech-processing technologies that can work in unconstrained real-world acoustic conditions. Speech recordings are inevitably contaminated by interfering sound sources and by the presence of reverberation. Whether for human or artificial listening, speech enhancement algorithms are necessary to improve speech quality and intelligibility. The vast majority of current algorithms rely on the use of deep neural networks trained in a supervised manner, using a dataset of noisy speech signals labeled with the corresponding clean-speech reference signals. Given the impossibility of acquiring such data in real conditions, datasets are artificially generated by creating synthetic mixtures of isolated speech and noise signals. However, the performance of supervised algorithms drops drastically when these synthetic data differ from the real conditions of use. The current trend is to create larger and larger synthetic datasets, in the unrealistic hope of covering all possible acoustic conditions. In contrast, the DEGREASE project proposes a weakly-supervised learning framework with the aim of developing more flexible, robust and ecologically-valid algorithms that can be trained on real unlabeled data and that are able to adapt to new acoustic conditions. At the crossroad of audio signal processing, probabilistic graphical modeling, and deep learning, we propose a deep generative learning methodological framework for multi-microphone speech signals, which combined with amortized variational inference techniques will allow models to be trained efficiently in a weakly-supervised manner.

An important task of data science is to represent and evidence the interrelation between coupled observables. The simple case of two observables that vary in time or space leads to bivariate signals. Those appear in virtually all fields of physical sciences, whenever two quantities of interest are related and jointly measured, such as in seismology (e.g. horizontal vs vertical ground motion), optics (transverse coordinates of the electric field), oceanography (components of current velocities), or underwater acoustics (horizontal vs vertical particle displacements) to cite a few. Bivariate signals describe trajectories in a 2D plane whose geometric properties (e.g. directionality) have a natural interpretation in terms of the physical notion of polarization usually used for waves. As an example, according to Einstein’s theory of general relativity, the recently detected gravitational waves (GWs) are characterized by two degrees of freedom, that are connected to the bivariate space-time strain signal measured by the detectors. The polarization state of the observed signal is directly connected to that of the wave, which in turn provides key insights into the underlying physics of the source. Polarization is thus a central concept for the analysis of bivariate signals. Moreover, recent years have seen an increased interest in exploiting bivariate signals in many applications. For instance, in underwater acoustics, the advent of a new modality called IVAR capable of measuring particle velocity has opened promising avenues of research. Indeed, unlike conventional single-channel (e.g. pressure) sensors, these new vector sensors permit to probe the “geometry” of the underlying propagation medium. Thus, being able to fully take advantage of the information gathered in bivariate signals – notably, its polarization properties – is a crucial step towards understanding complex physical phenomena. Unlocking these essential physical insights is bound to the development of new polarization-aware methodologies in bivariate signal processing. Including polarization information in the analysis and processing workflow is of general interest and can impact all elementary tasks in bivariate signal processing, namely analysis, modeling, filtering, detection, and statistical inference. The RICOCHET project aims at establishing a complete set of theoretical and methodological tools to fully exploit the polarization information of bivariate signals. Their relevance will be extensively demonstrated on important practical problems arising in physical applications, notably gravitational-wave astronomy, underwater acoustics and seismology. .

MicroVoice’s objective is to gain an in-depth understanding of the link between the micromechanics of vocal-fold tissues and their unique vibratory performances, and take the next step towards the development of new biomimetic oscillators. The strategy is: i. to investigate the vocal-fold 3D fibrous architecture and micromechanics using unprecedented synchrotron X-ray in situ microtomography; ii. to use these data to mimic and process fibrous biomaterials with tailored structural and biomechanical properties; iii. to characterise the vibro-mechanical properties of these biomaterials at different scales (macro/micro) and frequencies (low/high), using Dynamic Mechanical Analysis and Laser Doppler Vibrometry. iv. to validate their oscillating properties under “realistic” aero-acoustical conditions using in vitro and ex vivo testbeds. MicroVoice will provide a solid framework for the innovative design of fibrous phonatory implants.