The advent of foundation models such as large language models (LLMs) has led to impressive advancements in all areas of natural language processing. Exploiting their capabilities for many downstream applications has therefore emerged as an interesting research direction, as well as extending them to other modalities. Toward this aim, researchers have started investigating the combination of speech encoders and LLMs. Despite the potential of this emerging approach, systematic studies are still required to isolate its strengths and weaknesses compared to traditional systems, as well as to identify the most effective strategies for specific applications, spanning from architectural choices to selecting the optimal data and tasks for training the adapter between the speech encoder and the LLM. In this scenario, the PhD candidate will study existing solutions, identify their weaknesses, and develop innovative approaches to overcome them, contributing to the advancement of the field, either on a theoretical basis, with an application-oriented focus, or both.
