During the last decade, a new kind of computation-intensive and complex application has emerged: Large Language Models (LLM). These applications require the computing power offered by HPC clusters and are complex to implement efficiently: several kinds of parallelisms are possible, limited available memory is often a major constraint, accelerators (GPUs, TPUs, NPUs, ...) need to schedule data transfers and computations, the problem size imposes distributed executions, ... All these challenges are already well-known by developers of the first-class citizen applications running on HPC clusters: linear algebra, numerical simulation, ... That is why task-based runtime systems have been proposed: to ease the writing of HPC applications by providing an abstraction of the machine and its efficient programming. Despite task-based runtime systems being used for a long time now for classic HPC applications, they are not used to implement LLM applications. In this paper, we present our first experiments to try to understand why this is the case and whether using task-based runtime systems for LLM applications (both training and inference) is relevant. We describe our implementation of a small LLM with StarPU, discuss the different choices we had to make and evaluate performance.