The generally low energy density from most heat sources—the Sun, Earth as well as most human activities—implies that solid-state thermoelectric devices are the most versatile heat harvesters since, unlike steam engines, they can be used on a small scale and at small temperature differences. In this opinion piece, we first discuss the materials requirements for the widespread use of thermoelectrics. We argue that carbon-based materials, such as conducting polymers and carbon nanotubes, are particularly suited for large area and low-temperature operation applications, as they are abundant, low-toxicity and easy to process. We combine experimentally observed macro-trends and basic thermoelectric relations to evaluate the major performance limitations of this technology thus far and propose a number of avenues to take the thermoelectric efficiency of organic materials beyond the state of the art. First, we emphasize how charge carrier mobility, rather than charge density, is currently limiting performance, and discuss how to improve mobility by exploiting anisotropy, high persistence length materials and composites with long and well-dispersed carbon nanotubes. We also show that reducing thermal conductivity could double efficiency while reducing doping requirements. Finally, we discuss several ways in which composites could further boost performance, introducing the concept of interface engineering to produce phonon stack-electron tunnel composites. This article is part of a discussion meeting issue ‘Energy materials for a low carbon future'.