A natural consequence of the pay-per-use business model of Cloud Computing, is that Cloud Users need to evaluate and compare different cloud providers in order to choose the best offerings in terms of trade-off among performance and prices. But this is, at the state of art, one of the main lack in the cloud environment: no real grants are offered by providers about the quality of the resources offered and no clear ways exists to compare two different offerings. Moreover the high dynamicity of cloud resources (virtual machines can be added or removed in few minutes) make evaluation of such systems really hard to be done. In this paper we propose a framework for building ad-hoc benchmark applications, whose behaviour is strictly related with user needs and which can be used to compare different provider. The framework is based on the mOSAIC project, which offers a deployable platform and an API for building provider-independent applications. Due to such independence we are able to directly compare different cloud offers. The paper proposes a detailed analysis of the state of art and of the problem of benchmarking in cloud environment, before detailing the proposed architecture and illustrating it briefly with an example.