Distributed Leger Technologies (DLTs), most notably Blockchain technologies, bring decentralised platforms that eliminate a single trusted third party and avoid the notorious single point of failure vulnerability. Since Nakamoto's Bitcoin cryptocurrency system, an enormous number of decentralised applications have been proposed on top of these technologies, aiming at more transparency and trustworthiness than their traditional counterparts. These applications spread over a lot of areas, e.g. financial services, healthcare, transportation, supply chain management, and cloud computing. While Blockchain brings transparency and decentralised trust intuitively due to the consensus of a (very large) group of nodes (or, miners), it introduces very subtle implications for other desirable properties such as privacy. In this work, we demonstrate these subtle implications for Blockchain-based searchable encryption solutions, which are one specific use case of cloud computing services. These solutions rely on Blockchain to achieve both the standard privacy property and the new fairness property, which requires that search operations are carried out faithfully and are rewarded accordingly. We show that directly replacing the server in an existing searchable encryption solution with a Blockchain will cause undesirable operational cost, privacy loss, and security vulnerabilities. The analysis results indicate that a dedicated server is still needed to achieve the desired privacy guarantee. To this end, we propose two frameworks which can be instantiated based on most existing searchable encryption schemes. Through analysing these two frameworks, we affirmatively show that a carefully engineered Blockchain-based solution can achieve the desired fairness property while preserving the privacy guarantee of the original searchable encryption scheme simultaneously.