Data center storage systems of the future in Petabyte and Exabyte scale require very high performance (sub millisecond latencies) and large capacities (100s of Petabytes). The evolution both in scale (or capacity) and performance (throughput and I/O Per Second) is driven by the ever increasing I/O demands from the current and Internet scale applications. These large scale storage systems are basically distributed systems having two primary components or clusters. The first component is the storage server cluster which handles the primary I/O or data I/O for the applications. The second component is the meta-data server (MDS) cluster which manages a single global namespace and serves the meta-data I/O. In this thesis, we look in to the problem of performance deficiencies and scalability of these two components in a multi tenanted mixed I/O (sequential and random I/O) workload environment. To overcome the limitations of the conventional storage system architecture, the thesis proposes a 3-tier hybrid architecture utilizing next generation Non-volatile memory (NVM) like Phase change memory (PCM), Hybrid drives and conventional drives. NVM is used to absorb the writes to the NAND Flash based SSD. This improves both the performance and lifetime of the SSD. Hybrid drives are used as a low cost alternative to high speed Serial attached SCSI (Small computer system interface) or SAS drives for higher performance. This is achieved through a light-weight caching algorithm on the Flash inside the drive. On the storage server, we consider the problem of cache partitioning of next generation NVM, data migration optimization with placement across tiers of storage, data placement optimization of Hybrid drive’s internal cache and workload interference among multiple applications. On the Meta-data server, we consider the problem of load balancing and distribution of file system meta-data across meta-data server cluster that preserves namespace locality. The following are the major contributions of this thesis to address the primary I/O and meta-data I/O performance scalability in large scale storage systems. A heuristic caching mechanism that adapts to I/O workload was developed for a hybrid device consisting of next generation NVM (like Phase change memory) and SSD. This method called HCache can achieve up to 46% improvement in I/O latencies compared to popular control theory based algorithms available in the literature. A distributed caching mechanism called VirtCache was developed that can reduce I/O interference among workloads sharing the storage system. VirtCache can reduce the 90th percentile latency variation of the application by 50% to 83% under a virtualized shared storage environment compared to state-of-art. An Optimized migration and placement of data objects across multiple storage tiers was developed that can achieve up to 17% improvement in performance compared to conventional data migration techniques. We propose new data placement and eviction algorithms on the Hybrid drive internal cache based on the I/O workload characteristics. It reduces the I/O monitoring meta-data overhead by up to 64% compared to state-of-art methods. The algorithm can also classify hot/cold data 48% times faster compared to existing methods. While these solutions address the performance scalability on the storage server, for the meta-data server scalability we developed the DROP meta-data distribution. The DROP mechanism based on consistent hashing preserves locality and near uniform distribution for load balancing. The hashing and distribution mechanism can achieve up to 40% improvement in namespace locality compared to traditional methods. DOCTOR OF PHILOSOPHY (SCE)