Free space optical communication is becoming an increasingly important technology for the wireless market since it introduces a higher data transmission rate and large bandwidth. In this work, we have compared DPSK and OQPSK systems. Several measurements were performed including Bit Error Rate (BER) curves for optimum reception, eye diagram with the variation of input signal and distance of medium for each of the systems. Recent digital fiber optic communication systems address modulation and detection techniques for high spectral efficiency and robustness against transmission impairments. The proposed objective of this project is to design studies and analyze the simulation model of a Digital fiber Communication System using Opti system 7.0. The main objective of this work is to analyze different approaches to mitigate the FSO channel impairments influencing the transmission limitations in Optical Communication. Two modulation formats viz., Differential Phase Shift Keying (DPSK) and Offset Quadrature Phase Shift Keying (OQPSK) have been considered in the present analysis focusing on theoretical considerations, mathematical modelling and computer simulations. The major focus is to analyze, evaluate and characterize different types of optimal encoding schemes to infer the system design parameters in order to achieve an optimum performance.