Convolutional Neural Networks (CNNs) are highly effective for computer vision and pattern recognition tasks; however, their computational intensity and reliance on hardware such as FPGAs pose challenges for deployment on low-power edge devices. In this work, we present COMET, a framework of CNN designs that employ efficient hardware offset-binary coding (OBC) techniques to enable co-optimization of performance and resource utilization. The approach formulates CNN inference with OBC representations of inputs (Scheme A) and weights (Scheme B) separately, enabling exploitation of bit-width asymmetry. The shift-accumulate operation is modified by incorporating the offset term with the pre-scaled bias. Leveraging inherent symmetries in Schemes A and B, we introduce four novel look-up table (LUT) techniques -- parallel, shared, split, and hybrid -- and analyze them to identify the most efficient options. Building on this foundation, we develop an OBC-based general matrix multiplication core using the im2col transformation, enabling efficient acceleration of a fixed-point modified LeNet-5 model. FPGA evaluations demonstrate that the proposed co-optimization approach significantly reduces resource utilization compared to state-of-the-art LeNet-5 based CNN designs, with minimal impact on accuracy.