Currently, most tactile-based object recognition algorithms focus on single shape or texture recognition. However, these single attribute-based recognition methods perform poorly when dealing with objects with similar shape or texture characteristics. Research on integrating shape and texture attributes is still limited, and existing feature fusion mechanisms tend to rely on simple connectivity while ignoring the interactions between different features. To address this issue, we propose a novel attention-based fusion network, TSMFormer, which classifies by integrating shape and texture information and harnesses the global learning capabilities of attention mechanisms to explore interactions between shape and texture in tactile images. Considering the advantages of Transformer networks in handling large datasets, we expanded the existing tactile image dataset through data augmentation. Extensive comparative experiments on this dataset show that the accuracy of the network combining texture and shape information is significantly improved to 99.3%. Comparisons with existing fusion methods further validate the effectiveness of our proposed attention fusion mechanism. The results demonstrate that TSMFormer is highly valuable for research, as it fuses texture and shape information in tactile images through an attention mechanism. Additionally, it shows great potential for practical applications such as robot grasping and automatic quality inspection in industrial environments.