Chinese Sign Language (CSL) and Chinese are languages used in the Chinese mainland. As a dominant language, Chinese has great influence on all levels of CSL. CSL, as a visual sign language, is fundamentally different from Chinese in linguistic structure. Unlike English, Chinese, as a pictograph, has influence on Chinese and CSL. This study explains in detail the influence of Chinese characters on CSL at the lexical level, including many elements from Chinese, such as “仿字 fangzi” (form imitating Chinese characters), “书空 shukong” (writing in the air with the index finger), loan translation, finger spelling, and mouthing patterns. This influence is not a simple borrowing of Chinese characters, but a creative imitation and adaptation according to the needs of sign language to express meaning. After a long period of evolution, the characteristics of Chinese characters are naturally integrated into CSL loanwords, which makes the relationship between sign language and Chinese characters closer. CSL borrows a large number of Chinese words, most of which are signs to express non-core concepts. These borrowed signs are an indispensable part of the CSL sign language family, enrich sign language vocabulary, improve the accuracy of sign language expression, and play a positive role in promoting the learning, work, and lives of deaf people.