Abstract Background The incidence of hepatocellular carcinoma (HCC) has been increasing among the elderly populations. Trans-arterial chemoembolization (TACE), a widely used first-line non-curative therapy for HCCs is an issue in geriatrics. We investigated the prognosis of elderly HCC patients treated with TACE and determined the factors that affect the overall survival. Methods We included 266 patients who were older than 65 years and had received TACE as initial treatment for HCC. We analyzed the skeletal muscle index (SMI) and visceral-to-subcutaneous fat ratio (VSR) around the third lumbar vertebrae using computed tomography scans. Muscle depletion with visceral adiposity (MDVA) was defined by falling below the median SMI and above the median VSR value sex-specifically. We evaluated the overall survival in association with MDVA and other clinical factors. Results The mean age was 69.9 ± 4.5 years, and 70.3% of the patients were men. According to the Barcelona Clinic Liver Cancer (BCLC) staging system, 29, 136, and 101 patients were classified as BCLC 0, A, and B stages, respectively, and 79 (29.7%) had MDVA. During the median follow-up of 4.1 years, patients with MDVA had a shorter life expectancy than those without MDVA (P = 0.007) even though MDVA group had a higher objective response rate after the first TACE (82.3% vs. 75.9%, P = 0.035). Multivariate analysis revealed that MDVA (Hazard ratio [HR] 1.515) age (HR 1.057), liver function (HR 1.078), tumor size (HR 1.083), serum albumin level (HR 0.523), platelet count (HR 0.996), tumor stage (stage A, HR 1.711; stage B, HR 2.003), and treatment response after the first TACE treatment (HR 0.680) were associated with overall survival. Conclusions MDVA is a critical prognostic factor for predicting survival in the elderly patients with HCC who have undergone TACE.