Computer-aided diagnosis (CAD) systems based on deep learning approaches are now feasible due to the availability of big data and the availability of powerful computational resources.The medical image-based CAD systems are of great interest in numerous diseases, but especially for skin cancer diagnosis, deep learning models have been mostly developed for dermoscopy images. Models for clinical images are few, mainly due to the unavailability of big volumes of relevant data. However, CAD systems able to classify skin lesions from clinical images would be of great valueboth for the population and clinicians as an initial early screening of lesions that would leadpatients to visiting a dermatologist in case of suspicious lesions. This is even more pronounced in areas where there is lack of dermoscopy instruments. Thus, in this paper, we aimed to build a classifier based on bothdermoscopy and clinical images able to discriminate skin cancer from skin lesions. The classification is made among three benign and two malignant categories, which include Nevus, Benign but not nevus, Benign but suspicious for malignancy, Melanoma and Non-Melanocytic Carcinoma.The proposed deep learning classifier achieves an Area Under Curve ranging between 0.75 and 0.9 for the five examined categories.