Instead of spoken language, sign language is the commonly used language by the deaf. To express their thoughts, the deaf people combine handshapes, movements, directions of hands, arms and body, and facial expressions. Currently, there are several methods of hand sign language recognition, such as image recognition or sensor recognition. In this study, we tested the recognition of characters of hand sign language using VGG-16 network architecture to investigate the change in recognition efficiency when adding Dense layers with neural parameters. A test was conducted on a gesture dataset consisting of 26 English letters with 5,391 images, and there are 200 images for each subclass. By investigating the change in recognition efficiency through the change of the VGG-16 model, we have achieved a high recognition rate with an accuracy of 99.902% on the training set and 99.910% on the test set compared to the original model. In addition, the change in the distance when detecting hand gestures also affects the system’s accuracy.