We propose the parallel multi-class support vector machines (Para-SVM) algorithm to efficiently perform the classification task of the ImageNet challenging problem with very large number of images and a thousand classes. Our Para-SVM learns in the parallel way to create ensemble binary SVM classifiers used in the One-Versus-All multi-class strategy. The stochastic gradient descent (SGD) algorithm rapidly trains the binary SVM classifier from mini-batches being created by under-sampling training dataset. The numerical test results on ImageNet challenging dataset show that the Para-SVM algorithm is faster and more accurate than the state-of-the-art SVM algorithms. Our Para-SVM achieves an accuracy of 74.89% obtained in the classification of ImageNet-1000 dataset having 1,261,405 images in 2048 deep features into 1,000 classes in 53.29 min using a PC Intel(R) Core i7-4790 CPU, 3.6 GHz, 4 cores.