We propose the bagging support vector machines using stochastic gradient descent (Bag-SVM-SGD) for effectively classifying large-scale multi-class datasets. The Bag-SVM-SGD learns in the parallel way from under-sampling training dataset to create ensemble binary SVM-SGD classifiers used in the One-Versus-All (OVA) multi-class strategy for performing text classfication tasks with million of datapoints in thousands of classes. The numerical test results on two large scale multi-class datasets (LSHTC4, Book) show that our Bag-SVM-SGD algorithm is faster and more accurate than the state-of-the-art linear algorithm LIBLINEAR. An example of its effectiveness is given with an accuracy of 62.41% obtained in the classification of LSHTC4 dataset having 728,067 datapoints in 1,617,900 dimensions into 2,713 classes in 104.15 minutes using a PC Intel(R) Core i7-4790 CPU, 3.6 GHz, 4 cores.