People become busier at work in modern life, fast food is usually a choice. However, this daily habit can cause several problems such as overweight and obesity diseases. In this work, we propose a calorie determination system using deep learning approach, especially for dieters and athletes. This work firstly focuses on fruit calorie determination. First, we collect the dataset of popular fruits in Vietnam and other public sources, then we propose using deep learning with MobilenetV2 architecture for fruit classification. An application is built for automatically determining the calorie of those fruits from photo taken by smartphone. The method can achieve more than 90% accuracy and the application can work on both Android and IOS devices.