Mitochondrial DNA (mtDNA) has been implicated in various human degenerative diseases. However, the role of mtDNA in Osteoarthritis (OA) is less known. To investigate whether mtDNA haplogroups contribute to the prevalence of knee OA, we have carried out a comprehensive case-control study on 187 knee OA patients and 420 geographically matched controls in southern China. OA patients were classified on the Kellgren/Lawrence scale from two to four for the disease severity study and the data were analyzed by adjusting for age and sex. We found that patients with haplogroup G (OR = 3.834; 95% CI 1.139, 12.908; p = 0.03) and T16362C (OR = 1.715; 95% CI 1.174, 2.506; p = 0.005) exhibited an increased risk of OA occurrence. Furthermore, patients carrying haplogroup G had a higher severity progression of knee OA (OR = 10.870; 95% CI 1.307, 90.909; p = 0.007). On the other hand, people with haplogroup B/B4 (OR = 0.503; 95% CI 0.283, 0.893; p = 0.019)/(OR = 0.483; 95% CI 0.245, 0.954; p = 0.036) were less susceptible for OA occurrence. Interestingly, we found OA patients also exhibited a general increase in mtDNA content. Our study indicates that the mtDNA haplogroup plays a role in modulating OA development.