To describe the differences in knee structure and non-knee structural factors between offspring having at least one parent with a total knee replacement for severe primary knee osteoarthritis and age- and sex-matched controls with no family history of knee osteoarthritis, a population-based longitudinal study of 163 matched pairs (mean age 45 years, range 26 to 61) was performed at baseline and about 2 years later. Knee cartilage defect score (0 to 4), cartilage volume and bone size were determined with T1-weighted fat saturation magnetic resonance imaging. Body mass index (BMI), lower-limb muscle strength, knee pain, physical work capacity at 170 beats/minute (PWC170) and radiographic osteoarthritis were measured by standard protocols. In comparison with controls, offspring had higher annual knee cartilage loss (-3.1% versus -2.0% at medial tibial site, -1.9% versus -1.1% at lateral tibial site and -4.7% versus -3.7% at patellar site, all P < 0.05), a greater increase in medial cartilage defect score (+0.15 versus -0.01, P < 0.05) and a greater decline in PWC170 (-0.7 watts/kg versus -0.4 watts/kg, P < 0.01). There were no significant differences in change in BMI, lower-limb muscle strength, knee pain or tibial bone area between these two groups; however, the differences in knee cartilage loss and cartilage defect change decreased in magnitude and became non-significant after adjustment for baseline cartilage volume, tibial bone area, BMI and knee pain. This longitudinal study suggests that knee cartilage loss, change in cartilage defects and decrease in physical fitness all have roles in the development of knee osteoarthritis, which is most probably polygenic but may reflect a shared environment. Importantly, the cartilage changes are largely dependent on baseline differences in cartilage volume, tibial bone area, BMI and knee pain, suggesting that these factors might have a role in their initiation.