The first known construction measures for turning the imperial burgward, which was mentioned in the 11th century, into an imperial castle began in the 12th century. In 1404, Colditz Castle and Colditz Territory were sold to Margrave William I of Meissen by the Colditz family. From 1464 onwards, the imperial castle was rebuilt under the Saxon electors into a representative late-Gothic castle. Repeatedly, large parts of the castle were destroyed by fire, notably in 1504.
Between 1519 and 1525 Elector Frederick the Wise had the complex converted into one of the largest Renaissance style castles of Middle Germany. After further reconstructions under the electors Augustus I and Christian I the castle served as official residence at the end of the 16th century and also as occasional residence for the electors.
From the beginning of the 19th century until 1933 Colditz Castle was an asylum for the poor and insane as well as a hospital. During World War II it was used as prisoner-of-war camp for Western allied officers, afterwards again as hospital.