According to the first written records, Polhov Gradec Castle was built in 1315, though this date cannot be proved. It was originally made of wood, and the first stone works probably date back to somewhere between the 14th and 15th centuries. The height of the original castle presumably reached at least the height of the present day first floor, whilst the second floor was made of wood. Apart from the exterior appearance, the interior wooden ceiling and the stone portals are worth special mention.
The castle's appearance has been preserved in the copper engravings produced by Anton Trost and published in Valvasor's Contemporary Topography of the Dutchy of Carniola (1679) and the Glory of the Dutchy of Carniola (1689).
The size and volume of the castle building during the Renaissance, was defined under the proprietorship of the count Jurij Khisl, the offspring of the glassworks owner at Fužine in Ljubljana. At this time, all the castle’s windows were glazed and blown glass panes found during the archaeological excavations support this fact. By the end of the 17th century the castle facade had now become symmetrical, and the main entrance lost its Renaissance pediment, which was still identifiable on the copper engraving. The eastern loggia was also paved with rounded river cobbles. According to contemporary sources, Mark Anton Kunstel Baumgarten, who in 1658 became the owner, renovated the castle, and during that time had the chapel built, added the belvedere with a clock and placed Neptune's fountain in the yard.
In the late 18th and early 19th century, the Northwest wing was added to the castle. This was also the period when the present day main entrance was constructed and all windows on the front facade gained uniform stone mullions.
Following the Bilichgraetz family line, in the first half of the 19th century the estate transferred to the ownership of Ursini Blagaj, of the Carniolan nobility. In 1808 the count Rihard Ursini Blagaj (1786-1858) married the seventeen-year old baroness, Antonija Polhograjska (Antonia of Polhov Gradec), the castle's co-owner, and moved into Polhov Gradec. In his time, the castle became the meeting point of followers of the Enlightenment. Count Blagaj, who managed the estate for fifty years, supported and promoted the development of agriculture and cottage industry. He was an advocator and patron of the arts and contributed to the general development of the locality. The Count was also the first Mayor of Polhov Gradec. He was also very much involved in mineralogy and botany. In 1837, a local farmer brought him an unknown plant, which he immediately sent to his colleague Henrik Freyer, a natural historian and botanist to the then Estate Museum of Carniola in Ljubljana. Freyer later named this newly discovered plant Blagajev Volčin. (Daphne blagayana or ‘Blagay’s daphne’.)
In 1875 the estate was purchased by Luiza Urbančič from Preddvor, and the last owner was Ana Delago. After World War Two, the castle was nationalised and plundered. There then followed a period of housing an elementary school (until 1969), after which, the castle was left to decay until the 1990s, when it was restored and renovated by Ljubljana Regional Institute for the Protection of Cultural Heritage. In 1999 it was, together with the park and Neptune's fountain, declared a cultural monument of national significance, and in 2008 the Ministry of Culture passed it over to the management of the Technical Museum of Slovenia.