People were in the Blaise Estate area in the Bronze Age, Iron Age and Roman era, and it may have been occupied before that but there's no firm evidence.

The hill is thought to be the site of a Medieval chapel dedicated to Saint Blaise, the patron saint of wool combers. A sham castle was built on the hill for Thomas Farr in 1766, who owned the estate at the time. He used it as a summerhouse to entertain guests.

A manor house known as 'The Great House' was built at Blaise in the mid 1600s. It had formal gardens, laid out in a Tudor style. A statue of Neptune from these gardens was recently re-discovered.

In the 1700s another manor house stood where Blaise dairy is now. It was demolished after Blaise Castle House was built in 1796-8 as a home for the Harford Family. The last of the Harfords to live at Blaise died in 1919.

Blaise was sold to the Bristol Corporation, the forerunner of Bristol City Council. The house was unoccupied and the estate became a public park. In 1949 the house became a branch of Bristol Museums, Galleries & Archives, initially as a folk museum, now displaying social history.