People lived in the Blaise Estate area in the Bronze Age, Iron Age and Roman era. There was probably a Medieval chapel dedicated to Saint Blaise, the patron saint of wool combers, on the hill.

A manor house known as 'The Great House' was built at Blaise in the mid 1600s. It had formal gardens, laid out in a Tudor style.

Thomas Farr, who owned the estate in 1766, had a sham castle built on the hill. Farr had extensive investments in the slave trade. He used the building as a summerhouse to entertain guests.

In the 1700s another manor house stood where Blaise dairy is now. Blaise Castle House was built in 1796-8 as a home for the Harford Family, and the other house was demolished. The last of the Harfords to own Blaise died in 1919.

Blaise Estate was then sold to the Bristol Corporation, later known as Bristol City Council. It became a public park. In 1949 the house became a branch of Bristol Museum and Art Gallery, initially as a folk museum, now displaying art, archaeology and social history.