Blaise Estate was bought by John Scandrett Harford in 1789. He was a wealthy bank director and brass merchant from Bristol.

The manor house stood roughly where Blaise dairy is now. Harford wanted a new house though so he hired an architect to design what became Blaise Castle House. It was built 1796-8, the family moved in, and the old house was demolished.

Harford died in 1815 and was succeeded by his eldest surviving son, John Scandrett Harford, junior. He added a picture room to the house, classical friezes and statues, and a terrace. His close friends included William Wilberforce and Hannah More.

After he died in 1866, the estate passed to his nephew, John Battersby Harford. He had another home in Wales but when he died in 1875, his wife Mary moved to Blaise and lived there with her three daughters until she died in 1919. The daughters went to live in Bath and in 1926 Blaise Castle Estate was sold to Bristol Corporation (Bristol City Council).