Blaise Castle Estate was owned by Thomas Farr, 1762-1778, a merchant who owned American sugar plantations which relied on slave labour. He grew wealthy from his investments in the Transatlantic traffic in enslaved Africans. With his wealth, he created pleasure gardens at Blaise, including the castle.

In 1778, Denham Skeate bought Blaise, and then in 1789 it was bought by John Scandrett Harford, a banker and merchant whose many business interests included trading with Virginia, for example in glass.The Harford family also part owned a brass battery company, producing hammered brassware items for the domestic market as well for export to the west coast of Africa. It’s likely that they also produced manillas, a currency associated with the Transatlantic traffic in enslaved Africans. Harford was a member of the Bristol Merchant Venturers' Guild, who supported the slave trade.

His son, John Scandrett Harford Junior, and his wife Louisa, took over the estate in 1815. They were close friends with slavery abolitionists Hannah More and William Wilberforce, who they regularly met with at Blaise. Hannah More and William Wilberforce both died in 1833, just weeks after slavery was abolished in the British Empire.