One of the businesses that Edward Colston inherited from his father in Bristol was a share in the Norton House or St Peter's sugar house. Robert Aldworth, to whom Colston's father was apprenticed, had established the first sugar refinery in Bristol in 1612. Aldworth used sugars from Spain and Portugal, but later records suggest the partners in the business were buying sugar from St Kitts in the Caribbean. Wherever the sugar came from, it was probably produced by enslaved labour. This was continuing a family business: his father William and brother Thomas had imported sugars for 20 years from 1661.

This watercolour by Mary Katharine Moore shows St Peter's Hospital, as it was then, in 1894. The older wing of the building, on the left, had been the one-pan sugar house, with just one pan for boiling the sugar to produce refined white sugar from the raw brown sugar imported. As sugar imports increased over the years, the sugar houses grew in size with two or three pans to cope with the demand.