Castle was artist and pastoralist, son of Thomas Castle, of Clifton, Bristol, Gloucestershire. He came to South Australia in 1839 on board the Moffatt and took up land. He later became a magistrate for Hackham. On 13 December 1846 he married Kate Borrow. They had eleven children, five girls and six boys. He died on 29 November 1877 at Strathalbyn, South Australia.