He ordered that castles be built, including one at Dunheved in Launceston and at another at Trematon. By the "motte and bailey" castle had been constructed, and control of the manor transferred to it. The earldom later became extinct but all its possessions passed to the Duchy of Cornwall which King Edward III created in for his eldest son Edward, later known as the 'Black Prince'. The Duchy still owns the castle today.
According to a survey made in the bailey contained a great hall, a kitchen, a lodging chamber and a Lady Chapel. There was a deer park adjacent to the castle. By the time of the Black Prince's death in , the military significance of Trematon Castle was decreasing. However, in King Richard II had the castle repaired and garrisoned when there was a threat of invasion during the war with France.
By the middle of the 17th century the castle had become ruinous, occasionally offenders were tried and if found guilty, imprisoned there. It continued to be used as a prison until In the 18th century the castle's interior was used as a market garden, tended by a poor family living in the gatehouse. Vincent and later to become Surveyor-General to the Duchy of Cornwall, took a 90 year lease of the property from the Duchy and set about transforming the place.
He cleared away any ruins that remained within the bailey and knocked down a length of bailey wall to provide a view of the estuaries. In Tucker commenced building a 9-bedroomed house in the bailey using some of the stone from the demolished walls. Tucker built two lodges and constructed a remarkable length of drive that zig-zagged up the hill from the lower lodge now gone at Forder.
He landscaped the grounds, laid out gardens and in them built mock ruins into which he incorporated several carved stone doorways derived from the clearance work in the bailey. He built an orangery, glass-houses, stables and other outbuildings, and he created a huge walled garden on the northern hill-slope.
