If any of the provinces of the British empire cannot be made to contribute towards the support of the whole empire, it is surely time that Great Britain should free herself from the expence of defending those provinces in time of war, and of supporting any part of their civil or military establishments in time of peace, and endeavor to accommodate her future views and designs to the real mediocrity of her circumstances. (5.3.15)
By the end of this book, Smith is so exasperated with the senselessness of British economic policy that he recommends for the country to jettison all of its colonial territories. In his mind, having these colonies is like having a lot of jewels. They're really expensive and they don't serve any purpose other than showing people how awesome you are.
Smith is worried that Britain will bankrupt itself if it insists on keeping its empire going despite the economic strain it creates. In the end, he can only plead for people to be smarter and to run their economy more efficiently by having free trade and letting these colonies become independent territories.