Character Analysis
There are many things that Adam Smith is not a fan of, but one of the big ones is people who take away the natural freedom of others.
That's one of the big reasons he doesn't like Britain's colonial domination over other countries. But he also hates this domination because it pointlessly drains a lot of money from Britain's economy. As he writes:
The maintenance of this monopoly [over the colonies] has hitherto been the principal, or more properly perhaps the sole end and purpose of the dominion which Great Britain assumes over her colonies. (4.7.43)
In other words, Britain keeps its colonies thinking that the situation is making them richer. But all evidence shows that it's doing just the opposite.
Adam Smith has no sympathy for people who continue doing something dumb and immoral just because it makes them feel big. And for that reason he thinks that:
Under the present system of management, therefore, Great Britain deserves nothing but loss from the dominion which she assumes over her colonies. (4.7.44)
He thinks that having colonies is idiotic on every level, but he's also pessimistic about the chances of Britain willingly giving up its colonies, saying:
To propose that Great Britain should voluntarily give up all authority over her colonies, and leave them to elect their own magistrates […] would be to propose such a measure as never was, and never will be adopted, by any nation in the world. (4.7.45)
Unfortunately, Smith would be right for the next couple hundred years and Britain would only get more aggressive in its efforts at colonization.