Privilege just means private law.
It was because he wanted there to be conspirators. It was much better to imagine men in some smoky room somewhere, made mad an cynical by privilege and power, plotting over the brandy. You had to cling to this sort of image, because if you didn’t then you might have to face the fact, that bad things happen because ordinary people, the kind who brushed the dog and told their children bedtime stories, were capable of then going out and doing horrible things to other ordinary people.
Terry Pratchett, Jingo, Seite 177, 1997