I do, and where I live being the first to throw a punch towards anyone for almost any reason is generally frowned upon.

The reason that violence is dangerous in this context is that it can allow a violent minority to oppress and subjugate a majority. By removing it from society in general and de-legitimizing its use the influence of these sorts of people can be effectively minimized.