If one spends any significant amount of time in left-libertarian circles, one will hear of the concept of “thick libertarianism”. Thick libertarianism is basically extending the advocacy of freedom to the cultural sphere, creating a cultural background in which liberty can more fully flourish. This is controversial because it involves extending liberty beyond simple non-aggression.
However, the fault of “aggression is the root of the problem” is that generally, aggression is an expression of authority. The state gains its power from the fact that people obey its authority. Another example would be rape, which is fundamentally an assertion of authority over a victim. Aggression is far more likely to occur because of authority then out of opposition to it.
You can have authority without exercising force, but it’s pretty damn hard to have force without exercising authority.