You don't have to. Like you're perfectly allowed to just stay in the city proper and not deal with that. But like, there's quests and stuff that pay really well if you do want to go out and fight them.

[The catch is that money is a nice thing to have.]