Well if you look at it objectively: There is a small group of survivors and a much bigger number of zombies. It would be suicidal to engage them in battle.
However it's not like they would manage to sneak around the zombies without being noticed everytime
