That's the problem with gardens. They're under constant attack from weeds and pests. Relax your continuous vigilance and care and the entire thing is liable to be overrun.
And here's a tip. When a weed gets started, you need to pull it out immediately. Many of them put down deep tap roots that make removal vastly more difficult once they're established. And if you don't remove the entire weed, it can grow again from just a piece of broken root still buried deep.
Yank out and eradicate. Never rest.