The Doctor finds aliens everywhere he goes, but in all the time he spent on earth, he never knew the Silence were there. Or maybe he did, maybe he found them over and over again, but just like everybody else on the planet, he always forgot them the moment they were out of sight.
How do you fight something you can’t remember, something that can make you forget it so completely you don’t even question your own lack of memories?
The Doctor found a way. The people of earth will never remember the Silence, but they’ll kill them on sight.