Where the Doctor leads, others follow, usually humans. They can’t help themselves, because the Doctor isn’t like anyone they’ve ever met. He’s mysterious, and exciting, and probably dangerous, but he offers them an escape from their humdrum lives, a chance to see things very few people ever have. Who wouldn’t want that?
Some opportunities come along only once in a lifetime, if that, and saying no would lead to endless regrets and thoughts of what might have been. So when he asks, “Come with me?” what can they do but let him lead them into a universe of unimaginable adventure?