He hadn't expected Jo to be there.
But she is and so is Sarah and Rani and ooh, dear, Clyde's in a lot of trouble now, isn't he? But that's not the point — Clyde'll be fine — the point is that Jo is here and he hasn't seen her in, gosh, it's been a long time hasn't it? And Sarah's here as well — it's almost a UNIT reunion and suddenly he can feel his hair grow white and his tweed transform into velvet and the Brigadier is standing behind him telling someone to fire at some unseen menace, five rounds rapid.
But no, those memories will lead to other memories, and those other memories lead to Katarina and Adric and Romana and Donna and the Time War and no, no, no. He's got to stop now. Stop it, stop it, stop it. Focus on something you're used to — Sarah, that's the ticket. She's talking, unsure that it's him. Better do something about that, something she'll recognize.
"Hello, Sarah Jane."
"What Doctor? THE Doctor? MY Doctor?" Jo is the same, exactly the same, and it makes him jealous. Why can she go gallivanting around the world and be the same Jo Grant, bubbly and excited and always, always curious? When he goes gallivanting around the universe all he gets is pain and suffering and an ever-growing feeling that perhaps it isn't worth it.
And now the memories are coming back (stop it, stop it, stop it) and he can't stop them. He is drinking champagne, he is giving her the metebelus crystal, he is leaving the party without saying goodbye, he is driving across the moors, he is promising her she will see him again.
"–but into a baby's?"
He doesn't have time to think. What do you say to the woman who hasn't seen you in decades you haven't seen in centuries? So he comes up with a spur of the moment quip, something that perhaps is more hurtful than kind — and oh dear, that wasn't the right thing to say, why can't he think? Why can't he be the man he was — for her, just for her.
He can't go back. He can't. Why do they never understand? He can't go back, not for her, not for Susan. He can remember — and a man is the sum of his memories — but he blocks those memories, and so is he the same man? No, he isn't. He can't be.
But she is still Jo, still full of hope. And the truth is he hates seeing her like this, seeing Sarah like this. One step closer to dying. That is why he never goes to the Brigadier's funeral. To him, the Brigadier is still young and strong, protecting the Earth from any danger. And so is Sarah. And so is Jo.
But they can't understand. They can't understand how their memories slumber in his mind and cause him joy and pain. They can't understand why he doesn't go back, even though he promised, he promised.
Later he tells her that he went back to see all his companions. And the TARDIS could never find her, running around like that.
Rule number one: the Doctor lies.