Rory is an honourable man; the Doctor admires and respects him for that. He knows Amy will be in safe hands, no matter how the situation unfolds.
Trapped inside the Pandorica, time will pass Amy by while Rory stands guard, ceaselessly, tirelessly, because he loves her. It’s a greater sacrifice than the Doctor has ever made for anyone he loves, and it makes him feel small, unworthy, because he suspects in the end it makes Rory the better man.
The Doctor would never stay behind, give up everything, for love, but then he’s never loved anyone with everything he is.