Saruman accuses him of not consulting with the Order sooner:
The hour grows late and Gandalf the Grey rides to Isengard seeking my counsel.
and in the next scene:
The hour is later than you think.
Gríma echoes this to Théoden:
Late is the hour in which this conjurer chooses to appear.
Azog in The Desolation of Smaug says he should have come to Dol Goldur sooner:
You have come too late, Wizard! It is done.
And he arrives at Helm's Deep in the very late stages of the siege, though it was precisely when he meant to.