Nope, leak in the vacuum should raise the RPMs not drop them. At idle when throttle flap is closed there is limited amount of air getting in so if there is vacuum leak when you apply the brake that acts like slightly opening the throttle bumping the revs up.

If you had a leak it would also run out of vacuum while engine is off on first application of the pedal too.

Of course doing proper vacuum test would eliminate the guess work but I don't think you have a problem from what you described.

Cheers