How much airflow out of the vents do you have when you start driving and how cold is the air? When your A/C gets to the point of not pushing any air out of the vents can you still hear the blower fan working hard? If the fan is still working hard is the air coming out of any of the other vents?
The most likely cause of what you are describing is the A/C evaporator freezing up and becoming a solid block of ice that no air can pass through. The most common cause of freezing of the evaporator is a faulty or out of adjustment A/C thermostat. Next time the fault occurs pull over (where safe obviously) and open the bonnet. Have a look at the A/C pipes where they go through the fire wall (the photo shows which pipes I'm refering to), are they iced up?
Tony