That's a really tricky one Macca, very early risers packed and ready to go by 7am and waiting for others or leaving the group for periods can also be very disruptive in a group traveling trip. I have seen quite a few groups fall apart because of this. It can create significant tension at the start of the day for everyone. I respect what your saying don't get me wrong but I have seen this cause some very good people to be at logger heads and destroy big traveller trips.
It's really hard. This is probably the hardest part of running trips and running clubs.
Traveling trips should be done as a group wanting the same thing. Agreeing on a roll out time and everyone agreeing and adhering to it will self Police. Splitting groups can be a real problem, especially if the main group has a breakdown or major recovery.
If you were planning a big interstate trip with a big group and plan on a number side ventures then as long as everyone in the group is clear that that was the plan and people would be venturing off here and there then everything is manageable can be tough finding the middle ground.