I'm open for suggestions and at the moment the only place I can think of is about 20km south west of Ballarat at the Smythesdale Gardens.
https://caravanontour.com/free-camps...rest-area.html
Lots of shade about, flush toilets and hot showers, (gold coin donation but I'll have to check) and about 40 sites.
There are actually some reasonable 4WD tracks in the surrounding bushland, these are used by the local 4WD club for training and accreditation. (The tracks are always more interesting in the wet). No swimming but Lake Burrumbeet is about 15km away.
I'm happy to check it out in the next week or so if anyone is interested, yell out if anyone has a better suggestion for a site.