GU is drifting left and right, and "clunking" when applying brakes

[Sights]

So, I will do my best to describe a very frustrating problem that suddenly appeared after replacing front inner and outer wheel bearings.

After doing the bearings and taking my first trip into town, I noticed that the GU was drifting from the centre of the lane towards the shoulder or centre line.
Not a wheel wobble/shudder, but just veering L & R as I was trying to stay in the middle of the lane.
I'm not sure if it's related, but sometimes, not consistently, when applying the brakes, I have heard a metal scraping sound or clunking noise. These noises are not heard at the same time, and not every time I brake - it's intermittent!
I did replaced brake pads on all wheels around December/January, and all seemed to be working well until recently. The brake noise did start about a week of two before changing the bearings.
A "kind of a metallic rattle" started today, and got louder over about 4klm until pedestrians started to look at the car, obviously hearing the noise. I parked at the kerb, checked all the wheels for movement, looked under the front and back for - anything!
Finding nothing, I took off again and the noise had just disappeared! The only thing I had done it to use the brakes to pull to the kerb.

Have rechecked the bearing install and had a mechanic friend check at the same time - all good.
Have replaced the driver side tie rod end, as it wouldn't have passed rego, and had a wheel alignment done - this gave a minor improvement to the drifting, but it is still there.

Today, went back to the mechanic friends workshop and did some testing. Put the front on stands and looked at everything that moves and/or has rubber.
Noticed some knocking when moving the steering wheel left to right and adjusted the steering box as much as practicable.
Also noticed that when on full lock (left and right) and rocking either wheel, there is a much noisier "knock" then when wheels are cantered, which, try as we might, we could not find the source.
After using a stethoscope, it was decided that the pitman arm/steering rod [can't remember the right proper name for it ] connection was the most likely culprit, but outwardly at least, it seemed to be in good condition.

So, two mechanics, one is a 4 wheeler himself, have driven the GU, felt the drift, heard the noises and can't find the cause...........HELP!!!
I am open to any suggestions as it is quite unnerving driving under these conditions and wondering if a something disastrous is going to happen at 80kph (ain't going over that speed till I find the solution).