As others have said I have only seen bronze selector forks on the BA. These have had casting marks on them 43 BA and 42 BA (which are the part numbers not the order part number if you know what I mean)
A you have explained the dowels in the selectors and the grooves in the cam provide the means of moving the selectors sideways to move the appropriate gears, they do not play any part in keeping the selector shaft or selectors in the correct position for any gear. This is done by the detent groves machined at the end of the cam part of the shaft into which the pawl locates. Once the shaft has been rotated by the gearlever and the pawl has fully located in the relevant groove (under pressure from its spring) there should be no way thatthe selector can move sideways.
However, if there were any wear in the dowel pins or cam groove this could allow the selector fork to move sideways, although if this was excessive you would expect it to effect all gears. Same goes for excessive end float of the camshaft.
I think I would look at those selector forks. There may be something in this previous post that may help.viewtopic.php?f=4&t=6282