Ian,
I had a look at my replacement 6-spline layshaft (not yet installed) from Quafie. Generally with splined joints, the shafts are centered by the flanks of the spline, not the major or minor diameters. In fact the major and minor diameters should have a clearance so they do not perturb the alignment. The problem is when that spline is used as a dog clutch, some clearance is needed on the flanks to get the elements to engage and disengage. What to do about keeping the gear centered and square to the shaft? Douglas fixed this by making the minor diameter of the gear a close sliding fit on the shaft (and minor diameter of the spline being a continuation of that surface). This helped center the gear all along the shaft from 1st, neutral, 2nd, and top. Quafie used the major diameter of the spline while the gear is engaged on the spline and the minor diameter when it is on the shaft. The minor diameter between the keys of the spline are undercut to provide clearance so that they did not need to be a precise continuation of the shaft surface. As the gear slides along it can tip a little, because it has to have some clearance. The leading edge of the gear can catch where it makes the transition between being guided by the minor and guided by the major diameter. The major diameter seems to be what is catching. In trying the fit and feel of mine, if I apply pressure and tip the gear as I slide it, it definitely feels more 'gritty' sliding along the spine than it does sliding along the shaft. However I can not see any difference in the surface finish of the surfaces that would account for this.
I would not recommend reducing the major diameter of the layshaft splines to prevent fouling or stop the surfaces from touching and producing the gritty feel. If you do, you will loose some of the alignment of the gear while on the splines. Rather I would oil stone a small radius in the keyways of the gear where the major diameter punches through. I think it is the sharp, leading edge of the spline that is digging in when the gear is being slid back and forth that is causing the most trouble.
-Doug