From a machinist's point of view, your question lacks som vital information. Do you want a slip fit or a press fit? In my limited experience with handle material, I would shy away from a press fit ( typically .123" to 124" for a 1/8" pin which yeilds a .0005" to .001" press fit). Pressing through would likely result in chipping the edge of the hole in the handle material.
More likely, you would want to have the pin slide freely into the hole where it could then be epoxied or peened to hold the handle material. Another consideration is to check the actual size of your pin. There are sometimes variations in pins that can be decieving. DO NOT USE A CALIPER to measure round stock! It won't be accurate 100% of the time due to variations in the way we hold things. If you want a true measurement, use a micrometer ( we don't let anyone use a caliper to measure critical assemblies at my shop, there's too much room for error, micrometers work even with different people using them.)
The other thing is that drills, even factory fresh drills, rarely drill round holes! They may look round, but if examined under a comparator, they show themselves to be all sorts of interesting shaped, from ellipsoids to triangles! A quick check to see if your drill is off center, start a drill in a piece of material and drill down only a couple of thousandths, see if the mark is triangular, if so, the drill is uneven and won't drill a round hole. We resharpen all drills in my shop as soon as they come out of the box. We use a diamond wheel machine that costs about 3K and ONLY sharpens drills. This is the only way we can assure concentricity from one set of holes to another. ( I will sharpen up to 10 drills free of charge for any member of this forum, email me for information). Also, the point geometry is something that is often overlooked. There are better geometries for particular materials that are different from the standard 118° - 135° point styles. We can also change over drill geometry to suit a particular need. As you can see, drills are a lot more complicated than one would first suspect!