I would thin the first coat or 2 of BLO about 30-50% with good quality paint thinner. I would sand to at least 320 grit before applying the BLO, you are at 500 grit so you should be good to apply the BLO, just freshen up the surface with a pass of the same grit to remove any surface dirt that might have gotten on the handles. You can "sand in" the BLO by applying it with steel wool or wet or dry sandpaper, finer grit with each coat. This will help to fill pores.
The old time rule for BLO finishes was "coat a day for a week, coat a week for a month, coat a month for a year" I am usually using BLO on furniture so 3-4 coats is enough under a top coat like polyurethane, but a BLO only finish could go 6-8 coats depending on the wood and brand of BLO
I note that you buffed your handle, if you used compound you should remove any remnants from the handle with paint or lacquer thinner, or more sanding. The waxes in the compound could interfere with proper drying of the BLO, or make the finish look "muddy"