Hi Tom!
Welcome to BFC and welcome to the swordforum!
You didn't mention what kind of sword you have, whether European or Japanese or other, so I'll talk about European. If your blade is japanese, ignore this, you'd never want to treat it the way you'd treat a western blade.
First, I agree with you that you want to keep the pitting and patina after you have the blade cleaned up.
Start off with some 000 or even 0000 fine steel wool.
(At this point, steel wool is preferable to sand paper, as sandpaper would clog almost immediately and become useless.)
Gently work the rust off with with long smooth motions. DO NOT sit and rub hard at one spot until you get it shiny, just make long smooth strokes from quillion to point until you get it down where you can see some clean metal showing through the rust. You want do treat the blade as a "whole" rather than bit by bit.
After you've gotten the whole blade to more "shine" than rust, you'll want to take a cloth and a small amount of oil to the blade again, to remove any particulate rust from the surface.
Then wipe the blade down until it is as free of oil as you can get it. (Some will remain in the pitted areas, and where rust still remains on the blade.) Then switch to 1000 grit sandpaper and GENTLY begin working the blade with it, again working from quillion block to tip in smooth continuous strokes.
(1000 grit can be found in the Automotive Department at Wal-mart)
Stop every so often, and with the oily rag, wipe the blade down to remove the rust particles you've disloged, and continue gently sanding until you have gotten all of the rust off.
Some areas will be blackened, don't worry about these, and don't try to remove the discolorations. Just remove the rust and leave the discolorations, as this lends a lot of character to an otherwise not especially noteworthy blade.
If this sword is a "sharp" the edges will almost certainly require re-dressing. If you feel confident that you can do this, I advise you start with a NEW flat bastard file and gently re-work the edge until you have a good clean straight edge restored. Then move to a course whetstone, and if applicable (depending on what kind of sword it is) you can move to progressively finer grades of stone until you achieve the desired edge.
If you don't feel comfortable doing this, you may wish to take it to a professional sharpener.
Hope this helps!
