Not Dempsey or 20th century, but if you are interested in boxing as a means of self defence as well as manly sport previous to the present incarnation as a purely sporting form, I have several very good manuals on the topic at
http://flybynight.korolev.com , check the manuals section for Defensive Exercises, Art And Practice Of Boxing, and Science Of Self Defence (hosted on a friend's site also worth checking out)
The vertical fist was used in the old days, but the strikes weren't focused on the bottom three knuckles. Instead it was the index and middle or else as close to all of them as you can practicaly get. The focus was on fast, penetrating blows to critical targets, wrestling and the gouging of eyes were integral to the art, but kicks, elbows and knees weren't, though they occasionaly saw use. BTW, footwork and slipping was being practiced long before the FMA/Boxing conspiracy theory allows for, centuries before, in fact, and activity was one of the 5 attributes most highly prized by the pugilist.
The rules argument is a little misleading. Every martial art that does sparring has a set of rules. Otherwise there is too much risk with little reward. It is knowing when rules apply and when they don't that is critical. Likewise, there are rules on the street. You had better know the difference between a life and death struggle and a friendly brawl. If you don't, you're in trouble either way.
Also, the introduction of boxing gloves and rules isn't what killed boxing. Boxing has had both from very early on, and remained very martial. Queensbury and American Fair Play did a lot to turn it into a more acceptable sport, as the art itself was frequently outlawed for the same reason dueling was. However, Broughton and London Prize Ring rules are much more flexible while allowing a reasonable ammount of flexibility, and there was an understanding of the difference between a friendly sparring match, a prizefight, and a streetfight.
Incidently, boxers used to practice methods of hardening their hands and drunk sailors were breaking boards before modern karate tournaments made it popular in the present era. There isn't so much danger in using the fist to strike the head if you know where you're striking, how to strike, and have conditioned yourself to it. Mike Tyson and the like breka their hands because they are used to punching in big weird gloves, and never recieved scientific training in how to use the bare fist. The fact that they are 1/8-ton giants capable of generating force far in excess of mere mortals probably doesn't help any, either. The gloves, or "mufflers", were worn to protect your sparring partner, not your fists.
As far as that goes, in a protracted prizefight between men of extraordinary strength and courage, you can and often do end up with swelling of the hands. Much emphasis is placed on hardening the soft tissues by striking the bag and applying tannic solutions, but it is still meat, it can swell and bruise. However, this is under the most extreme and protracted of conditions. The few hits it takes to stun a man before you throw him, or in some cases to just plain lay him flat, generaly have no meaningful effect on a good set of knuckles. The most reasonable argument I've heard for open palms is regarding a clenching instinct when drawing a pistol or on a shooting range leading to accidental discharges. Somehow I doubt that extended fingers are any less vulnerable to injury than clenched fingers...