Elm can be a very good choice for handle material. It compares well with white ash. See the charts on page 4 and 5 of this document.
http://www.fpl.fs.fed.us/documnts/fplgtr/fplgtr113/ch04.pdf
Besides ash and hickory have a look at these woods:
American Beech
Sweet and Yellow Birch
Rock Elm
Honey Locust
Black Locust*!!
Sugar Maple
Cherrybark and Scarlet Oak
Live Oak
Black Locust isn't hardy this far north but Honey Locust is increasingly being planted to replace EAB killed Ash trees. The farmer across from where I grew up (and now live again) planted Locusts around his house already 100 years ago and although he and the farm are long gone the now-huge trees are still there.