I've got 3 benchtop drill presses, but I'd say 99% of my drilling is done on my knee mill. It's got a full size vise, longer/larger capacity, and with the DRO (or just the dials) I can indicate and position holes EXACTLY where I want them. I use a keyless chuck on an R8 arbor and use a small pneumatic impact to zip it on and off the drawbar, so changing things out as I need it is very fast and simple, and the machine is just gonna be more solid and accurate than any drill press from the big box stores.
That's not to say there isn't a time and place for dedicated drill press, and there are times I wouldn't mind a decent floor model, but if I could only have one or the other, it'd be a knee mill, no question. Granted, a full sized mill (used) is gonna set you back probably a couple grand more than your largest drill press from Home Depot, but you'll find it about 1000x more useful and well worth the investment.
If it was between a mini mill and dedicated floor press, I'd take the mini mill too.
Now, all that said, keep an eye out on your local classifieds for old industrial grade drill presses. A lot of times they just need a little bit of clean up, but even if you have to rebuild the chuck or put on a new motor or bearings, you can potentially get a pretty solid machine for a pretty good price that's gonna blow away anything from the big box stores, and it'll likely still be cheaper than buying a mill with all the attachments. Also, the machining forums will often have dozens of threads on rebuilds for certain models that walk you through changing various parts.