SFM dictates RPM. RPM, number of teeth and feed per tooth dictates feed rate. SFM for a HSS cutter in 304 should probably be around 70 SFM.
For a 1/8" HSS cutter in 304, you should probably be turning around 2000 RPM. You should feed about .001" per tooth. That is hard enough that you're not rubbing, but not so hard to overload your cutter. A four flute that is probably around 10 IPM (wild guess) or 5 IPM for a two flute.
Your depth of cut should be shallow enough that your setup doesn't chatter at your RPM and feed rate. Your small mill should probably be .010" per pass.
304 work hardens badly, so you should avoid rubbing such as dwelling in one spot or feeding too slow. Avoid high RPM because chatter will cause hardening. 2000 is nice and slow for 1/8" cutter. Cutting oil with chlorine and sulfur (the stinky stuff, not WD40) can reduce galling and work hardening from friction. Avoid dull cutters.
Plunging with an endmill in SS can be iffy. I'd drill out the ends with a 3/32" drill so the mill can plunge mostly in air. I'd also use an undersize cutter for stock to clean up with finishing cuts. Lift into the air to feed over in Y for the finish cuts.