Probably not too bad, unless in very large quantities (in relation to total water volume). IIRC, some Phosphate removal medias (like PhosBan) are made with a form of Ferric Oxide...
If you're worried, a large water change and running a PolyFilter would help.
A few "stainless" steel out there are not for marine environment. Many powerhead use SS shafts that rust slowly, for example. It is pretty harmless, for there are some non-ferrous metal in SS that can be bad for the inverts. Should be alright in most cases unless you have large quantity of fast corrosion due to galvanic effect.