I've seen little need to completely clean off a strop, i.e., stripping away all of the swarf & old compound. Instead, on strops with compound, just wiping it down with a rag (microfiber towel works great), perhaps moistened with a little solvent like WD-40, stoddard solvent, isopropyl alcohol, etc, does enough to remove the stuff that needs to be removed. It won't matter if it still looks somewhat black or dirty; just remove the loose, messy stuff that otherwise would end up on your blade. Then reapply compound.
On something like a denim/fabric strop, I've just used a dry microfiber towel to wipe them down, before applying new compound. I avoid using solvents on those, as it might create a problem with the glues used to adhere the denim to the hard backing.
I've only completely stripped or resurfaced strops if I didn't like the smoothness or character of the substrate itself. For example, I've sanded leather to give it more 'nap' to accept stick-type compounds, as it doesn't apply very easily to smooth leather. And I've sanded or planed wood strops smooth, to get rid of surface irregularities (gouges, bumps, 'waves', etc), to improve their performance. And a time or two, I've done it to prep the surface for a different, finer compound, per that point mentioned earlier.