I do make sure there isn’t any paint left on the stirrer. And this has only pertained to water-based paints. A toilet-brush could be sanitized but is a different beast altogether. As was stated by Tom the ick factor is just too strong!
I use a regular kitchen sponge that is gripped by one of these:
Whatever it is called…