Yeah, I have been trying to wrap my mind around this, and this must be how it works, I think. We have two clear states, the mundane world with a spirit fully manifested (and visible to everyone) and the spirit world, that only looks a bit like the mundane world (and is invisible unless you discorporate and go there, and from which the mundane world is equally invisible).
But something like a ghost, or perhaps the spirit of a boulder, or a non-manifested disease spirit on the prowl, seem like spirits that exist in the mundane world but non-manifested and invisible except to second sight. At least that’s what makes sense to me. If you can’t see a ghost using second sight, then it’s weird.