Yeah 3 days is about right. It's not that it won't effect you, but it takes about twice as much to get the same high.
Actually, it takes a while for the effects to wear off. I always felt a bit jelly-headed the next day at least. So, I guess it isn't just that it won't affect you, but that you are actually already affected from the previous trip. You need to start out from dry sober to truly feel the difference that the psychedelic makes (or take a lot more).