Salvia is nothing compared to weed, or lsd. Salvia is absolutely different than anything else out there, do not let it turn you off anything, and definitely do not compare, especially to pot.
You just need to "break through" with salvia. Something is being done very wrong to have smoked half a gram and feel mild effects. Read around a bit on it. I have seen even experienced LSD and shroom users try salvia and get humbled once they got the effect.