This one time at work I was talking with this lady and she said "I am hungry, literally." I laughed and said "Are you ever figuratively hungry?" She said "No." I thought it was hilarious.
Well it made me laugh, so its funny to me...but usually we keep those kinds of jokes inside of our head... Plus your punchline could have been delivered in a different, more better way. You could have said sarcastically "I'M hungry, FIGURATIVELY." With emphasis on the words in higher case. Or something like,"Oh well, I'm hungry literally too, figuratively speaking of course". Now that one would have been real witty! Overall your sense of humor is OK but you should try and involve people in the jokes. If you don't then the jokes are only personal, and those (like I said above) should stay in your head