Pretty self-explanatory. Without getting to deep into it, I (straight, except for him now) managed to make myself fantasize about him to the point where half of what I fantasized about was him. He's not very attractive nor is his big, from what I've heard, but I somehow did that...
If we are honest, I imagine that we have all been guilty at some point in our lives. But fantasy is a far cry from reality, so be careful not to let it spoil everything. This applies to all aspects of our lives, not just relationships.
It is normal for curious people to explore sexual offerings beyond the boundaries of what is familiar to the average Joe and Jane. I'll leave it at that, since you don't say what exactly attracts you in this case.
his penis... literally just his penis. Weird thing is, I don't like any other guy like that. Just him
You didn't "manage to get" yourself to fantasize about him. It was never your goal, it just happened. You're not unique in being turned on by the ex's cock. It happens to me quite frequently. Sperm Competition It's one of the reasons why my wife, her boyfriend, and I (and anyone else who has MMF* threesomes) enjoyed that threesome so much. And the three of us did not need to be together to experience that. She could be off with him and I'd still experience the turn on. We all felt its effects. It transcends physical or temporal (time) proximity. All it needs to kick in is just for your mind to harbor the thought, "Hey, his cock fucked her!" It's literally just about his cock and the fucking. That's all it needs. I'm not going to cover much detail about sperm competition. (Too long-winded even for me!) Among it's effects in humans is that it causes better fucking and better orgasms! Yay! That's why your mind is on his genitals. Thinking of his cock being inside her, regardless of when it happened, revs you up. For you, it has caused the side effect of being turned on by that cock, ergo, your fantasies. Enjoy 'em! *I usually just write MMF because for me it's the equivalent of MFM. They're the same, in my worldview.