Its an interesting concept. Doctors can be wrong and your doctor is an idiot. Sorry. Its just the most ridiculous thing I've ever heard.
I have heard that the hormones associated with pregnancy affect the breasts. I have heard that previously stable breasts will change during the first pregnancy. My source: A memory from the Phil Donahue show.