25 and High Blood Pressure?

The last few times I've gone to the doctor and got my general stats taken, my blood pressure has been pretty high (consistently 180/120). I am 25 years old, 5'8" and 150lbs. I eat pretty healthy (not a lot of deep fried crap; varied diet of fruits/vegetables/poultry), don't smoke, and exercise pretty regularly (I try to get my heart rate up at least twice a week in the swimming pool). The only major problem I have is a congenital birth defect that has caused degenerative disk disease and deformities in my legs. I recently saw a pain specialist about my back and he was the one that's been monitoring my health information over the past few months. He said that my high blood pressure could be due to elevated levels of pain or stress, but that I shouldn't worry too much about it. However, everything I've read points to this being a serious problem that I should take action on it right away. Any thoughts on this would be greatly appreciated. Thank you.