TeenHealthFX is very concerned about you. Although you do not mention how old you are, there is an obvious age gap between you and this man. The first and most important question to ask is if the sex is consensual (are you having sex because you want to)? In other words, if you are being forced, physically, verbally or through any other means to participate then this is rape! Regardless, TeenHealthFX strongly urges you to speak with a trusted adult immediately, whether a parent or other family member, a doctor or counselor, or even the police. For further information and support regarding rape and sexual assault, contact the Rape, Abuse and Incest National Network Rape Crisis Hotline (RAINN) at 1-800-656-HOPE.
Even if sex is consensual and you are a willing participant, TeenHealthFX is still very concerned. Although you do not mention where you live, it is against the law for an adult to have sexual relations with a minor. TeenHealthFX urges you to stop having sex with this man and to speak with a trusted adult who can help you sort out your feelings.
You do not tell TeenHealthFX if your boyfriend knows about this situation - if not then his grandfather is not only taking advantage of you, but also his grandson. Again, TeenHealthFX recommends ending this relationship immediately.