I know, I believed it too until J.K. Rowling shot that theory down during an interview with Melissa Anelli and Emerson Spartz.
MA: What about Harry's family — his grandparents — were they killed?
JKR: No. This takes us into more mundane territory. As a writer, it was more interesting, plot-wise, if Harry was completely alone. So I rather ruthlessly disposed of his entire family apart from Aunt Petunia. I mean, James and Lily are massively important to the plot, of course, but the grandparents? No. And, because I do like my backstory: Petunia and Lily's parents, normal Muggle death. James's parents were elderly, were getting on a little when he was born, which explains the only child, very pampered, had-him-late-in-life-so-he's-an-extra-treasure, as often happens, I think. They were old in wizarding terms, and they died. They succumbed to a wizarding illness. That's as far as it goes. There's nothing serious or sinister about those deaths. I just needed them out of the way so I killed them.
MA: That sort of shuts down Heir of Gryffindor [theories], as well.
JKR: [Pause.] Yeah. Well - yeah.
MA: Another one bites the dust.