Actually, you can patent game rules. They're considered an invention. Check out US patent number 6,017,034 which is a patent relating to collectible card games. Fortunately, most rpg publishers have not pursued patents for their games, meaning all you have to deal with is copyright (and you can't copyright game rules, just their artistic expression). Apparently, there's so much in-breeding of ideas in RPGs that it would be fairly hard to patent most conventional RPG "technology" now.
And I am not a lawyer, so take all that accordingly.