First question: Define modern. Early modern steampunk? Today? Twenty-minutes-into-the-future cyberpunk? High cyberpunk dystopia? Post-apoc hellscape? Early space flight and planetary romance? Caped crime fighters? Superheroes?
Second question: What do you want it to -feel- like? Please do not take this as a barb; you seem to be coming in from a D&D/PF mindset. Now, I love me some PF (and D&D before 4th, but let's not debate that), but each of those has a power fantasy of being a bad ^$(@(*^& who wrecks house. There are a lot of games that cater to that, especially in high fantasy. But in "modern" games, there are settings where the players are bit players in a cosmic struggle. There are games where money and influence are the real weapons. There are games where street samurai and computer hackers raid low-orbit stations. There are games where drunken mages channel their alcoholism into magic power as it destroys their lives. You need a tone first.
Scion, Changeling the Dreaming, Mage the Ascension, Mage the Awakening, and Call of Cthulhu can all be stripped of their attending intellectual property if you want to use them as a sort of "generic" modern magic setting. Shadowrun as written combines cyberpunk with a fantasy kitchen sink, if that's your thing. Strip out the fantasy, and you want Cyberpunk 2020. Modern zombie/horror style? All Flesh Must Be Eaten. If by modern you mean sci-fi, Traveler or Stars Without Number are excellent - both have space opera and psychic powers but that can be cut out to make room for the ground-game, so to speak. I've seen a few people do a "Dark Terra" thing, using Dark Heresy rules but eliminating the 40k IP and turning it into struggles on earth. Deadlands and Deadlands: Hell on Earth go for magical dark fantasy in the early industrial revolution and then in the post-apoc world that follows.
One of the best and creepiest games I've ever run was Silent Hill inspired, using pure Old World of Darkness mechanics (Advantage: extremely simple to use, Disadvantage: combat is -lethal- and takes too many rolls to resolve an attack (Normally 4)). The dark powers were lifted willy-nilly from OWoD and CoC sourcebooks.
Do not, for the love of God, use D20 Call of Cthulhu.
So after all that, a few recommendations:
Call of Cthulhu: The granddaddy of occult horror games. The PCs try to stop cosmic horrors from Lovecraftian style fiction while battling their own insanity. Incredibly lethal. Incredibly fun.
Shadowrun: It's not for everyone, but a quirky blur of fantasy and cyberpunk. The characters are amoral mercenaries in a world straight out of a bad campus socialist's pamphlets, hacking computers, casting spells, and slicing up guards for a paycheck from some other heartless multinational company. Also, dragons.
Unknown Armies: In addition to being one of the simplest and creepiest games out there, UA features really unique takes on "postmodern" magic. For example, the Pornomancer is an individual who indeed gets power from sex. Unfortunately, it's from religious devotion to a porn star who ascended into the heavens on tape. Sounds fun? Wouldn't it actually be horrifying to have your sexuality reduced to blindly mimicking someone else's degrading acts? To never develop yourself an identity, desires, or love because you're too busy re-enacting scenes from a terrible skin vid, all for a charge of magic? Not for immature players, but an amazing game - and a huge advantage is that the rules are super simple and it's designed for players to play more or less whatever they want.
World of Darkness (Old or New): These are complete settings and they are massive. There's a lot of hatedom between old and new WoD, which is just silly. In the WoD, the masses of humanity are generally unaware that vast conspiracies of every type of supernatural creature in existence is out there. Beyond that...there is no easy summary of this one. The number of powers and books out there can be overwhelming.
Scion: Like WoD in the Modern Era, except you're a walking bomb, a child of a god who's ready to lay the eternal smackdown. Very like Exalted or what Paizo promises Mythic Rules will be like.
Deadlands: In the 1860's, America and all Earth recovered magic, and horrors were set loose upon it, turning earth into a terrifying place. The American Civil War still rages on, undead things stalk the night, and spirits empower both the servants of God and seers from native tribes. You step into this with a gun, a longcoat, and maybe a Bible, deck of cards, or a shaman's paraphernalia. Features really cool mechanics involving poker hands and chips for a better Old West feel. The sequel, Deadlands: Hell on Earth, assumes the bad guys in Deadlands won in one timeline and is set in the nightmare created as a result.
Traveller: This is a space opera game for people who aced all their sciences. Aside from FTL, it actually aims for a very hard-science approach. Character creation as a mechanic alone is worth mentioning; your character gets there stats, then has a career before the game. Events during that career shape your character into who or what he will become when the game begins. Strip out any future tech, and it works for modern.
All Flesh Must Be Eaten: Incredibly simple. Bad zombies. Stay down.
Dark Heresy: The ruleset can be adapted to a modern game, though it would take work.