Expanding/editing my comments from What Makes Something Broken G+ thread:
“Broken”, for me, is anything that makes normal character choices, tactics, or roleplay irrelevant.
“Normal”, for me, is whatever’s described in the setting. This is sometimes implicit, but there are books mentioned in the D&D prefaces, Fafhrd & the Grey Mouser, or John Carter of Mars, or some Cimmerian thief named Conan.
The World of Greyhawk looks like 100-Years-War to Napoleonics, with Elves and Dwarfs and Giants on the fringes. Blackmoor/First Fantasy Campaign looked like the Italian Renaissance on a border with Mongol hordes, with cults armed with alien tech. Empire of the Petal Throne looks like the Aztec Empire, Vedic period India (pre-600 CE), a lot of advanced tech, weird aliens, and quasi-Lovecraftian gods; things that are broken in default setting or Greyhawk may be more passable in Blackmoor or EPT.
Exhibit 1: Sleep:
Sleep: A Sleep spell affects from 2–16 1st-level types (hit dice of up to 1 + 1), from 2–12 2nd-level types (hit dice of up to 2 + 1), from 1–6 3rd-level types, and but 1 4th-level type (up to 4 + 1 hit dice). The spell always affects up to the number of creatures determined by the dice. If more than the number rolled could be affected, determine which “sleep” by random selection. Range: 24”. [mdh: 240′ indoors, 240 yards outdoors]
—Book I: Men & Magic
Sleep spell is my classic example of “broken”. Many players love original edition no-save long-range area-effect Sleep, rampaging thru a dungeon with it, until a Referee has an NPC magic-user TPK them with it. Walk into the far end of a 240′-long corridor, or cross a field at 1/7 of a mile away, and there’s no save, fall asleep, throats cut. Then everyone’s unhappy.
The only defense is taking up to 16 equal-Level mercenaries along, so some of them will get slept instead of the PCs, but then the PCs are probably paying equal shares and make no money, which in original edition either directly becomes XP, or can be spent on training/orgies to get XP. So progress grinds to a halt because of this stupid spell.
This is not “normal” in anything except maybe EPT, where a small army of slaves were de rigueur dungeoneering gear, and got expended in traps and spells.
There is nothing else you can do. Die or cower behind a wall of cannon fodder. No rational NPC Magic-User would ever have any Level 1 spell but Sleep ready.
In practice there’s a polite agreement where the Referee doesn’t give common hostile NPCs Sleep, because the game ends if they do. Just searching TSR’s B-series PDFs, I see one hostile NPC with Sleep in B4 Lost City’s Tier 3, one boss you shouldn’t stand-up fight in B6 Veiled Society, a deadly magician in B7 Rahasia, and a couple in B9 Castle Caldwell. All of the other sleep-inducing gasses and other effects give saving throws.
In less dogmatic games, the Referee just house-rules a saving throw in and maybe reduces the range, and it’s a non-broken spell again.
Exhibit 2: The Ranger:
The additional classes (from Strategic Review, early Dragon, and Supplements I-III) beyond the first 4 are generally broken. Rangers, Paladins, Cavaliers, and Barbarians are better than Fighters in every way. Assassins, Bards, and Monks are better than Thieves in almost every way (trivially lower skill chance). Druids are better than Clerics at spell-casting, weapons, AND can turn into bears! Illusionists are the only somewhat balanced additional class. There is no reason to take a stock Fighter, Thief, Magic-User, or Cleric if you can take one of the additional classes.
The setting implies mostly human warriors, thieves, wizards, and priests of varying types. Demi-humans without level limits, balanced stats, or some other drawback are broken, because there is then no reason to be Human. In OD&D, Dwarfs are limited to 6th Level, Elves are limited to 4th/8th Fighter/Magic-User, Halflings to 4th Level; non-Humans rarely appeared in Gary Gygax’s groups.
You can make a setting for that, I used to run an urban-vs-wilderness setting where the primary classes were Assassins, Cavaliers, Illusionists, and Nightblades (reskinned Bards) vs Rangers (toned down from the RAW), Druids, and Barbarians (toned down even more). A generic “fighter” or “thief” didn’t exist, and would have been completely overshadowed by better classes.