An entertaining theory:
The Hobbit is a great book; a quick adventure through a sketched-out world with a lot of monsters and clever puzzles, a battle that doesn’t drag on forever, and a satisfying end. The Lord of the Rings is the evil opposite, the result of an academic allowed to write without an editor, with obsessive world-building supporting a feeble thread of a story, vaporous non-characters, and little action for endless thousands of pages before collapsing in an anticlimax. The Silmarillion is literally the Bible of Middle Earth: It is a fiction anthology fabricated long afterwards to support someone’s religion, and bears almost no resemblance to actual facts on the ground.
However, the one Hobbit-like part of LotR was Tom Bombadil and the Barrow Downs. For one moment, there’s an adventure in a spooky world, monsters, and inexplicable magic. Naturally the kind of humorless wankers who like the endless tedium of Lord of the Rings hate it, and that jackass Jackson cut Bombadil in the movies, which means you get an hour more walking scenes instead of a dungeon adventure.
Real primitive societies develop religions, make up dozens or hundreds of gods, and spend serious effort on ritual and sacrifices even without a trace of evidence. But Tolkien’s Catholic lunacy meant he couldn’t acknowledge other gods, even in fiction, and so there’s this bizarre hybrid of a singing god and angelic and demonic beings, and then things he wanted to borrow from mythology which only make sense with pantheons of gods, and since he couldn’t commit blasphemy nobody bothers to worship any of them.
My take has always been that Bombadil is one of the Old Gods, one of three known to exist in Middle Earth (himself, Ungoliant, and maybe Morgoth?), even though nobody worships them, and the hobbits who live a day’s travel away mostly don’t know Bombadil. Seen as one of the Old Gods, a meddler like Loki but with less range, Bombadil’s behavior is perfectly typical: He lets you get in trouble, saves you so you owe him, and sends you off with more power to do his dirty work of killing Morgoth’s pet Sauron.