>What about higher level spells then?
Only relevant for a few classes, and honestly you can get often get away with clever use of the ones you have to hit above your weight class?
>What about better saving throws?
An emergency fallback for a situation where you fucked up.
Basically, it's the OSR mantra of "if you need to roll, you fucked up". You got hit by a trap you could have avoided, got in a fight you shouldn't have, and got caught above your ears in shit.
If you're out of bad guys, plots and treasure, congratulation! You've now won D&D. You can now go home to your father and say say "Yes, I AM winning, Dad".
Rejoice, user, for you have tasted the nectar of the gods!
>It's the main reward for most adventures: swords and other magical treasure are just there to help out.
What the fuck kind of number-munching games are you playing? The treasure is the reward in and of itself - experience is at best a tertiary concern, an added safety-net.
Levels don't get you castles and retainers, fame and renown - hard cash gets you the first two (remember, D&D has never "automatically given a castle at name level", despite what some may claim), and hard cash can also buy you some of the latter. Mostly fame and renown comes from actual actions rather than abstract "experience", though. If you're a famous dragon-slayer, it's because you slayed yourself some dragons. (Or spread false rumors about it, I guess.)
Seriously, though, consider what magical shit gives you. Take an OD&D Fighter and give them +2 sword and plate, and you know what you get? A Fighter with the saves and hit rates of a Fighter three levels higher, with slightly less hit points but also way better AC. (AC0 is some shit in OD&D.)
If your high-level Fighter dies or is inconvenienced, congratulations! Your new guy has a significant head-start. It's more roguelite than roguelike, if that makes sense.
Storygames tend to care less about levels, this is WotC bullshit.