Is tax-loss harvesting actually worth it, or am I just deferring the tax?

It's largely deferral plus arbitrage, and both can be valuable. Harvesting a loss lowers your basis in the replacement shares, so you'll owe more later when you sell, meaning the tax often comes back eventually. The real wins are: deferral itself (paying later lets the saved money grow), rate arbitrage (using losses to cancel high-rate short-term gains or $3,000 of ordinary income while future gains may be taxed at lower long-term rates), and the possibility that the basis resets tax-free at death via step-up. If your bracket will be lower in retirement, or heirs inherit with stepped-up basis, the deferred tax may shrink or vanish. It's not free money, but for taxable investors in higher brackets it's usually a net positive.