Growing numbers of Social Security beneficiaries have to pay taxes on those benefits but even that burden falls far below what is assessed against typical employer-sponsored annuities, according to the Congressional Budget Office.

Social Security has complex tax provisions:

Beneficiaries with income equal to or below $25,000 (for unmarried taxpayers) or $32,000 (for married couples filing jointly) pay no taxes on their benefits. About half of all recipients were in that tier in 2014.

Beneficiaries with income that is higher than those thresholds but no higher than a second set of thresholds—$34,000 for unmarried filers and $44,000 for joint filers—pay taxes on up to 50 percent of their benefits (with the percentage increasing from 0 percent to 50 percent as income rises).

For beneficiaries with income that is above those higher thresholds, as much as 85 percent of the benefits are taxed (again, with the percentage increasing as income rises).

The net result of that arrangement is that higher-income beneficiaries have a higher tax obligation.

“Less than 30 percent of all Social Security benefits paid out in 2014 were subject to income tax. By contrast, distributions from defined benefit pension plans were entirely taxable, except for the part of each distribution that represented the recovery of an employee’s “basis”—that is, his or her after-tax contributions to the plan,” a report said. “If benefits paid by the Social Security program were treated the same way—by taxing all benefits that exceeded the basis—federal revenues would be more than $400 billion higher over the next 10 years, the staff of the Joint Committee on Taxation estimates.”

CBO added that the Social Security taxation thresholds haven’t been raised over the years for either inflation or income growth, meaning that more and more beneficiaries are paying taxes on those benefits as incomes from those benefits and other sources have increased. It said that if no such adjustments are made moving forward, the total share of Social Security benefits that goes back to the government in the form of taxes will rise from the 6 ½ percent of today to more than 8 percent by 2024.