The charity is calling for a £50 cap for those wanting to access their pension pot.
Over 55s using the new pension freedoms are losing as much as 10% of their retirement savings in exit fees, according to new research from Citizens Advice.
The charity carried out in-depth interviews with more than 500 people who have accessed their pot since the Government introduced the new rules in April 2015.
Read: Pension freedoms: all you need to know for more on how the new rules work.
It found that people have been faced with "high" and "confusing" exit fees and other charges when trying to get at their savings, which could be impacting how many people shop around and make the right choice for their situation.
Interested in SIPPS or Stocks & Shares ISAs? Visit the loveMONEY investment centre
The fees
The analysis revealed that as many as 160,000 people have paid to access their pension since the freedoms were introduced, paying out £1,577 on average.
Worryingly, those with smaller pots faced much higher average charges.
People with pensions of £20,000 or less faced an average exit fee of £1,966 – which means almost 10% of retirement savings for those with smaller pots is being swallowed up by providers.
The Financial Conduct Authority (FCA), recently proposed a cap on exit fees of 1% of a person’s pot value.
However, Citizens Advice thinks that’s still way too high and is calling for a flat £50 fee to cover a provider’s administration costs.
Other limits to freedoms
The research, which forms part of a new report called ‘Drawing a pension’, also threw up other concerning findings that highlight limits to the pension freedoms actually benefitting retirees.
The charity found 70% of those accessing their pension savings were not shopping around for products, meaning they risk being left with a poor value deal unsuitable for their needs.
This is despite the fact that only a third (36%) said they trusted their existing pension provider, while less than one in three (30%) said they had a product that met their needs.
Worse still, three in ten (29%) stayed purely because it was the easiest way to access their savings and one in six (15%) stuck with their provider because they didn’t want to be hit with exit charges.
People who buy annuities were more likely to shop around, with 57% checking products with other providers. In contrast 39% who bought a drawdown product and just 14% taking cash looked at other options.
Providers are also not offering the full range of freedoms to customers. Nearly half (44%) of those that switched provider say they did so to access the product they wanted.
The report also found long delays for those choosing to access their pension cash. It found that more than half of people received their first payment within a month, but 16% had to wait over two months.
Citizens Advice is calling for the Government to tackle transfer delays by adopting the same transfer time limits that currently apply for ISAs.
It also wants the creation of an independent drawdown comparison tool reviewing the whole market to help consumers compare products in one place.
Gillian Guy, chief executive of Citizens Advice said: “Picking a pension product is one of the biggest financial decisions people will ever make, so it’s worrying that so many aren’t shopping around.
“More and more consumers are choosing drawdown products but our research shows they aren’t checking whether they’re getting the best deal. The Government and industry needs to work together to make it easier for consumers to compare drawdown products and choose the one which best meets their needs.
“The threat of excessive charges can also put people off making the right pension choices for them. A standard £50 exit fee across all types of pensions will mean consumers can make the most of the pension freedoms.”
Interested in SIPPS or Stocks & Shares ISAs? Visit the loveMONEY investment centre
Read these next:
Probe launched into £800 billion occupational pensions blackhole
HMRC Personal Savings Allowance tax bungle leaving pensioners short of cash
Bank of England's chief economist: ‘I don’t understand pensions’