There needs to be a “massive reduction” in the number of people sent to prison for 12 months or less, ministers say.
Prisons minister Rory Stewart said short sentences made offenders more likely to commit crime.
He also suggested prisoners released on licence could be used to fill labour shortages in low-skilled jobs caused “partly” by Brexit.
The government is keen to reduce the UK’s prison population, which has doubled over the past 25 years.
Meanwhile, Justice Secretary David Gauke told The Times: “Twenty-five years ago the population was 44,000. Today it’s 84,000. I would like it to fall.”
Mr Stewart said the increase had been driven by longer sentences for violent offenders and “many, many more sex offenders”.
“Nearly a quarter of our prisoners now are sex offenders, many of them older men in for historic sex offences,” he told the Sunday Politics programme.
There were also a lot of prisoners “cycling” in and out of the system on sentences of a few weeks or even less, in the case of those recalled to jail on licence days, he added.
“The statistics demonstrate that sending someone to prison for a very short period makes them more likely to re-offend compared to community sentences.
“So it’s not good for them and it’s not good for public safety.”
Exceptions to any new sentencing guidelines would “probably” have to be made for violent offenders, he said, and it was important to retain an element of “punishment” in sentences.
Community sentences have fallen out of favour with judges and magistrates in recent years over concerns about their effectiveness, amid budget cuts and the part-privatisation of the system.
But Mr Stewart said reoffending rates were “far too high from short sentences” and they were also disrupting the prison system.
He also backed comments by his boss, David Gauke, who suggested prisoners could be let out for the day to work in sectors like catering, construction and agriculture, which were expected to be hit by Britain’s departure from the EU.
“Partly because of Brexit, there is a labour shortage,” said Mr Stewart.
“Which means that prisoners with the right kind of training, the right kind of attitudes, as Pret a Manger are finding, or Greggs the bakers are finding, or Timpson’s the shoe company are finding, can provide very loyal, good employees.”
The government has made a series of U-turns on allowing prisoners out to do work under licence, amid concerns some were absconding to commit further crime.
Mr Stewart admitted the government had gone “back and forth” on the issue but, he said, prisoners who had a job to go to were less likely to re-offend, and “only one in 10,000 people released on temporary license go on to commit a serious crime”.
In a separate development, Mr Gauke claimed middle class drug users should feel “guilt and responsibility” for fuelling the rise in fatal stabbings in the UK.
“People who do that have to recognise they are fuelling the industry that’s resulting in the knife crimes, resulting in the difficulties we’re having in prisons,” Mr Gauke told Sky’s Ridge on Sunday.
“There’s a responsibility for middle class people that take cocaine at a dinner party, that when they see a story of a 15-year-old boy stabbed in Hackney (east London) they should feel a degree of guilt and responsibility.”
Police Federation deputy treasurer Simon Kempton has also blamed the wealthy for creating the demand for cocaine, while security minister Ben Wallace warned the UK was “fast becoming the biggest consumer” of the drug in Europe.