Two goals from the returning Leon Clarke were enough to cancel out Will Grigg’s penalty early in the second half and send Bury through.
Grigg coolly put Latics in front from the spot in a game they very much had the better of, but the visitors were clinical in front of goal, and took their chances to ensure their progression.
Latics made four changes to the side which lost at Coventry on Saturday, with Sanmi Odelusi and local lad Jordan Flores were brought in for their debuts. Leon Barnett and Kevin McNaughton were also drafted in, replacing Chris McCann, Donervon Daniels, Jonjoe Kenny and Francisco Junior.
Caldwell’s men made a bright start to the contest, with Michael Jacobs and Will Grigg looking like they have already built up a good understanding, with the pair linking up well together in the early stages.
Jacobs was unlucky to see several clever pieces of play just cut out, with the visiting defence just doing enough before the forward could find Grigg up top.
It was the local lad Jordan Flores who created the first real opportunity of the game in an eye-catching performance in the ninth minute, when he produced a blistering run deep into Bury territory and found Jacobs, who saw his shot blocked on the edge of the area.
Latics continued to dominate in the opening half an hour, with Jacobs, Grigg, Flores and Sanmi Odelusi all looking good for the hosts, who were just lacking that clinical touch in the final third.
Wigan Athletic continued to turn the screw, and the hosts had two golden opportunities within sixty seconds in the 33rd minute to go into half time ahead.
The first came from Flores, with whom Bury were struggling to cope, when he burst from inside his own half and eventually rounded Christian Walton in the six yard box, who recovered to produce an excellent save from close range.
Latics kept up their momentum, and their second chance within a minute followed, when Jacobs narrowly put the ball over the bar with his effort from the left hand edge of the area.
Bury had the ball in the net from a rare break five minutes from the interval, when Clarke fired past O’Donnell, but the linesman had his flag up for offside, meaning the sides went into half-time level.
Latics seemed revitalised by the break, and flew out of the traps straight from kick-off in the second half, and the hosts won a penalty in the opening twenty seconds, when Jacobs was brought down by a clumsy challenge by Peter Clarke on the right hand side of the area.
Will Grigg stepped up to take the spot kick, and coolly slotted the ball into the middle of the net to put his side in front.
Wigan Athletic were electric after the goal, and had a number of opportunities to increase their lead, with Grigg again proving to be a good target man up top – this time linking up well with Sanmi Odelusi.
The game was turned on its head midway through the second half however, when the visitors equalised completely against the run of play, predictably, through former Latics man Leon Clarke.
The Shakers broke on the left hand side, and the ball in from out wide was met by Clarke, who rose well to head the ball into the top right hand corner.
Latics were rocked by the equaliser, and took a short while to regain their composure, but Caldwell’s men continued to have the better of it throughout the contest.
Gary Caldwell took action, and introduced Ryan Jennings for his debut and forward Craig Davies, but Latics continued to struggle to find that elusive winner.
Bury were looking dangerous on the break, and the visitors broke Latics’ hearts just two minutes from time, when referee Mark Heywood awarded a penalty after Craig Morgan fouled Tom Pope inside the area.
Leon Clarke stepped up for the penalty, and made no mistake, to send Latics crashing out at the first hurdle.