Phil Foden missed from the spot for City while West Ham scored all five of theirs to reach the quarter-finals.
City had won this competition in six of the past eight seasons, including the last four.
In fact Tuesday marked five years to the day since they were last eliminated from this cup, a 1-0 defeat at Manchester United.
But they were held to a goalless draw by a dogged West Ham side, who then finished the job when Said Benrahma converted the winning penalty to the delight of a sold-out London Stadium.
Club captain Mark Noble handed an increasingly rare start in what is his farewell season, enjoyed a fine match and also netted his penalty, as did Jarrod Bowen, Craig Dawson and Aaron Cresswell.
It was Noble who had the first effort on goal with a drive from the edge of the box which was punched clear by City keeper Zack Steffen.
At the other end of the career scale, City’s 19-year-old striker Cole Palmer started up front in a team showing nine changes but still packed with internationals.
Palmer, who followed up his goal against Wycombe in this competition with one in the Champions League at Club Brugge, could have added to that tally when he received a cut-back from Riyad Mahrez, but his shot was too close to Alphonse Areola in the West Ham goal.
As the first half drew to a close Palmer saw a shot deflected over and Nathan Ake headed a Mahrez free-kick wide.
West Ham were a much-changed side as well but still showing plenty of endeavour, with Noble setting up the overlapping Arthur Masuaku who fired just over.
They came even closer to breaking the deadlock when Masuaku stung the palms of Steffen, with Andriy Yarmolenko delaying his attempt to bury the rebound allowing Oleksandr Zinchenko to block.
At the other end the impressive Palmer teed up Ilkay Gundogan, whose shot fizzed across goal, before Kevin De Bruyne’s deflected effort was scrambled wide by Areola.
From the corner, the Hammers’ back-up keeper made a stunning point-blank save to keep out a header from John Stones.
It seemed neither team fancied the idea of penalties, with both attacking at will. Tomas Soucek was next to try his luck for West Ham but sidefooted wide when he should have hit the target.
Hammers boss David Moyes sensed his side, who beat Manchester United in the last round, could possibly record another statement win and made a triple substitution on the hour with Bowen, Benrahma and Pablo Fornals all sent on.
But City were still the more threatening and Palmer squandered a glorious chance when he lashed Raheem Sterling’s pull-back over before Areola saved with his feet to deny Gundogan and tipped Zinchenko’s drive away.
Foden, Jack Grealish and Gabriel Jesus had been summoned from City’s star-studded bench, and after Soucek curled another chance wide for the hosts, Areola held Sterling’s header to take the tie to penalties.