Sri Lanka, who chose to bat first, posted an imposing 321 for six from their 50 overs before England fought hard in the chase, finishing on 305 for nine.
Mendis was the standout performer, smashing 121 from 100 balls, including 11 fours and three sixes. He provided the foundation for Sri Lanka's total after Kamindu Mendis had contributed an important 54 from 65 balls.
The pair added 137 for the third wicket, putting Sri Lanka in a strong position after Pathum Nissanka had fallen for 25 and Kamindu departed with the score on 182.
Mendis continued to attack after reaching his century, helping Sri Lanka build the kind of total that would put England under serious pressure.
Gus Atkinson was England's most successful bowler, taking two wickets for 55 runs, while Rehan Ahmed also claimed two wickets.
England's reply began strongly, with Tom Banton leading the charge. The opener made 73 from 62 balls, while Jos Buttler kept the hosts in the contest with an aggressive 69 from 49 deliveries.
England reached 143 before losing Ben Duckett, but wickets at regular intervals prevented them from turning their promising start into a match-winning partnership.
Joe Root made 15 before falling at 203 for four, while Harry Brook was dismissed for four. England were still within reach when Buttler was at the crease, but his dismissal for 69 left the hosts needing a late surge.
Dunith Wellalage then played a crucial role for Sri Lanka, taking three wickets for 49 runs. Eshan Malinga also made an important contribution with two wickets as Sri Lanka kept England under pressure during the closing overs.
England needed 17 runs from the final over but could not complete the chase, ending on 305 for nine.
The result gives Sri Lanka a significant response after England won the opening ODI by 89 runs in Durham. The three-match series is now level at 1-1, with the final ODI to decide the series.
Anthony Gordon believes his move from Newcastle to Barcelona has changed perceptions of him, with the England international feeling his technical and tactical qualities are finally receiving greater recognition.
Jamie Gittens has withdrawn from the England Under-21 squad after suffering an ankle injury during the 4-2 European Championship qualifying win over Kazakhstan.
Tom Banton marked his return to England's one-day side in style, smashing his maiden international century as England thrashed Sri Lanka by 223 runs at The Oval.
David Miller produced a career-best innings and Temba Bavuma added a century while the spinners dominated for the Proteas as they secured a second consecutive victory over Australia, winning the second ODI by 32 runs at the Wanderers in Johannesburg.
