Conservative MPs have voiced fears that they will be next to be voted out after Boris Johnson led the party to a double by-election defeat.

The Prime Minister vowed to “keep going” after his authority was dealt a series of blows, including the resignation of Cabinet minister Oliver Dowden on Friday.

Speaking 4,000 miles away at a Commonwealth summit in Rwanda, Mr Johnson vowed to “listen” to voters after losing the former Tory stronghold of Tiverton and Honiton to the Liberal Democrats and Wakefield to Labour.

Mr Dowden quit as Conservative Party co-chairman, saying he and Tory supporters are “distressed and disappointed by recent events” and telling Mr Johnson that “someone must take responsibility”.

With 324 Tory MPs elected in 2019 with smaller majorities than in the Tiverton and Honiton constituency, MPs including Conservative grandee Sir Geoffrey Clifton-Brown expressed their concerns that they could lose their seats at the next general election.

The Prime Minister spoke to Chancellor Rishi Sunak by phone for his daily meeting after receiving a warning call from Mr Dowden following an early-morning swim at his hotel.

Speaking to broadcasters, Mr Johnson said he would take responsibility, but insisted the cost-of-living crisis is the most important issue for voters and it is “true that, in mid-term, governments post-war lose by-elections”.

“It’s absolutely true we’ve had some tough by-election results. They’ve been, I think, a reflection of a lot of things, but we’ve got to recognise voters are going through a tough time at the moment,” he said at the conference centre in Kigali.

“I think, as a Government, I’ve got to listen to what people are saying – in particular to the difficulties people are facing over the cost of living, which, I think, for most people is the number one issue.

“We’ve got to recognise there is more we’ve got to do and we certainly will – we will keep going, addressing the concerns of people until we get through this patch.”

In an interview with Channel 4 News, Mr Johnson added: “I, of course, take responsibility for the electoral performance of the Government.”

In the rural Devon constituency of Tiverton and Honiton, the Lib Dems overturned a 24,000 Tory majority to win, while Labour reclaimed Wakefield.

The contests, triggered by the resignation of two disgraced Tories, offered voters the chance to give their verdict on the Prime Minister just weeks after 148 of his MPs cast their ballots against him in a confidence vote.

Mr Dowden, who was due to appear on the morning media round for the Government before resigning, said in his letter to Mr Johnson that the by-elections “are the latest in a run of very poor results for our party”.

“Our supporters are distressed and disappointed by recent events, and I share their feelings,” he said.

“We cannot carry on with business as usual. Somebody must take responsibility and I have concluded that, in these circumstances, it would not be right for me to remain in office.”

Sir Geoffrey retained his Cotswolds seat with a majority of 20,000 at the 2019 general election but accepted it will be a challenge to retain it next time round.

“I think, factually, if I were to run under a bus today it would be difficult to hold my seat. There’s no doubt about that,” he told BBC Radio 4’s Today programme.

Former minister Jesse Norman said Mr Johnson is insulting the electorate and making a decisive change of government at the next general election “much more likely” by prolonging “this charade”.

Veteran Tory MP and long-standing Johnson critic Sir Roger Gale said the Prime Minister has “trashed” the party’s reputation.

He told BBC Breakfast Mr Johnson is choosing to “hang on to the door handle at No 10” but “it can’t go on forever, and it certainly won’t go on until the next general election”.

A Conservative Party source said Mr Johnson was in his hotel pool by 6am Kigali time and was surprised to receive the call from Mr Dowden warning him he was about to resign.

The Prime Minister went on to hold his typical daily meeting with Mr Sunak, this time over the phone, and with chief whip Chris Heaton-Harris, the source added.

Despite the political drama, Mr Johnson was said to be planning to stay the course in Rwanda before heading to a G7 summit in Germany.

“To not be at the G7 would be an abdication of responsibility for any prime minister,” the source said.

A swing of almost 30% from the Tories to the Liberal Democrats saw Richard Foord secure a majority of 6,144 in Tiverton and Honiton.

The vote was triggered by the resignation of Neil Parish after he was caught watching pornography in Parliament.

The new Lib Dem MP used his acceptance speech to call for Mr Johnson “to go, and go now”, claiming his victory has “sent a shockwave through British politics”.

Lib Dem leader Sir Ed Davey said it is “time for Conservative MPs to finally do the right thing and sack him”.

In Wakefield, Simon Lightwood was elected with a majority of 4,925 on a swing of 12.7% from the Tories to Labour.

The previous Wakefield MP, Imran Ahmad Khan, quit after being found guilty of sexually assaulting a 15-year-old boy – a crime for which he was jailed for 18 months.

Wakefield was one of the so-called red wall seats won by the Tories in the 2019 general election after being Labour since the 1930s.

Mr Lightwood said: “The people of Wakefield have spoken on behalf of the British people.

“They have said, unreservedly ‘Boris Johnson, your contempt for this country is no longer tolerated’.”

Labour leader Sir Keir Starmer said: “Wakefield has shown the country has lost confidence in the Tories.

“This result is a clear judgment on a Conservative Party that has run out of energy and ideas. Britain deserves better.”

He said the result shows Labour “is back on the side of working people, winning seats where we lost before, and ready for government”.