Boris Johnson has urged Conservative MPs plotting to oust him not to focus on the issues he has “stuffed up” after his authority was diminished by a double by-election defeat.

The Prime Minister insisted questions over his leadership were now settled after the loss of Wakefield and former stronghold Tiverton and Honiton triggered a Cabinet resignation.

He insisted the “endless churn” of allegations is “driving people nuts”, as he pushed on with his Rwanda trip despite suggestions further ministerial resignations could follow.

Mr Johnson is clinging on to his leadership after Oliver Dowden resigned as Tory party co-chairman, saying he and Conservative supporters are “distressed and disappointed by recent events” and telling the Prime Minister that “someone must take responsibility”.

Former Conservative leader Michael Howard also urged the Prime Minister to resign for the good of the party and the nation, and he called on the Cabinet to consider resigning to force him out.

But Mr Johnson was in a buoyant mood as he said: “I love my colleagues and of course I would urge them respectfully: golden rule of politics, the more we focus on Westminster politics the more irritating it is to voters.”

He told journalists at the British high commissioner’s residence in Kigali that it is “reasonable” to suggest he might need to change to win back voters’ trust.

“But the answer is always to remember, it’s not about me, it’s about them,” Mr Johnson said.

“Because what’s driving people nuts is this endless churn of stuff about things that I’m meant to have stuffed up or whatever about my colleagues, their views of me, my character, the leadership, Tory blah blah…”

But there are suggestions of a challenge to change the rules of the 1922 Committee of Conservative MPs in order to allow another vote of confidence in Mr Johnson within the next year.

Asked if he believes questions over his leadership are settled, Mr Johnson replied: “Yes.”

Instead of partygate and other scandals, Mr Johnson insisted the public want to hear about the cost of living and economic plans.

“That’s the change that I think will be most welcome on the last few months,” he said. “That’s, I think, what people will see.

Conservative veteran Sir Geoffrey Clifton-Brown, who retained his Cotswolds seat with a majority of 20,000 at the last vote, is among the Tories to have expressed fears they could lose their jobs at the next general election.

In West Yorkshire, Labour seized back Wakefield with a majority of 4,925 on a swing of 12.7% from the Tories.

Wakefield was one of the so-called red wall seats won by the Tories in the 2019 general election after being held by Labour since the 1930s.

Foreign Secretary Liz Truss argued that the dismal performance for the Conservatives in Thursday’s by-elections will not be the predictor of the next general election.

Speaking to reporters in Rwanda, she said: “The reality is that incumbent governments often lose by-elections and often people want to send a message in a by-election to raise concerns with the Government.

“But that doesn’t make by-election results the predictor of election outcomes, it hasn’t been the predictor in the past and I don’t believe it will be the predictor of the next general election.”