More than half of voters think the Conservative party should give up Frank Hester’s £10m donation because of his offensive remarks, new polling suggests.

A snap poll, conducted after the Guardian revealed offensive comments the Tory donor made about Diane Abbott, has found the public want to see consequences for Hester’s words.

In a survey of 1,001 people, more than 60% agreed with the statement that “mainstream political parties should not accept donations from people who are found to have made racist or offensive remarks”.

The poll, carried out by JL Partners for the campaign group 38 Degrees on Tuesday, shows 52% of people say the Tories should not keep the £10m, with 73% of them saying the money should be handed to a charity focused on tackling racism. Less than a quarter (24%) think the money should be returned to Hester.

Hester said in a 2019 meeting that he did not hate all black women. But he said seeing Diane Abbott, who is Britain’s longest-serving black MP, on TV meant “you just want to hate all black women because she’s there”.

Rishi Sunak’s press secretary suggested Hester was not a racist, telling reporters: “I’ll point you to his own words where he talks about racism is a poison, there’s no place for it in public life.” She added: “You can look at his record, his actions and also his words yesterday.”

More than half of people believe the Conservative party has a “problem of racism” among its MPs, donors and members. Among Tory 2019 voters, 39% think there is a racism problem within the party.

Of all those polled, only 11% believe the prime minister has succeeded in his aim of making the country more united, and 36% of the public believe the Tory party “takes allegations of racism seriously”.

Our morning email breaks down the key stories of the day, telling you what’s happening and why it matters

after newsletter promotion

4:11

'Racist and misogynist': Starmer challenges Sunak over Tory donor's comments – video

Andy Street, the Conservative mayor of the West Midlands, added to the pressure being applied by Labour and the Liberal Democrats by saying the prime minister should “think about the company” he kept and return the donation. The chancellor, Jeremy Hunt, meanwhile said Hester should not be “cancelled” for remarks he “made in the past”.

Matthew McGregor, the chief executive at 38 Degrees, said: “If donors who make racist statements don’t face real consequences, the Conservative party is showing us that it can’t deal with racism and sexism in its own ranks. And as other recent headlines show, this isn’t an isolated incident. Rishi Sunak is presiding over a country that is becoming more and more divided, and a party that can’t control racist members or donors – and British voters can clearly see that.”

A statement from TPP said Hester “accepts that he was rude about Diane Abbott in a private meeting several years ago but his criticism had nothing to do with her gender nor colour of skin”. The statement said Hester had tried to apologise directly for the “hurt he has caused” and that he was “deeply sorry for his remarks”. The statement also said he abhorred racism.