Three prominent Labour backbenchers have accused Keir Starmer of an “insulting” and “colonial mindset” over his reluctance to discuss reparations for the transatlantic slave trade.

Ahead of a summit of the Commonwealth heads of government in Samoa last week, the prime minister rejected calls to discuss reparations with Caribbean and African nations, with No 10 insisting it was “not on the agenda”.

Starmer said slavery was “abhorrent” but his stance was “looking forward rather than looking backwards”, adding that he wanted to focus on discussing “current future-facing challenges” at the summit instead of “very long, endless discussions about reparations on the past”.

On Sunday the Labour MP Bell Ribeiro-Addy addressed a cross-party reparations conference in London, saying it was “very insulting [to] tell people of African descent to forget and move forward”.

“Reparations are not about relitigating historic injustices, they are about remedying the deep-rooted inequalities that still shape our world today,” she said. “At a time when there is growing awareness of how racial hierarchies that endure to this day were constituted to justify the enslavement and colonisation of African peoples, state-led action on reparations is sadly lacking.”

Another Labour MP, Clive Lewis, said it was surprising Starmer had thought he could take a “colonial mindset” to the summit and “dictate what could and could not be discussed”.

Diane Abbott, the first Black woman to become an MP, co-chaired the conference in London. She said the Labour party previously had plans to establish a national reparations commission but Starmer “seems to have forgotten that”.

“Reparations isn’t about the past, it is about the here and now,” she said. “The descendants of African slaves and colonised peoples continue to suffer from the consequences generations later.

“Real reparations aren’t just about compensation, they’re a way of tackling colonialism’s damaging legacy of racism and inequality. They are about the total system change and repair needed to heal, empower and restore dignity.”

At the conclusion of the Commonwealth heads of government meeting (Chogm), the leaders resolved that “the time has come” for a conversation on reparatory justice.

A document signed by the 56 leaders, including Starmer, stated it was time for a “meaningful, truthful and respectful conversation” about justice for the transatlantic trade in enslaved Africans and chattel enslavement, with the aim of forging “a common future based on equity”.

Ribeiro-Addy, who chairs the all-party parliamentary group for Afrikan Representation, said: “I’m very proud those nations refused to be silenced.”

In a press conference after the summit, Starmer downplayed the significance of the paragraph in the document that called for a conversation about reparatory justice, telling reporters it was a small part of “quite a long communique”.

“None of the discussions have been about money. Our position is very, very clear in relation to that,” he said. He added that the issue would be revisited when a delegation of Caribbean nations visits the UK next year.

Last week, amid mounting pressure, a source in No 10 told the Guardian that the UK could support some forms of reparatory justice, such as restructuring financial institutions and providing debt relief.

“There is a general sense that these multilateral institutions give out loans to developing countries then charge large interest rates for repayments,” the source said.