How can anyone vote for someone so… fill in the blank… racist, sexist, unconstitutional, hateful, unhinged? This is the question asked frequently in the UK and here in the States, where I have spent the past three weeks trying to understand the Trump phenomenon.

Behind the question is an implied superiority; that we, the clever people, have identified the monster that is Donald Trump, but the deluded masses are too stupid to see it. But what I have found at the Trump rallies I’ve been to is not stupidity, but frustration, pain and a longing for respect.

Tucker Carlson at Madison Square Garden captured this sentiment with his usual swagger. “They tell you, the people who can actually change a flat tyre, who pay your taxes and work 40 hours a week, that you are somehow immoral. We have a message for them: you are not better than us, you are not smarter than us.”

To dismiss this as the politics of grievance is to dismiss what it feels like to be disrespected, to feel “a stranger in your own land”. To feel as though the college-educated are looking down at the non-college-educated.

Even now, after his overwhelming victory, many still fume that Trump has returned to power on the back of a pack of lies, sometimes very big lies (like he won the 2020 election). And of course that’s part of the story. But his supporters have some justification for believing that his win has in fact been forged from a powerful truth. The economy is not being run in their interests, government is not working for them, and mainstream political parties have not been up to the job in recent years.

This is what appealed to so many people including lifelong Democrats such as Bill, who was the first person I chatted to at a rally in Latrobe, just outside Pittsburgh. “I was a Democrat all my life, a local organiser. I was invited to a fundraiser, bought a new suit to look smart, turned up and listened to all the speeches. By the end of the evening, there had been a programme aimed at everyone – those on benefits, single mothers, new immigrants – but nothing of any kind at me, a dad of two children, trying to pay the mortgage, working hard to get on. I realised the Democrats were no longer for me.”

What motivated so many of them was a lack of order and control in their lives

Yes, there was a cultish feel to the mass of red Maga hats and rhythmic chants of USA, USA, and yes, a full buffet of conspiracy theories was often on the menu. But what motivated so many of them was a lack of order and control in their lives. If you don’t know who is coming across the border, you feel uneasy and at risk. If you can’t predict how much your groceries will cost week to week, you feel the pressure.

And to solve this? You need a disruptor. Someone who doesn’t go along with the stale, failed, norms of political discourse, someone from outside politics who can hack through the undergrowth even if in doing so he might offend. If Trump was polite, generous, restrained and conciliatory, his supporters would find it impossible to believe he would give the system the good shake they believe it needs.

So, Trump’s appeal is there in plain sight. It is not an aberration. It is not inexplicable. And now we know for certain, it’s not going away. The truth is the Democrats lost people – head and heart. They failed at being good technocrats (the head) with high inflation and open borders. And failed at telling a story in which struggling working families could feel seen and heard (the heart).

This is now the challenge for the Democrats in the US fighting to win back power, and Labour in the UK trying to make a success of their victory. Trump’s win could be a moment, like Margaret Thatcher’s victory in 1979, where the old rules of politics are turned on their head and where the buildings blocks of a new progressive project need to be rebuilt from first principles.

The outlines of what needs to happen will emerge. A project that is squarely back on the side of working people. Where we do the “heavy lifting” to get better and bolder policies on the cost of living, making work pay, securing our borders, providing for the aspirations of those who don’t go to university.

Analysis and opinion on the week's news and culture brought to you by the best Observer writers

after newsletter promotion

Where we understand that the way we govern is not working for too many people and needs to change fundamentally if we are to rebuild trust. Where we get to grips with a diffuse and polarised media and communicate far more cleverly. And where we tell a story about the common good, of belonging and respect, that is sufficiently hard-headed to bring people together.

Trump won because he was the better candidate with a better message. I believe both his policies and approach will not in the end work, and will probably do a lot of damage on the way. But to millions, whether we admit it or not, he offered real hope – of greater prosperity, more security and fewer wars. Many looked to him as a protector – from a world of change and from patronising elites.

We now have a choice: rage at Trump supporters – or curiosity. We can spend the coming months in fruitless intellectual contortions about whether he meets the criteria for being a fascist, or we can properly understand what has just happened and get to work deepening, widening and improving a new progressive agenda with the vim and vitality to mount a serious fightback.

Peter Hyman is a former adviser to Keir Starmer and Tony Blair, currently working on a project to rebuild trust in politics and tackle far-right populism

• 
  Do you have an opinion on the issues raised in this article? If you would like to submit a letter of up to 250 words to be considered for publication, email it to us at observer.letters@observer.co.uk