Free speech will be a big concern for voters at the next general election, with the government’s current approach on the issue perceived by many to be bad for Britain, polling suggests.

Three in five people said a political party’s position on free speech would affect how they voted at a general election. The human rights organisation Liberty, which commissioned the polling from More in Common, claimed the results showed that Labour’s failure to protect free speech could cost it votes.

A significant proportion of those polled felt certain policies and rhetoric by politicians had been detrimental for free speech. For example, two in five said arrests of people for holding signs in support of the banned group Palestine Action had had a negative effect on freedom of speech. That compared with one in five who said it was positive.

Only about one in seven thought Andy Burnham cared more about protecting free speech than his predecessor, Keir Starmer.

Akiko Hart, the director of Liberty, said: “The ability to express ourselves freely makes us all who we are, and free speech is of vital importance to the British public. These findings should serve as a stark wake-up call for the government, which has lost the public’s confidence on protecting this fundamental human right.

“We’re fortunate to live in a nation shaped by many identities and perspectives, which means differences of opinion and sometimes uncomfortable views that could upset or offend others. This requires the government to adopt a consistent, non-partisan approach to safeguard everyone’s right to speak freely while protecting people from discrimination.”

The polling, released ahead of a joint Liberty and More in Common panel event at the Labour party conference on Monday, also found that more than a third of people believe politicians calling for phrases to be banned at protests is bad for free speech, compared with just over a quarter who think it is positive.

When asked about police getting greater powers to restrict protests or marches, a similar proportion – a third – believed it was good as thought it was bad. More than half of people polled said calling protests “hate marches” or “thuggery” had been bad.

Starmer called for “tougher action” against people who used phrases such as “globalise the intifada” – intifada is an Arabic word that translates as uprising or “shaking off” – at pro-Palestinian demonstrations, and there have been arrests.

The Crime and Policing Act, which received royal assent in April under Starmer, gave police increased powers to restrict protests, including requiring them to consider the “cumulative impact” of repeated demonstrations in the same area when imposing conditions. Many civil society groups were critical of the measure.

Since Palestine Action became the first direct action protest group to be proscribed, in July last year, more than 3,600 people are believed to have been arrested for showing support for the organisation, most for holding placards saying “I oppose genocide, I support Palestine Action”. Arrests have continued since Burnham entered office.

The UK supreme court will hear an appeal against the ban in November, having decided it is arguable that proscription represents a disproportionate interference with articles 10 and 11 of the European convention on human rights, which protect the rights of freedom of speech and protest respectively.

Both left and right have complained about so-called “cancel culture” in recent years, with each side portraying the other as the chief culprit.

Hart said: “The prime minister must show urgent leadership on an issue that is dividing the country, and change course away from heavier restrictions and towards a principled defence of free speech.”

A Home Office spokesperson said: “Freedom of speech and peaceful protest are fundamental rights that this government will always protect. But there is a clear difference between expressing a view and engaging in unlawful behaviour – such as inciting hatred and violence or supporting proscribed organisations. It is right that as a society and a government, we draw that line clearly.”

The Labour party was also approached for comment.

More in Common polled a nationally representative sample of 2,017 people in Great Britain between 11 and 15 September.