Deepfake democracy: Behind the AI trickery shaping India’s 2024 election

New Delhi, India — As voters queued up early morning on November 30 last year to vote in legislative elections to choose the next government of the southern Indian state of Telangana, a seven-second clip started going viral on social media.

Posted on X by the Congress party, which is in opposition nationally, and was in the state at the time, it showed KT Rama Rao, a leader of the Bharat Rashtra Samiti that was ruling the state, calling on people to vote in favour of the Congress.

The Congress shared it widely on a range of WhatsApp groups “operated unofficially” by the party, according to a senior leader who requested anonymity. It eventually ended up on the official X account of the party, viewed more than 500,000 times.

It was fake.

“Of course, it was AI-generated though it looks completely real,” the Congress party leader told Al Jazeera. “But a normal voter would not be able to distinguish; voting had started [when the video was posted] and there was no time for [the opposition campaign] to control the damage.”

The astutely timed deepfake was a marker of the flood of AI-generated, or manipulated, media that marred a series of elections in India’s states in recent months, and that’s now threatening to fundamentally shape the country’s coming general elections.

Between March and May, India’s nearly one billion voters will pick their next national government in the world’s, and history’s, biggest elections. The threats posed by deceptive AI-generated media caught the world’s attention when faked sexually explicit images of the artist Taylor Swift appeared on social media platforms in January. In November, Ashwini Vaishnaw, India’s information technology minister, called deepfakes a “threat to democracy” and Prime Minister Narendra Modi has echoed those concerns.

But with the increased availability of handy artificial intelligence tools, teams across India’s political parties, including Modi’s Bharatiya Janata Party and the Congress, are deploying deepfakes to influence voters, managers of nearly 40 recent campaigns told Al Jazeera. While several AI tools used to generate deepfakes are free, others are available on subscription for as little as 10 cents per video.

‘Creating perception’

The BJP, arguably India’s most technologically sophisticated party, has been at the forefront of using illusions for campaigning. As far back as 2012, the party used 3D hologram projections of Modi so that he could simultaneously “campaign” in dozens of places at the same time. The strategy was deployed widely during the 2014 general elections that brought Modi to power.

There was little deception involved there, but in February 2020, Manoj Tiwari, a BJP member of parliament, became among the world’s first to use deepfakes for campaigning. In three videos, Tiwari addressed voters in Delhi ahead of the capital’s legislative assembly elections in Hindi, Haryanvi and English – reaching three distinct audiences in the multicultural city. Only the Hindi video was authentic: The other two were deepfakes, where AI was used to generate his voice and words and alter his expressions and lip movement to make it almost impossible to detect, just on viewing, that they were not genuine.

In recent months, the Dravida Munnetra Kazhagam (DMK), which rules the southern state of Tamil Nadu, has used AI to resurrect its iconic leader M Karunanidhi from the dead, using lifelike videos of the former movie writer and veteran politician at campaign events.

Now, consultants and campaign managers say the 2024 elections could turbocharge the use of deepfakes even further.

“Politics is about creating perception; with AI tools [of voice and video modulation] and a click, you can turn the perception on its head in a minute,” said Arun Reddy, the national coordinator for social media at the Congress, who oversaw the party’s tech-savvy Telangana election. He added that the team was bursting with ideas to incorporate AI in campaigning, but that they didn’t have enough “trained people” to execute them all.

Reddy is strengthening his team – as are other parties.

“AI will have a resounding effect in creating the narrative,” Reddy told Al Jazeera. “The political AI-manipulated content will increase multifold, much more than what it ever was.”

‘Campaigns are getting weirder’

From the desert town of Pushkar in western India, 30-year-old Divyendra Singh Jadoun runs an AI startup, The Indian Deepfaker. Launched in October 2020, his company cloned the voice of Rajasthan state’s Congress chief ministerial candidate Ashok Gehlot for his team to send personalised messages on WhatsApp, addressing each voter by their name, during November assembly elections. The Indian Deepfaker is currently working with the team of Sikkim’s Chief Minister Prem Singh Tamang for holograms during upcoming campaigns. Sikkim is one of India’s smallest states in the northeast, perched on the Himalayas between India, Bhutan and China.

That’s the clean, official work, he said. But in recent months, he has been swamped by what he describes as “unethical requests” from political campaigns. “The political parties reach out indirectly via international numbers on WhatsApp, burner handles on Instagram, or connect on Telegram,” Jadoun told Al Jazeera in a phone interview.

In the November election, his company denied more than 50 such requests, he said, where potential clients wanted videos and audio altered to target political opponents, including with pornography. As a startup, Jadoun said his company is particularly careful to avoid any legal trouble. “And it is a very unethical use of AI,” he added. “But I know many people who are doing it for very low prices and are readily available now.”

During the election campaigns for the state legislatures of Madhya Pradesh in central India and Rajasthan in the west last November, police registered multiple cases for deepfake videos targeting senior politicians including Modi, Shivraj Singh Chauhan, Kailash Vijayvargia (all BJP) and Kamal Nath (Congress). The deepfake content production is often outsourced to private consulting firms, which rely on social media networks for distribution, spearheaded by WhatsApp.

A political consultant who requested anonymity told Al Jazeera that numbers of ordinary citizens with no public profile are registered on WhatsApp and used for the campaigns to make it harder for anyone to directly trace them back to parties, candidates, consultants and AI firms.

This consultant ran six campaigns in assembly elections last year for both the BJP and Congress. “In Rajasthan, we were using phone numbers of construction labourers to run our network on WhatsApp,” they said, “where deepfakes were primarily circulated.”

Meanwhile, AI-manipulated audios are particularly valuable tools in smaller constituencies, “targeting candidates with forged call recordings about arranging ‘black money’ for elections or threatening someone to buy votes,” the consultant said, whose own candidate was targeted with one such recording. The recordings are generally masked with candidates’ voices to cast them as evidence of corruption.

“Manipulating voters by AI is not being considered a sin by any party,” they added. “It is just a part of the campaign strategy.”

India has 760 million internet users – more than 50 percent of the population – behind only China.

Among all the requests, one from a constituency in southern Rajasthan stood out to Jadoun. Ahead of the state election in November, the caller requested that Jadoun alter a problematic but authentic video of their candidate – ​​whose party he did not disclose – to make a realistic deepfake. The aim: to claim that the original was a deepfake, and the deepfake the original.

“The opposition had a troubling video of their candidate and they wanted to spread it quicker on social media to claim it is a deepfake,” he said, bursting into awkward laughter. “Political campaigns are getting weirder.”

Threats to election integrity

Indian laws currently do not define “deepfakes” clearly, said Anushka Jain, a policy researcher at Goa-based Digital Futures Lab. The police have been using laws against defamation, fake news or violation of a person’s modesty, combined with the Information Technology Act, to try and tackle individual cases. But often, they’re playing whack-a-mole.

“The police are prosecuting on the effect of the deepfake and not because it is a deepfake itself,” she said.

Analysts say that the Election Commission of India (ECI), an autonomous body that conducts polling, needs to catch up with the shifting nature of political campaigns.

In the days leading up to the voting in Telangana state elections last year, ruling Bharat Rashtra Samithi party leaders repeatedly warned their followers on social media to stay alert against deepfakes deployed by the Congress party. They also appealed to the ECI against the deepfake clip that the Congress shared on the morning of the vote.

But the video remains online and the party never received notice from the ECI, two Congress leaders aware of the issue told Al Jazeera.

Al Jazeera has sought comments from the ECI but is yet to receive a response.

“Even if one person is misled into believing something and that changes his mind, it vitiates the purity of the election process,” said SY Quraishi, former chief election commissioner of India. “Deepfakes have made the problem of rumour-mongering during the polls graver by a thousand times.”

Quraishi said that deepfakes need to be moderated in real time to minimise the damage they can cause to Indian democracy.

“The ECI needs to take action before the damage is done,” he said. “They need to be a lot more prompt.”

‘Truth is out of reach’

The Indian government has been pressing major tech companies, including Google and Meta, to actively make efforts to moderate deepfakes on their platforms. IT minister Rajeev Chandrasekhar has met officials from these firms as part of deliberations over the threats posed by deepfakes.

By asking the tech sector to take the lead, the government escapes any criticism that it is trying to selectively censor selective deepfakes, or that it is trying to crack down on emerging AI technologies more broadly.

But by passing the buck to private companies, the government is raising questions about the sincerity of its intent to regulate manipulative content, said Prateek Waghre, the executive director of India’s Internet Freedom Foundation, a leading New Delhi-based tech policy think-tank. “It is almost wishful thinking,” he said.

Arguing that the tech companies have not been able to deal with the existing problems with content moderation, Waghre said that “the rise of AI now” has compounded challenges. And the current approach to content moderation ignores what’s really at the heart of the problem, he said.

“You are not solving the problem,” he said. “The design [of algorithms] is just flawed.”

On February 16, major tech companies signed an accord at the Munich Security Conference to voluntarily adopt “reasonable precautions” to prevent artificial intelligence tools from being used to disrupt democratic elections around the world. But the vaguely worded pact left many advocates and critics disappointed.

YouTube has announced that it will enable people to request the removal of AI-generated or altered content that simulates an identifiable person, including their face or voice, using its privacy request process.

“I’m not very optimistic about the platform’s capabilities to detect deepfake,” said Ravi Iyer, managing director of the Neely Center for Ethical Leadership and Decision Making at the University of Southern California’s Marshall School of Business. “With low digital literacy and rising consumption of videos, this poses a grave risk to India’s election integrity.”

Identifying every AI-manipulated media is not a reasonable task, Iyer said, so companies need to redesign algorithms that don’t promote polarising content. “Companies are the ones with the money and resources, they need to take reasonable steps to tackle the rise of deepfakes,” he said.

The Internet Freedom Foundation has published an open letter urging electoral candidates and parties to voluntarily refrain from using deepfake technology ahead of the national elections. Waghre isn’t confident that many will bite, but he said it’s worth a try.

Meanwhile, political campaigns are bolstering their AI armouries – and some, like Reddy, the national coordinator for social media at Congress, concede that the future looks dark.

“Most people using AI are out there to distort the facts. They want to create a perception that’s not based on truth,” said Reddy. “Combine the penetration of social media in India with the rise of AI, the truth will be out of reach of people in the elections now.”