Spurs have collected just two points from their opening five matches, with De Zerbi frustrated by another match in which his side struggled to maintain its level after a promising start.
"It's difficult to explain because, in my career, 14 years as a coach, I've never seen one team like us," De Zerbi said. "So fragile, so inconsistent."
Tottenham had created several chances in the first half but fell behind while Micky van de Ven was temporarily off the pitch receiving treatment following a clash of heads. De Zerbi felt his side should have coped better despite being reduced to 10 men.
Villa then doubled their lead through Nicolas Jackson, who scored from a long clearance by goalkeeper Zion Suzuki. De Zerbi was particularly unhappy with the goal after Tottenham had specifically prepared for Villa's increased use of long balls.
"We knew this season Aston Villa are playing 86% more long balls than last season and we spoke in many meetings about it," he said.
The Spurs boss also highlighted a lack of communication and leadership when games turn against his side.
"Maybe the communication between the players [is the problem], maybe the leaders have to be stronger," De Zerbi said. "We need leaders to drive and lead the team."
De Zerbi also demanded more from Mateus Fernandes and Dominic Solanke, while acknowledging that Solanke's physical condition was not yet where it needed to be.
"I like Dom as a player, and I stay with him until the end," he said. "He needs to stay better, to feel better. The physical condition is not good enough."
Tottenham eventually scored twice late on after Villa had moved 3-0 ahead, but De Zerbi remained unhappy with the manner of the defeat.
"We can't lose the game in this way," he said. "We can't lose balance just because we are losing."
Villa manager Unai Emery praised his players for showing belief and committing to the defensive side of their game as they secured their first Premier League win of the season.
"The players did well because they believed," Emery said.
Tottenham now have the international break to regroup before returning to Premier League action away to Manchester United.
Chris Hughton has backed Roberto De Zerbi to guide Tottenham through their difficult start to the season, insisting the club have the players and manager needed to turn their results around.
Granit Xhaka has withdrawn from the Switzerland squad while he faces an investigation over his use of a forged Covid-19 vaccination certificate.
Matchweek 5 triggered significant shifts across the English Premier League, leaving early-season expectations shattered just before the league paused for the extended international break.
Michael Carrick has urged Manchester United to ensure the refereeing controversy from their derby defeat to Manchester City does not define their season, while admitting the officials' apology meant little.
