Nielsen’s data-backed panel currency is ‘unstable, unpredictable’: VAB

In a report out today, the Video Advertising Bureau alleges significant discrepancies in Nielsen’s data-backed audience measurement methodology that serves as a basis for getting and selling TV ads. Nielsen has strenuously denied the allegations, calling the report “seriously flawed and manipulated,” a spokesperson said in a press release shared with Marketing Dive.

The VAB claims that Nielsen’s Big Data + Panel currency is “unstable, unpredictable and decimating demographics,” per information shared with Marketing Dive. Accredited by the Media Rating Council in January, Big Data + Panel unites the firm’s historic panel measurement approach with data from cable, satellite set-top boxes and smart TVs from across 45 million households. 

“It seems the begrudging Upfront use of Nielsen’s not-really-ready Big Data trading currency has already gone to worst-case-scenario on early returns,” said Sean Cunningham, president and CEO of VAB, in a press release. 

The VAB’s report is based on an evaluation comparing the differences between Big Data + Panel and Panel-Only audiences drawn from 4 weeks of information from 33 networks. It found between 45% and 58% of total hours analyzed had greater than a 20% audience variance between the 2 currencies, leading to what it describes as a “deep instability” within the viewership assessment of major trading demographic groups for viewers aged 25-54, 18-49 and 18-34. 

The report also spotlights issues with how NFL game audiences have been measured. NFL games accounted for 48 of the 50 top-rated programs in the course of the reporting period. VAB claims the audience must be stable: the programming is essentially the identical, week to week, and the audience is large and dependable. But as an alternative of stability, the VAB found high and irrational variability in NFL audiences, whether analyzing hour-by-hour, network-by-network or cumulatively across networks. Between 16% to 40% of hours with NFL games had double-digit audience variances — which could mean the difference of lots of of 1000’s of viewers in key demos during higher-rated games.

If the VAB’s report is correct, this could be particularly troublesome for advertisers which have flocked to football because the last bastion of huge, live audience engagement. However, Nielsen claims that the VAB pulled data for live sports without accounting for a similar games airing in numerous time zones across the U.S. Elsewhere, Nielsen claims the VAB made other unclear or incorrect assumptions about its data, at times “comparing apples to oranges.”

“From what we now have seen, the VAB incorrectly pulled our data and the bureau doesn’t know how you can do a correct rankings evaluation,” the Nielsen spokesperson said. “The VAB is wasting the money and time of its members.”

Herculean efforts

Overall, a “significant percentage” of hours analyzed by the VAB had over a 50% variance between Big Data + Panel audiences and Panel-Only audiences amongst key demographic groups, and one in ten hours had “severe” variances on the 18-34 demographic, per the report. During a press briefing discussing the report, Cunningham attributed the issues to issues with Nielsen’s Household Demographic Assessment Model.

“This HDAM thing that we complained about until we were blue within the face and did so for the last 18 months is exactly what has thrown all these demographics off the rails. It’s going to take some sort of Herculean effort to correct this. This is, straight away, a broken demographic model,” Cunningham said in the course of the briefing.

Nielsen in a July product note said that HDAM was “not driving significant differences in demographics between panel only and large data + panel services.”

The VAB is a TV promoting trade body that counts amongst its members publishers including heavyweights Disney, NBCUniversal and Warner Bros. Discovery, together with scores of media organizations and firms within the measurement and adtech space.

The organization has been a frequent critic of Nielsen and called out the measurement giant around underreporting of audiences in the course of the early pandemic period. The Media Rating Council would go on to strip Nielsen of its National TV rankings accreditation, which Nielsen regained in 2023.