Late on Friday, Meta made some changes to an upcoming policy and acknowledged that the action was a response to criticism from business customers and members of the media.
“We’ve heard you and want to clarify: the crawl terms we’ve consolidated only apply to our ads and business products, and we’ve clarified that in the terms themselves,” said Friday’s statement to the Mercury News.
The policy as set out before the change made clear that businesses using Meta’s products are subject to a regime in which Meta can use those businesses’ website data for essentially anything, including AI training. The new terms (a preview of which is available here) appear to reduce the remarkably large umbrella under which businesses’ data can be used. The explicit use of the term “train” is no longer in the terms, and the new document emphasizes “advertising and business products.”
But – also according to the Mercury News – one of the prominent critics of the new policy, News/Media Alliance spokesperson Sam Quigley, said of the new policy: “It seems like the new language just limits what Meta allows publishers to use content for, but not their ability to crawl and get the content in the first place.”
Quigley added, according to the Mercury News, “They may have taken training out of the listed applications, but with it not being expressly ruled out, it’s still potentially on the table.”
Quigley also points out that Meta’s statement says there will be no changes to how it uses web crawlers, but the company doesn’t explicitly state this in the new terms of service.
The question is whether Meta fully respects the wishes set out in a site’s robot.txt file – best seen as a red light or a green light for web crawler bots, which have been respected as the rule of the road on the internet since ancient times. If Meta actually plans to honor robots.txt requests, the thinking goes, it shouldn’t be a problem to explicitly do so in its terms of service.
The ambiguity comes at a sensitive time for robots.txt, as the norms surrounding old-fashioned web crawlers are shaking as companies excited by their AI ambitions appear to be redefining what a bot – broadly defined – is allowed to do online.
The term preview refers to data collection by “automated software (such as webscrapers, bots, robots, spiders, crawlers, and user agents).” Seeing such verbiage at a time when agents are becoming more common may not be all that reassuring, even the company in question says it will not change its use of web crawlers, which are a 90s invention used in conjunction with search engines.
Agents get their own separate rules, according to Google, and Google says it will respect those. However, a common sentiment expressed online holds that as companies like these agents roll out products, we should all expect a day to come soon when agents ignore robots.txt and simply do whatever they want. To be clear, nothing of the sort is previewed in the Meta Terms of Service.