Google restricts the access of ChatGPT and Bard to publishers’ content

Since the introduction of ChatGPT in November 2022, generative AI has been a widely discussed topic. However, it has faced challenges, including plagiarism and concerns about data sources. Publishers, in particular, are apprehensive about AI tools scraping their data. Now, Google is providing publishers with the capability to oversee the data accessible to generative AI and other tools. In a blog post, Danielle Romain, Google’s VP of Trust, stated, “Web publishers have expressed the desire for more options and control regarding the use of their content in emerging generative AI applications.

New regulations for publishers

Romain revealed that Google is introducing a new level of control, referred to as Google-Extended, which web publishers can employ to decide whether their websites contribute to enhancing Bard and Vertex AI generative APIs, including upcoming model iterations that power these products. “By utilizing Google-Extended to manage content access on a website, administrators can determine whether they want to contribute to the ongoing improvement of these AI models in terms of accuracy and capabilities,” she explained. Essentially, the choice to provide data to AI models driving Bard, ChatGPT, and similar systems lies with the publishers.

Google-Extended is a standalone product token that empowers web publishers to manage whether their websites contribute to the improvement of Bard and Vertex AI generative APIs. Various companies, including Google, use web crawlers to gather data for the enhancement of their tools. For example, Google’s regular crawlers are employed to build search indices, perform crawls specific to products, and carry out analyses.

According to Google, providing tools such as Google-Extended through robots.txt is a vital step in providing transparency and control, which they believe all AI model providers should implement. Romain noted that as AI applications and tools continue to expand, web publishers will face increased challenges in managing various uses on a larger scale.