The latest developments in a high-profile criminal probe by  US special counsel John Durham show the extent to which the world’s internet traffic is being monitored by a coterie of network researchers and security experts inside and outside the US government. The monitoring is made possible by little-scrutinized partnerships, both informal and formal, among cybersecurity companies, telecommunications providers and government agencies.

The U.S. government is obtaining bulk data about network usage, according to federal contracting documents and people familiar with the matter, and has fought disclosure about such activities. Academic and independent researchers are sometimes tapped to look at data and share any findings with the government without warrants or judicial authorization…

Unlike the disclosures by former intelligence contractor Edward Snowden from nearly a decade ago, which revealed U.S. intelligence programs that relied on covert access to private data streams, the sharing of internet records highlighted by Mr. Durham’s probe concerns commercial information that is often being shared with or sold to the government in bulk. Such data sets can possess enormous intelligence value, according to current and former government officials and cybersecurity experts, especially as the power of computers to derive insights from massive data sets has grown in recent years.

Such network data can help governments and companies detect and counter cyberattacks. But that capability also has privacy implications, despite assurances from researchers that most of the data can’t be traced back to individuals or organizations.

At issue are several kinds of internet logs showing the connections between computers, typically collected on networking devices such as switches or routers. They are the rough internet equivalent of logs of phone calls—showing which computers are connecting and when, but not necessarily revealing anything about the content of the transmissions. Modern smartphones and computers generate thousands of such logs a day just by browsing the web or using consumer apps…

“A question worth asking is: Who has access to large pools of telecommunications metadata, such as DNS records, and under what circumstances can those be shared with the government?…Surveillance takes the path of least resistance…,” according to Julian Sanchez, a senior fellow at the Cato Institute.

Excerpts from Byron Tau et al., Probe Reveals Unregulated Access to Data Streams, WSJ, Feb.. 28, 2022