Rather than the search giant's minimalist front page, employees at BreakingPoint were frequently seeing four frames in their browser: one containing Google's site and three others that jumped to affiliate advertising sites. iGoogle, the search giant's personalized service had apparently disappeared along with other popular pages at the search site. Reports from employees' families suggested that the issue not only affected BreakingPoint but many home users in the Austin, Texas area as well, the researcher said.

"Friends and family ... were seeing the same thing on their home DSL links and asking employees if they knew what as going on," said Moore, director of security research for BreakingPoint and the founder of the well-known Metasploit Project.

To Moore, who creates exploit code as part of Metasploit, it was clear what was happening: Google had not been hacked -- somehow, BreakingPoint's domain-name service (DNS) servers were returning a fake entry for Google and routing requests for the search engine's pages to a fake site set up by a scammer trying to profit from the attack.

A quick investigation confirmed that one of the BreakingPoint's two name servers had used an AT&T computer as a forwarder, asking it for domain information, but the entry held by the AT&T server for google.com had been poisoned with the address of the attacker's Internet host. From BreakingPoint's perspective, about half the time, an employee's browser would get addresses from the AT&T server and be sent to a spoofed Internet site.

"Once everyone got to work and started noticing it, we investigated, identified the poisoned cache server, changed our upstream forwarder, and contacted AT&T," Moore said.

The attack is the latest fallout from the controversial partial disclosure of a major security issue in the domain-name service (DNS) system earlier this month. On July 8, security researcher Dan Kaminsky, along with software makers and network-infrastructure providers, announced that they had coordinated a patch for serious issues in the way domain-name lookups were handled. Kaminsky's attack put a new spin on a well-known issue in the domain name system: spoofing domain names by poisoning the DNS cache. For thirteen days, details of the flaw were a matter of speculation, until a series of escalating disclosures painted a detailed portrait of Kaminsky's proposed attack last Monday.

Within 48 hours of the details being released, Moore and another programmer created Metasploit exploit modules to turn the theoretical attack into a serious worry for many system administrators.

Given Moore's role in developing the exploit to take advantage of Kaminsky's findings, the latest attack is ironic. However, it was also limited, Moore said. A check of AT&T's other local domain-domain servers -- more than 30 -- showed that they were not poisoned. AT&T declined to comment on the issue, but sent SecurityFocus a general statement on its response to the the domain-name service (DNS) flaw released.

"AT&T employs best practices in the management of its DNS infrastructure," the company said in the statement. "Upon learning of the recent vulnerability and patches available to defend against it, AT&T immediately obtained the patches and began testing and certifying them for production use. Having completed that certification, AT&T is now expediting the deployment across their entire production infrastructure."

While the attack witnessed by BreakingPoint Systems appears to be the only DNS cache that has been confirmed poisoned, the telltale signs of attacks have been witnessed by many other network administrators.

One submitter to the mailing list for GMAME, an open-source content management system for newsgroups, showed a snippet of a log file that appeared to indicate that an attacker had tried to poison his server's entries for eBay, Microsoft, Google, Facebook and other popular online destinations.

Data collected by security firm Arbor Networks showed a massive increase in domain-name lookups since the July 8 DNS-flaw announcement. The data, however, is less a shadow of attacks on the infrastructure and more a measure of the worry of security-conscious users checking the patch level of their name severs, said Danny MacPhearson, CSO of Arbor Networks.

"I can't verify that any of the traffic was malicious," MacPhearson said. "The traffic we were seeing could have all been customers verifying that their server was patched or not."

Many security experts and bloggers have criticized major Internet service providers for moving too slow to fix the problem. Neal Krawetz, principal researcher for Hacker Factor Solutions, scanned DNS servers for the degree of randomness in their lookup requests -- an indication of the level of vulnerability of the servers -- and found major providers susceptible to attack. Among the 28 servers that appeared to be vulnerable on July 21 were those operated by Level3, Verizon, and Adelphia.

"Some of the ISPs have handled (patching) very well; others have surprised me with how poorly they have handled it," he said.

At present, ten servers continue to appear to vulnerable, Krawetz said.

Yet, not everyone believes that Internet service providers' plodding pace is an indication that the companies are taking the DNS issue lightly. After all, the ISPs are adding a new software component to their networks, an act that should require some consideration, Paul Mockapetris, chief scientist for Internet infrastructure provider Nominum, said in an e-mail interview with SecurityFocus.

"Nobody wants to install software that doesn't have a track record," Mockapetris said. "This attack is exceptional, so it justifies an exceptional response. But the fact that I make that judgement doesn't mean that every ISP agrees."

Being leery of patches is just smart business, he said. This week, for example, Cisco released version 2.0 of its advisory on the DNS patch warning of issues that can affect customers that rely on its products' port-address translation (PAT).

"Telecom history is loaded with updates that brought down networks," Mockapetris said. "It's a judgement call that's hard to make and can have huge consequences."

Meanwhile, AT&T and other Internet service providers continue to patch.

If you have tips or insights on this topic, please contact SecurityFocus.