Заколебешься по одному собирать. Вот все User-Agent'ы из двухнедельного лога:
-
Avant Browser (
http://www.avantbrowser.com)
DownloadData
Download Master
Gigabot/1.0
Googlebot/2.1 (+
http://www.googlebot.com/bot.html)
http://www.almaden.ibm.com/cs/crawler [c01]
ia_archiver
IpScan
Ivan Susanin validator
Microsoft Data Access Internet Publishing Provider Protocol Discovery
Microsoft Office Protocol Discovery
Modem Booster
MOT-c350M/ULS_G_09.10.1AR MIB/2.0
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Mozilla/2.0 (compatible; MSIE 3.01; Windows 95)
Mozilla/2.0 (compatible; Uralweb.ru UrlChecker)
Mozilla/4.0
Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with
http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with
http://grub.org)
Mozilla/4.0 (compatible; MS FrontPage 6.0)
Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.00; Windows 98
Mozilla/4.0 (compatible; MSIE 5.01; Windows 95)
Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FunWebProducts)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE2)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MyIE2; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 5.0; Windows 2000) Opera 5.12 [ru]
Mozilla/4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.01 [ru]
Mozilla/4.0 (compatible; MSIE 5.0; Windows 2000) Opera 6.0 [en]
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; MRA 2.55 (build 00423))
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; MyIE2)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com;
http://www.girafa.com)
Mozilla/4.0 (compatible; MSIE 5.0; Windows XP) Opera 6.05 [ru]
Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 95)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; URBrowser55)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MRA 2.55 (build 00423))
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MyIE2)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Microsoft Scheduled Cache Content Download Service
Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows ME) Opera 7.03 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.01 [en]
Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows XP) Opera 7.0 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Alexa Toolbar)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Hotbar 4.2.13.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; MyIE2)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Opera 7.11 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Opera 7.50 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Q312461; H010818)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; spie-16-82-71dc8a9b; MyIE2)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; spie-26-82-4bddba57)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows ME) Opera 7.22 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; FunWebProducts; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MyIE2)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.23 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.23 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.2.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 2.55 (build 00423))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 2.55 (build 00423); .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2 0.3)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2; ESB{02FB4055-8407-4587-8848-C30F6B6EBB09})
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2; MRA 2.5 (build 00387))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.11 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.20 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.21 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.23 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.23 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.50 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.50 [ru]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; (R1 1.3))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; spie-12-82-c097ce16)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; spie-26-82-4bddba57)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2) Opera 7.11 [en]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2) Opera 7.20 [ru]
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net;
http://www.WISEnutbot.com)
Mozilla/4.51 [ru] (Win95; I)
Mozilla/4.75 [en] (X11; U; Linux 2.2.16 i686)
Mozilla/4.76 [en] (Win98; U)
Mozilla/5.0
Mozilla/5.0 (compatible; Konqueror/3.1; Linux)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 Galeon/1.2.7 (X11; Linux i686; U;) Gecko/20030131
Mozilla/5.0 (Windows; U; Win98; ru-RU; rv:1.3b) Gecko/20030206
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4.1) Gecko/20031008
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.1) Gecko/20020823 Netscape/7.0
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7b) Gecko/20040316
Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.3b) Gecko/20030206
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0+) Gecko/20020518
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040207 Firefox/0.8
MSFrontPage/6.0
msnbot/0.11 (+
http://search.msn.com/msnbot.htm)
MSProxy/2.0
Opera/5.02 (Windows 98; U) [en]
Opera/7.11 (Windows NT 5.0; U) [ru]
Opera/7.23 (Windows 98; U) [ru]
Opera/7.23 (Windows NT 5.0; U) [ru]
Opera/7.50 (Windows NT 5.0; U) [en]
Opera/7.50 (Windows NT 5.1; U) [ru]
PHP/4.2.3
StackRambler/2.0 (MSIE incompatible)
Yandex/1.01.001 (compatible; Win16; H)
Yandex/1.01.001 (compatible; Win16; I)
Yandex/1.01.001 (compatible; Win16; M)