University of Florida Search - Information for Webmasters

Information for Webmasters

The University of Florida uses multiple Google Search Appliances (GSAs) to run its Web site search. These appliances handle the crawling and indexing of up to two million publicly available Web pages under the ufl.edu domain.

Crawling

Currently, Web Administration maintains three Google Search Appliances on campus. You may see hits to your Web site from one or more of the following IP addresses:

192.5.2.196
192.5.2.197
128.227.217.63

If you feel that any of these GSAs is placing unreasonable load on your server, please email webadmin-search-l@lists.ufl.edu with the following information:

The URL of your Web site (e.g. http://www.ufl.edu/)
The IP addresses of the offending appliances (one or more from the list above)
A sample of your access logs illustrating the problem (including the date, time, and timezone of each request)

UF's Google Search Appliances follow the Robots Exclusion Protocol, commonly known as robots.txt. To prevent the GSAs from crawling some or all of your Web site, use the user agent of ufl-gsa-crawler. For example:

# Prevent UF's Google Search Appliances from crawling any of your Web site
User-agent: ufl-gsa-crawler
Disallow: /

# Prevent UF's Google Search Appliances from crawling your images directory
User-agent: ufl-gsa-crawler
Disallow: /images/

Note that there may be some delay between when you create this file and when UF's GSAs pick it up.

Indexing

UF's Google Search Appliances typically crawl and index the UF Web presence within about a week. If you have recently released a new site, you may notice some broken links in search results until your site is crawled again. In the meantime, Web Administration recommends that you configure your Web server to issue redirects from the old URLs to the new ones.

Additionally, we highly recommend using Google Webmaster Tools on your Web site. Pay close attention to any crawl errors and suggestions for improving your site.

Search