In a recent interview with Webpro news Matt Cutts suggests that site load speed will have an effect on how a site is ranked, rewarding the faster sites over those that are less fast or just downright slow.

See: http://videos.webpronews.com/2009/11/13/matt-cutts-interview/

The good news is that the Google Caffeine launch will not take place early in 2010 to keep Webmasters happy over the holidays.

It seems to me that Barry Schwartz from searchengineland.com has hit the nail on the head in this article: http://searchengineland.com/site-speed-googles-next-ranking-factor-29793

Conclusions
This change is likely to be a very small part of the Google algorithm and unless your site is really slow and surrounded by other competing sites that load very quickly I don’t believe that most people will see a change in how their website is ranked.  The message is if it isn’t broke don’t fix it.

If you do want to improve your site load speed, check out the following sites:

http://www.webpagetest.org/

http://code.google.com/speed/ or http://code.google.com/speed/page-speed/

The following video explains how to improve your website performance with Page Speed: