May 28th, 2006Bandwidth problems
I tried to access my website this morning but I got a “Bandwidth Exceeded” error.
My website is now up again thanks to James and I had a look at my site stats. My bandwidth usage for this month is 10GB. For March and April it was 9GB. It used to be 3 to 5GB before. I don’t understand what happened during the last three months. I’ll have to look into the matter and find ways to prevent the same thing from happening again.
. I had the choice either to have the...
Hmm that sounds strange.. perhaps someone is hotlinking some of your files!?
Are you killing the spiders before they hit your site with a robots.txt file?
I know google nailed my bandwidth suddenly, before I started killing them.
I wanted to add a little more, but hit the submit button… grrr.
You know my site, I often have streaming media and so forth. Unless you are getting tens of thousands of visits a day, your bandwidth should not be anywhere NEAR 10G in a month.
My account only allows for 2GB a month… and I have only blown that once, when Google was nailing me.
Obviously you have a lot more friends than I do =) but I still get nearly 1,000 unique hits a month.
Images and text should not put you past 10G.
Like what kyle might have said,some bandwidth eating robots or websites might be the culprit.Have you tried downloading a bandwidth monitor program to keep a check on your website?This is just a suggestion here though
Hmm.
It could be robots or spiders spidering your site. One of the more potent ones to look out for is the Gigablast spider. You have quite a lot of weblog entries and hence, I feel that spidering your entire site uses quite a lot of bandwidth, especially when the spiders scour your site regularly.
As mentioned in one of the above comments, you can use the robots.txt file. Although I must say it isn’t really effective because certain spiders/robots tend to ignore the robots.txt file. The best bet is to block them altogether, by preventing all search engines from listing your site. That’s the most extreme (but safest) way out.
Maybe you can limit the number of posts that Wordpress shows.
Or maybe you can somehow check if it’s also because of spammers trying to spam (did Wordpress accumulate a lot of spam in the database?) Good luck with it anyway.
Yah I think it’s spiders or robots … Mine was also recently exceeded and I also got more spam comments from my blog.
Brenda: Crawlers load only text data from websites, i doubt those bots are able to crawl this blog so many times so as to generate such bandwidth waste.
Aline: Most search engines have FAQ on their websites on how to regulate the frequency of visit by their bots. Google, for example, launched Sitemaps (BETA) to help webmasters, especially bloggers with the crawling of their webpages. Instead of crawling every pages of your blogs all over again on each visit, google gives crawling priority to most recent pages. In some cases, it helps saving bandwidth yet having your contents indexed and appear on google more quickly.
> Reference: http://www.google.com/webmasters/sitemaps/siteoverview?hl=en
But search engines are not the only reason of high bandwidth consumption. You could make your blog more efficient.
- Turn on HTTP Gzip compression and caching system codes & RSS feeds. (i assume your blog system offer those features).
- Add .htaccess (can be done via cpanel) to prevent image hotlinking.
- Large image can also be a problem, you would prefer use JPG for pictures and PNG to replace GIF as it is said it can reduce file size by 20%. (If you intend to post hi-res images, you could always upload it on free image hosting servers - less control but saves a great deal).
- Use CSS (I know your blog is CSS compliant but just for the record: ESPN saved 2TB of bandwidth by switching to CSS).
- You could also make articles shorter, post a summary or preview on frontpage but show only the full entry when user asks for it (i.e: clicking on more).
- Finally, the free browser that you all like could cost you more than you ever guess. Read how you can disable FasterFox (a firefox module) on your blog.
> http://www.skattertech.com/2006/02/how-to-block-fasterfox-requests/
/There will always be misc users who will try to waste your bandwidth.
James, jae.hk
This layout is CSS only already…. no tables used.
I only use the ‘More’ thing when I post a lot of pics.
Gzip compression was already on.
I think it’s because of hotlinking. Many of my images get hotlinked,esp used on forums with many users. I just removed myself from google image search and I’m changing the file names to Creole instead of English.
I had tried the hotlinking prevention in the past (from cpanel) and it caused my wordpress pages option to ‘disappear’ (lack of better word to describe it). I’ll have another try soon.
Thanks for the Fasterfox thing link. I’ll add those two lines to my robots file.
thanks to everyone for the advice. I’ll look into the matter.