Thu. Jan 20, 2005
Verio and Comment Spam
Last month, I wrote an article entitled “MT + Comment Spam = Dead Site,” which drew attention to the issue of spammers taking down entire web servers with their digital assaults. Part of the problem was a bug in Moveable Type, and it soon got fixed. Another problem was that Google’s indexing of these spammers on our sites gave the scumbags the Page Rank boost they sought, and we just got some help there.
That leaves the spammers themselves. Almost every provider has a published Acceptable Use Policy (AUP) which bans customers from sending spam via their network. Including Verio, one of the web’s largest providers and hosts. But they’ve also currently got one of the web’s largest spammers on their network, and so far appear to have done nothing about it. Their published AUP is violated by this customer most every minute of every day, yet abuse@verio.net has been completely unresponsive.
In fact, it can be said that the spammer has been more responsive to complaints than Verio has.
Verio definitely understands the impact spam can have. Their AUP defines spam as “Sending unsolicited bulk and/or commercial messages over the Internet (known as “spamming”). It is not only harmful because of its negative impact on consumer attitudes toward Verio, but also because it can overload Verio’s network and disrupt service to Verio subscribers.”
As you’ll see below, the spam currently issuing from Verio’s server is overloading networks and disrupting service at other web hosts. One would think Verio would understand that, since they don’t like the impact spam has on their network.
In fact, last September, Verio forced its customers to use “Aggressive spam filtering that generates false positives on legitimate messages. For some reason, Verio representatives seem to think that this is OK, because they have to protect their servers.”
At about the same time, Our Spammer had already set up at 161.58.59.8 on a Verio box, and was flooding other networks. Five months ago. One would think Verio’s abuse department has complaints dating back to then.
And this month, there can be no doubt about the number of complaints or evidence accumulated. From Tim Bray:
John Sinteur traces the bad buy back to a single IP address. Arve Bersvendsen offers more research. And Anne Elisabeth discovers that those off-the-air sites aren’t staying that way.
This is big and involves a lot of domain names and a well-known IP address; wouldn’t be nice if some capable law-enforcement organization found whoever it is that’s doing it and THREW THEIR ASS IN JAIL?
It would be nice, but there’s currently no law that would allow it. The only legal recourse is the binding contract between a provider and their customer, which clearly covers a matter like this. It’s difficult to understand why Verio wouldn’t see that this is “harmful because of its negative impact on consumer attitudes toward Verio.” Due to their lack of action, it would appear we must make their own words come true.
So here’s this consumer’s opinion and advice: Do not use Verio, they harbor one of the biggest spammers on the web.
That’s not just my opinion. It’s a widespread opinion: “The hits are all fed through open proxies or zombied servers, but they all track back to 161.58.59.8 which seems to be a server colocated at Verio. Now if that server happened to spontaneously explode I suspect a lot of referral spam would stop for a while.”
You can see for yourself that Verio controls 161.58.59.8 (“hosts 2,078 websites”), and I’ve placed the traceroute and IP block info as I found it, as well as this screen shot of my referrals for last Sunday. Other than Google, the Top 20 referrers are all from this spammer. And here’s my original e-mail to abuse@verio.net, listing just a few of the offending domains.
It should also be noted, this spammer is using a zombie network or some program to rotate proxies. This would seem to fall under the portion of Verio’s AUP that bans the usage of “any software, program, product, or service that is designed to violate this AUP, which includes the facilitation of the means to spam, initiation of pinging, flooding, mailbombing, denial of service attacks, and piracy of software.”
It’s my opinion this is “multi-use spamware.” As Mike notes, “their software targets blogs and tries to leave comment spam, but in my experience it branches out and starts to referrer spam all the other pages on the site.” My Textpattern-driven site is pretty immune to comment spam that isn’t entered manually, and I think when this software fails to post a comment, it unloads the Referral Bomb.
And it’s a big one. The effect of this is not unlike a Denial of Service attack. From Paulo: “The flood of inbound traffic from this spammer’s zombie network is so heavy that it operates like a DOS attack: consuming bandwidth, sucking up server resources, and slowing — or even bringing down — the victim site.”
Bring it down, indeed. It briefly took down my site (and 600 others) at Textdrive on Tuesday. I sent my first e-mail to abuse@verio.net on Sunday morning, and since the morning they knocked my site offline, I’ve sent two more. But there’s been no response, or any visible action taken to shut down their spamming customer.
As they detail in the Wordpress Forum, this is an odd two step scam. This spammer has dozens and dozens of domains, but when you visit the link they leave behind, you find a very odd “administrative page” that says “This legend is currently under terms of service violation – Due to miss-proper use of the hosting account … ACCOUNT TERMINATED!”
Check out “Due to miss-proper use” as a unique tracking phrase via Google: currently 463 results. The first page has links to people talking about this spammer. Most of the remaining 30+ pages of returns are his spam sites.
Anne Elisabeth points out “All those ‘Account Terminated’ notices are a sham. All of those URL’s will start working one way or another after the bloggers have checked them.” And she’s right. Many of them replaced the “Account Terminated” page with a redirect to poker/drugs spam sites. I guess the idea is that a site owner first goes to the site in the referrers, sees that it’s been shutdown, and will hopefully forget about it. Next thing you know, that URL that was “no worry” leads somewhere else entirely.
And Verio knows all of this. I’ve not only sent three detail packed e-mails to abuse@verio,net, I contacted the Public Relations department. I said I’d hold this article for their official response, and submitted a series of questions on this matter. I was contacted by one of their PR representatives just before my original deadline, and while she said she was working against time zone conflicts to try and get a formal reply for me, she was unable to do so at that time. Since she’d been so responsive, I gladly extended that deadline another 24 hours, in hopes of progress.
Here’s the formal response I finally got: “Verio’s Acceptable Use Policy prohibits sending spam on Verio’s network and we take appropriate actions with any customers found to be in violation of such prohibition, up to and including termination.” There was an indication I might be sent “more information,” but for now, that’s it.
That’s a nice statement of intent. However, it is entirely contradictory to Verio’s actual actions (or rather, lack thereof), as we can document complaints to abuse@verio.net about this specific IP dating back at least to November.
And as I told their PR rep, the problem is not Verio’s public relations department. Because anyone with a spam issue contacts their abuse department, and very few are so bull-headed as to continue when they get no results. Unlike me. So I will gladly state that Verio’s PR department was responsive and professional. Verio’s abuse department is a virtual black hole. And has been for some time.
In fact, other than the initial autoresponse for abuse@verio.net, I’ve heard nothing in reply from Verio’s abuse department at all. The PR rep claimed on the phone (paraphrasing) “it takes time to gather evidence” in a case like this, but this has been known for several months, even if I’d never sent the first e-mail. Ann Elisabeth e-mailed Verio’s abuse department with details on the 14th, with links to evidence of this spammer’s abuse dating back to last September.
And she also points to this comment by Paul Beard:
What puzzles me is how colocation and broadband providers never seem to monitor their networks well enough to see this: if individual sites can see these storms, I imagine are even easier to see on the sending side.
I suppose the only recourse is to ensure no reputable business uses shoddy hosting providers: perhaps we need to start publishing a score card that tracks what provider networks are responsible for the most outbound crap.
Here’s the first entry to go on the score card. One of the site owners who’s been inundated with this spam made contact with the alleged spammer … who politely said they would remove his URL from their database.
Now, my point isn’t that this politeness somehow makes their actions “OK.” My point is that this spammer has been more responsive to complaints than Verio.
So all I can do is repeat my opinion and advice: Do not use Verio, they harbor one of the biggest spammers on the web, and appear unable or unwilling to do anything about it. If you are a web host, you need to block the IP (161.58.59.8) in order to end the abuse to your servers.
Because Verio won’t.
Published 01:20PM, Thu, Jan 20 2005
Category: Weblogs Internet
Previous: «« Avian Deja Vu ««
Next: »» Bird Bowl Preview »»
Peanut Gallery
Well, Noel, you’re a lot more up on the server-side aspect of it than I am, so I’m theorizing. I wasn’t so much thinking about HTTP_REFERRER as any packet of any type originating from that IP. If there isn’t a way to do that now, someone needs to get crackin’ on mod_ipblock, or something. In this short term case, even if you had to make a central plain text list of all 700+ domains, then this theoretical “mod_ipblock” could call that for reference.
Like I said, I’m no Apache wrangler (caused my own site to go 500 for about 25 minutes this morning). But if there isn’t a way devised as of yet, this is sure a circumstance that calls for it.
Because this guy just opened up on a server in China, too. And if you think Verio has been unresponsive…
In one real sense, Verio is just the mothership of a very complex multi-level marketing organization. AUPs are nice things to show the public but buissness relationships often get in the way of enforcing them. Once upon a time I could have made this account go away in half an hour. Times change. Soon enough this spammer will trade the speed and connectivity of a good connection for the immunity of an off-shore connection. Then what? In the mean time I see no reason not to null route the IP at the border and move on.
You know, Jan, throughout this, I really did think about the ol’ Mindspring Abuse department. I thought about how this all would have been over with by about an hour after my first e-mail. There might have even been a verifying sound, if you listened at the right time. Or maybe a puff of smoke. And I wondered if there’s any abuse departments left out there that have that kind of “heft.”
That’s the aspect of this I find most frustrating. A clear cut case of violation of published rules … followed by the sound of One Big Empty.
The problem is that whoever sends the referrer is basically just sending a header, just a variable, and it’s not attached to an IP, unless you ping that referring domain and resolve the IP-so the IP that you are getting is called REMOTE_ADDR or REMOTE_IP which is just the visitor’s IP address, which helps with blocking, if it stays static, however this guy is on a rotating proxy setup-so that blows blocking any IP out of the water.
So then you move on, what if there was an apache module that resolved IPs of referrers? sure, it would work, but it would also slow down ever incoming visitor too, while their referring domain’s IP gets resolved,checked against rules, and dealt with.
In the end, it seems as though the solution is unreachable, unless the intermediary automatically sent IP addresses with the link clicking (perhaps the browser? i don’t know the logistics of that 100%) as HTTP_REFERER_IP.
I called Verio, and talked to somebody that didn’t have anything to do with the hosting department. And she told me I needed to send that abuse mail to THIS address:
abuse **a**t verio-hosting **dotcom (munged).
I got the feeling it would be ignored if I didn’t send it to the right department. I did get a reply, but haven’t seen that server actually shut down or the spammer’s domains shut down so far.
Also, one of the e-mails you sent Verio contained some factual errors. Verio’s server is unlikely to see any action concerning mt-comments, because the spammer never goes near that server when he sends out his spam. The only thing they can see is the stuff that ends up in referrer logs, or comments we send them. They can’t verify that with their own logs, only that the domain is hosted on their server, if that much.
The account is most likely a reselller account. Which means the spammer is adding these domains as fast as he can (no need to ask Verio to add them), and the only way to shut this down is to remove any and all reseller accounts touched by one of those domains.
“And she told me I needed to send that abuse mail to THIS address”
Well, they list no such address on their contact page . Further, a search of their site says “Sorry, no matches were found containing abuse@verio-hosting.com.” Not to mention the e-mail advertised in the WHOIS listing for the IP in question (abuse@verio.net). In fact, if you search the whole dang web for that address, Google shows two returns.
If that is the only address that can take action, it is a real shame they’ve kept it such a secret, even on their own site.
“Verio’s server is unlikely to see any action concerning mt-comments, because the spammer never goes near that server when he sends out his spam”
What I meant in that e-mail is that Verio surely has webhosting clients running MT, and they have therefore surely seen the spikes associated with comment spam attacks … not from this particular spammer, but from the sea of comment spammers out there that hit most every host with MT on its servers.
You know what worries me? Somewhere an argument is being waged that this doesn’t break the AUP. You can’t link the originating sources with the Verio box. We all know that’s certainly not the case. But if you wanted to argue about keeping a paying customer online against complaints it’s tenuous but feasible.
Think of it this way. What if the URL of one of our sites was the referring spam instead of the usual rubbish? By the same argument do I break my hosts AUP? The source is nowhere near me, but my site is in the referral…see where I’m going with this? It’s obviously not the case but I’ve seen worse arguments.
If Verio/Verio Hosting are taking this course then it really does have to degrade into a name and shame campaign as Paul Beard suggested. They may be seeing hosting spikes themselves, but in my experience business concerns win over technical concerns every time. Morals just doesn’t enter into it. But then it never does in legal/contractual discussions.
The question for Verio now is what price have they staked on their reputation?
“Somewhere an argument is being waged that this doesn’t break the AUP”
I think you’re probably right. Because someone is parsing the AUP by semantics rather than intent. This spammer could indeed have their proxy ‘bot running from a DSL/cable connection via most any non-Verio vendor. They could then argue, “hey, the spam packets are not even traversing the Verio network.”
No, just the profit side of the equation. The packets that sink the spammer’s hook. It’s like the hourly-rate motel that says, “hey, we don’t hire prostitutes, we just take their money for the room. So don’t complain to us, we’re not a part of that ‘profiting-from-sex’ business.”
In my opinion, here’s what the bottom line ought to be from a corporate point of view…
A “customer” is using our network to profit from spam runs that are negatively impacting other servers, and ultimately causing a negative impact on consumer attitudes towards Verio. Our abuse department, heck, even our PR department are having to expend man hours just to open the complaints, nevermind deal with them. How much is this guy paying us per month?
This guy is costing us money.
I know that with email spam you can report this sort of thing to various blacklists and the offending ISP/host will find a significant portion of the Net dropping their traffic until they fix the situation, but is there anything similar for comment/referer spam havens?
Not at this time, not to my knowledge. Which is sort of the point of this article. Since there’s no “official method,” we have to do what we can on our own.
I think maybe all this talk about this spammer and their Verio box did have an effect, if not the desired one (i.e., action from Verio). Over the past few days, this spammer has moved all of their domains to a box hosted … in China.
So, it looks like at this point, Verio has blown the chance to shut down this guy, after over five months of using his Verio hosted server to host his spam domains.
Only if the complaints are not loud enough. Witness the victor of the battle between millionaires and a bird.
However, I also believe that, the larger the corporation, the harder you have to yell. That’s why I tried to pinch the PR department for a yelp. No luck this time. But this is my strategy for 2005 … call for comment. You never know what you’ll stir up.
this is just a test. want to know what a text=pattern driven site is.
I have been suffering from comment-spam at my graymatter powered site for a week or so. I traced back the referers all to one registrar, Moniker.com – and it turns out that Monkier.com itself is a spammer! So, when the spammers become registrars, what hope do we have? I contacted internic and icann, but I haven’t heard anything, and doubt that I will.
Well, the only solution for fast action is to have a lawyer write up a letter saying that you will seek damages or a possible class-action suit if they do not stop the spammer. Alas, this is the way America works and a corporation only respects laywers and possible financial exposure.
If you think about it, they would lose money by tasking technicians to track these people down so you have to make it (at least potentially) more costly to ignore it.
Sue them and don’t look back. I’ll bet you could find some geek lawyers that would type up the letters for free. Good luck.
I just found out that the Verio server is still full of spammy sites. Many of them spamvertized November 2004. Almost all of them are .info sites. Using the same dns service as the domains we got off that server. And similar contact info. If you look crosseyed at it, it looks like an earlier version of the same spamming outfit.
This might be helpful:
http://sourceforge.net/projects/sentrytools/
First off, love the photos. {end digression}
Secondly: The best method of mitigation mentioned is/was the use of .htaccess. Links from AnnElizabeths site lead to a great example of this and also the Kuro5hin post is a good resource.
The reason I mention this is that it is the most economical in terms of processing and request handling for your server(s). One other suggestion I’ve read is to re-direct the requests to the offending site. I have to investigate this one and see what happens.
And last but not least is a suggestion to use iptables rules to black hole the requests even before they get to apache. I think that this is the most coding intensive but would be quite rewarding.
Recent Comments
- Ryan Ward (The Original Dither)
- Reid (The Not So Public Option)
- elburro (The Not So Public Option)
- Stephanie Hunter (The Not So Public Option)
- Reid (We Are Covered In Bipartisan Bull)
- Susan (We Are Covered In Bipartisan Bull)
ReidStott.com
Photography
by Reid Stott
A decade of web design experience. Two decades of photography experience. All available to you, and your project. View my portfolio online, then let's talk about your needs.ReidStott.com
It remains, that the biggest problem, and really an unsolvable problem, is the fact that HTTP_REFERRER can not be blocked by IP address, but by domain only—if you get 700 domains spamming your referrer logs and they are all from the same domain, you have to block ALL 700 of them.
So the problem can’t be stopped just by blocking that IP—cause you can’t block it in the right direction.
Verio, seriously, get this MFer off your god damn machines—I’m well endowed, and i really sleep quite well, and I really hate texas holdem (FYI). So, I don’t need any help with those things. ty