Spam Control and Blacklists

As a provider of mass mailing services, one of the most persistent and pervasive problems faced by Blue State Digital and, by extension, our clients is the management of relationships with email providers to prevent mailings sent through our infrastructure from being classified and treated as "Spam." This paper aims to clarify both the nature of this problem and the countermeasures Blue State Digital takes to minimize its impact on our clients.

 

What is Spam?

 

Surprisingly for a concept which draws so much vitriol on the internet as a whole, the definition of spam is somewhat difficult to pin down with any kind of precision. Perhaps the most universally recognized definition is the Spamhaus definition of spam:

 

An electronic message is "spam" if (A) the recipient's personal identity and context are irrelevant because the message is equally applicable to many other potential recipients; AND (B) the recipient has not verifiably granted deliberate, explicit, and still-revocable permission for it to be sent.

 

...

 

Spam is an issue about consent, not content. Whether the Unsolicited Bulk Email ("UBE") message is an advert, a scam, porn, a begging letter or an offer of a free lunch, the content is irrelevant - if the message was sent unsolicited and in bulk then the message is spam.

 

Spam is not a sub-set of UBE, it is not "UBE that is also a scam or that doesn't contain an unsubscribe link". All email sent unsolicited and in bulk is Spam.

As this definition makes clear, there are many who would consider at least some portion of our outgoing mailings to be UBE or "Spam." This is not to say that Blue State Digital is a provider of UBE, but the combination of allowing bulk uploads through the control panel of addresses gathered offline and the fact that we have unconfirmed opt-in (definition) for addresses added to an email list means that the verifiability of the consent provided is often called into question and can be difficult to prove conclusively.

 

What is the Legal Status of UBE ("Spam")?

 

It may seem surprising given the lack of regard for legality on both sides of the fight against UBE, but legality of the emails we send on the behalf of our clients is a question we get asked fairly frequently.

 

Unfortunately, providing legal advice is beyond the scope of this document.

 

It is nevertheless worth mentioning the most frequently cited document in this discussion: the CAN-SPAM Act of 2003 (source). While there is much debate over the efficacy as written and the enforcement of this statute, it is fairly clear from a layman's reading of the act that our bulk mailings fall firmly on the correct side of the law. It is likewise equally clear that most if not all ISPs go significantly beyond the legal provisions of the act in policing against UBE, rendering an in-depth legal discussion on this subject a purely academic endeavor.

 

How is UBE Treated by Internet Service Providers?

 

It should be apparent by this point that there is a surprising lack of consistency surrounding the question of UBE, beginning with even the definition of the term. Perhaps unsurprisingly, treatment of UBE varies widely from mail provider to mail provider. All treatments have the same three phases, however:

 

  1. Detection
  2. Sanction
  3. Rehabilitation

 

There are as many variations on the themes above as there are mail providers, but they tend to follow a similar general pattern. Thus, rather than provide an exhaustive list of ways in which these phases manifest, it's probably more productive to discuss the common properties of each stage and provide a few representative examples.

 

Detection

 

This is the phase in which a mail provider makes the decision to mark a mail sender as a source of UBE. Some providers take a purely volume based approach, reasoning that only a UBE source would send a certain magnitude of messages. A common variant on this theme is looking for servers which send an inordinate percentage of undeliverable mail, reasoning that a list with a lot of undeliverable addresses was probably gathered through harvesting (definition) or purchased from an email harvester. Others rely on DNSBL (definition) services, Bayesian analyses (definition) of sample messages, or the aggregate preference of their users via "Mark as Spam"-type interfaces. Still others rely on dedicated appliances or software with complex proprietary algorithms such as Barracuda (http://www.barracudanetworks.com/) or Postini (http://www.google.com/postini/index.html).

 

In general, ISPs are reluctant to discuss their spam detection techniques. The extent that we know the detection algorithms of particular ISPs is based more on statistical determinations from past experience than any more formal information sharing.

 

Sanction

 

During this phase, a mail provider will apply some kind of often graduated "punishment" (for lack of a better word) to misbehaving mail sources. Some common forms of sanction include rejecting mail outright with a 500-series SMTP code (Common SMTP Codes), graylisting (definition), and accepting delivery but placing the incoming message in some kind of UBE quarantine or "Spam Folder" in the mail clients of the recipients. Frequently the only notification of this treatment comes in the messages returned by the receiving mail server which end up in the logs of the sending server.

 

Rehabilitation

 

During this phase, the administrators of the sending mail server have become aware of the problem and are attempting to re-establish the good name of the sending mail server which is being sanctioned. The process here varies widely but is usually guided by the log entries mentioned above. Most common are specifically formatted mails sent to postmaster addresses at the receiving domain or web forms of varying complexity. In some cases, rehabilitation is impossible due to the design of the sanction or the action of vindictive administrators. In still other cases, payments can be requested for removal; an ethically gray area which it is our policy to avoid becoming involved in.

 

"Blacklist" Policies and Procedures

 

Whatever the cause, when a mail provider has sanctioned one or more of our servers this is commonly referred to as being "blacklisted" by that mail provider. In order to minimize both the number and severity of these blacklistings, we take a number of actions both preventative and corrective in nature.

 

Prevention of Blacklisting

 

Some email providers provide mechanisms to register outgoing mail servers with them in order to prevent blacklisting. Usually this involves providing contact information for an administrator, data about the type and general content of messages being sent, and data about the business taking responsibility for the mail server. Thereafter, the server in question may be partially or fully exempt from UBE checks on the theory that if the server becomes a problem, the mail provider knows to whom they can talk about the problem. Blue State Digital seeks out new so-called "whitelists" wherever they are applicable, effective, and offered in good faith, and we maintain a list of such whitelists for application in bringing up new mail servers. As a matter of policy, proprietary systems which interfere with content or presentation are not pursued, nor are "pay to play" schemes in which membership on a whitelist is sold for a price.

 

Blue State Digital also maintains a list of frequently recurring blacklists and proactively checks servers against them to minimize impact from these bottlenecks. As of this writing, this is a manual process, due to the idiosyncratic nature of de-listing procedures.

 

In addition, we maintain a fault tolerant pool of mail-sending servers, allowing us to drop badly performing servers out of the pool temporarily while their issues are being resolved.

 

Finally, we exercise some basic checks against uploaded bulk subscriptions to ensure that they contain valid emails so we can avoid hammering mail providers with many undeliverable mailings.

 

Detection of Blacklistings

 

Despite the above measures, blacklistings still occur frequently as a result of the nature and volume of emails sent through Blue State Digital architecture. In order to detect these issues, we compile and monitor daily aggregate information across all our clients and review it for changes in either percentage of failed deliveries or percentage of recipients who view our clients' messages. Changes in deliverability or email open rates caught in this review process are investigated and resolved, as well as reported internally on a weekly basis. In addition, requests by clients to investigate delivery to particular destinations are pursued immediately; if such an investigation is needed it can be instigated by a client services representative.

 

Resolving Blacklistings

 

See Rehabilitation above for details on how blacklists are handled. There is no general rule for how a given mail provider handles getting a server back into their good graces, but in general if rehabilitation is possible, the steps to do so will be discovered during the detection phase.

 

Dealing with "vigilante" blacklists

 

While there is no hard and fast definition of a "vigilante" blacklist, the general criteria are some combination of the following:

  • Lists that allow for unilateral, manual (non-automated) list additions.
  • Lack of clearly-stated objective criteria for non-automated list addition/removal.
  • Self-righteous or condescending statements on their website.
  • Lack of any automated removal process
  • Removal requires payment

 

In short, lists that allow an individual maintainer to make a subjective, unilateral or arbitrary choice to add us to their blacklist, and furthermore requires (possibly the same) individual to make yet another subjective decision on removal, particularly when that list is managed in condescending or arrogant tones, is likely to be deemed "vigilante."

 

Some examples are:
njabl: www.njabl.org (This one comes up a lot, many people use them, however they have a demonstrated history of bad faith actions toward BSD and others)
ahbl: www.ahbl.org
rfc ignorant: www.rfc-ignorant.org
5-10: blackholes.five-ten-sg.com

 

If the vigilante blacklist in question has an automated removal process, we will attempt to follow it. However, we will not engage in direct communication with the list maintainers. If a client wishes to pursue direct-communication with a vigilante blacklist, they may do so at their own risk. Past experiences prove there is little to gain, and much to lose by embarking on this path. However, should a client choose to do so, BSD may provide information or support to the client for this purpose at our discretion based on our risk assessment of the situation.

 

Lastly, our opinion, as well as the consensus of these lists' maintainers, is that membership on any single blacklist should never be used as sole criterion for blocking mail. If a particular ISP or mail hosting provider is doing so, this is a situation that BSD cannot fix. We suggest the client contact the ISP or email provider directly.

 

Things to consider

How will I know if a Blue State Digital mail server is blacklisted?

 

Ideally, you won't. Under the normal course of events these problems are caught and corrected on an ongoing basis, and you shouldn't have to make decisions about when to send your mailings based on the current health of our architecture. We aim for and regularly achieve a less than one percent bounce rate in the aggregate. In other words, because of how our architecture is designed, blacklists are a maintenance issue, not a catastrophic failure waiting to happen.

 

If I have lost a large number of addresses from my list suddenly, is that an indication of a badly timed blacklisting?

 

Usually not. More often, this is simply normal bounce unsubscription. It does, however, bear investigation. If you wish to investigate a large loss of subscriptions to your list, contact your client service representative (help@bluestatedigital.com), and they will help you pursue the matter.

 

If a large number of email addresses bounced due to a blacklisting, will they be unsubscribed?

 

No. A given address must in general bounce three times to be unsubscribed (see Bounce Unsubscription Policy for details).

 

If I lost a large number of addresses from my list due to being on a blacklist, can I resubscribe those lost addresses?

 

In general, yes. Contact your client service represenative (help@bluestatedigital.com), and we will investigate the matter and can resubscribe constituents to your list as appropriate.

Didn't find the answer you were looking for?

Email us at help@bluestatedigital.com