Cerberus Spam Filter
Cerberus Procedures
White Listing
Occasionally you may find Emails incorrectly tagged with the ***SPAM*** Subject line modification. While emails that are certainly spam are blocked entirely, we make use of the tagging system to flag emails that our filters are uncertain of.
We can deal with incorrectly tagged emails in two ways:
News Lists - if you are subscribed to a genuine mailer system to receive news or other updates via email, we can create filter rules to check for particular formats or layouts of emails, in order to match and reduce the spam score.
Human Emails - emails not sent from a mailing system are far less likely to be tagged with our ***SPAM*** tag, but it can still happen. If the email was sent through servers that have been previously known to deliver spam, or include certain keywords in the text, they can also be tagged.
In both cases, forward the email to whitelist@knowall.net and we will ensure future emails are cleared through our Cerberus system.
Spam email can sometimes be missed - especially if it is new, contains little but a single image, or text that cannot be matched to any currently known spam formats or keywords. Our servers will scan for sender servers, telltale chain-email markers, and known spammer addresses or website links, but new streams of spam often have previously unseen addresses and formats. In these cases, we can specifically block emails in similar ways to our Whitelisting rules above:
Most spam will originate from random email addresses - often spoofed from genuine and legitimate email sources, so direct address blacklisting cannot work. However, all spam is mass-emailed, and will contain a pattern we can recognise and block, even if the text in the email is different each time. In these cases, we can add this pattern to our filter rules and begin to add greater spam scores to subsequent similar emails.
Some spam comes from familiar addresses - unwanted news list mailers and phishing emails pretending to be from Banks or similar organisations. We can also simply blacklist the addresses these emails come from or the websites the emails link to.
For both methods, we ask that you foward unwanted emails to spam@knowall.netwhere our technicians will add their contents or addresses to our databases.
Emails that are blocked entirely will never reach our client's servers. We use a tiered system to flag potential spam email, which should provide a buffer to completely prevent any legitimate email being stopped in this way. Our system, in general, only blocks emails that are certain spam. It categorises spam like this by having seen multiple copies of the text, layout, format, or links in the past, either with the system's automatic Bayesian learing database, or by our team's manual additions to the filter databases.
Attachments are probably the biggest reason to block legitimate spam. Our servers currently do not allow certain file types through at all. At this time those file types are:
- vbs
- pif
- scr
- cmd
- com
- cpl
- dll
Each of these attachment types are strongly associated with Viruses or Spyware, and will be detected if they're in a compressed (zip) file, and even if the file itself is not named with the three letter extension in the list above. Any email containing these attachments are blocked with no bounce message.
There are many other reasons why your email may have never come through, and although we are confident that our systems should never block a good email (it is far more likely to be ***SPAM*** tagged), we can check our queues, quar nes and message log files for reasons behind your blocking. Contact support@knowall.net in these cases.
Configuring -Spam filtering Rules in Outlook 2000, XP or 2003
First, create a new folder in Outlook - wherever you like, but as a subfolder of Inbox is probably best. Call it whatever you want, but I'd suggest "Spam".
1 In Outlook, Click Tools > Rules Wizard
1a - If you are using Outlook 2002, you may be asked for an additional selection at this stage. Choose "Apply Changes to this folder: - Inbox [Microsoft Exchange Server]" To save your new rule to the server.
2 Create a new rule by clicking the New... button.
3 Click next to the default first setting ("Check Messages when they arrive"). If the option is available to "Start from a blank rule" Please also select this. This depends on the version of Outlook you are using, and will be on the same screen as the "Check Messages when they arrive" Option.
4 Select from the list "with specific words in the message header"
5 Click the words "specific words" in the box below the list of rules
6. Type into the new window “**spam**” (without the quotation marks) and click OK
7 Click next on the Rules Wizard Box
8 We're now choosing what to do with the matched email, so click "move it to the specified folder".
9 Click "specified folder" in the box beneath the list of options and select the spam folder you recently created.
10 Click OK, next and finished, and then OK, and that will now start filtering spam into the spam folder.
Once a domain has been added to the Cerberus filtering system, it will need updates to the DNS MX records for the domain to be filtered.
The records need to be changed to point at Cerberus and Chimaera, the two divisions of our server farm, and the old records need to be removed. If any old records are left in place, even as lower priority records, Spammers will deliver mail directly to those servers, bypassing the filtering.
The correct records should look like this:
domain.com IN MX 20 cerberus.knowall.net
domain.com IN MX 20 chimaera.knowall.net
Another way to write this would be:
Priority 20: cerberus.knowall.net
Priority 20: chimaera.knowall.net
The records should both have the same priority, which allows a degree of load balancing. If we have a problem with either side of our networks, the mail will always go to the live record, so this also enables a backup system.
Firewalling
It is also possible to stop spam bypassing our systems by blocking access to your mail servers using a firewall. Once setup with our Cerberus filtering system, your mail servers should never receive mail from anywhere but our servers.
To enable this, block all access except from the following IP subnets:
80.45.110.32 Subnet Mask 255.255.255.248
81.29.76.80 Subnet Mask 255.255.255.240
HOW Cerberus Works
Cerberus Anti Spam Techniques
1) Cerberus checks delivery headers for falsified information, non-existent domains or servers, and automatically generated mail. To cover their tracks, spammers will almost always falsify information such as email address, the sending server’s IP, and server handshake messages. Spam often comes from a well-known domain, such as yahoo.com, aol.com or hotmail.com. Cerberus knows what a connection from such servers looks like, and if the connection is false, will detect it.
2) Spam is often entirely or partially randomly generated to avoid anti-spam systems that check for previous duplicates (all the nonsense words often seen at the bottom of spam). Unfortunately for the spammer, email generated by software looks very different in the transmission headers to ordinary mail sent from a user in Outlook or other normal email clients. Cerberus can detect these differences and spot that a human didn’t send the mail, instantly giving the email a high points score.
3) Cerberus has a regularly updated list of URLs, Telephone numbers and street addresses that often appear in spam messages (“call 1-800-555-2345!”), and will reject mail that includes them.
4) Cerberus maintains it’s own automatically updated database of keywords and other key information about previously received mail that has scored highly against other rules. Any new spam that matches enough of the keywords is given points. The database also keeps a record of non-spam, and actually reduces the score on mail that doesn’t match its database. The database, of course, updates in real-time upon receipt of every single mail it gets.
5) The Razor anti-spam database is a remotely maintained database of known spam. Every message that passes through Cerberus is checked against it. If the spam has been seen before by any one of thousands of users that regularly submit spam, it will match and get points. Cerberus sends updates to the Razor database whenever a spam message gets over a fixed number of points, so we are also contributing.
6) Every message is checked against the following DNS and Relay blacklists:
a. http://www.dnsbl.njabl.org/ - NJABL
b. http://www.dnsbl.sorbs.net/ - SORBS
c. http://opm.blitzed.org/ - OPM
d. http://www.spamhaus.org/sbl/ - Spamhaus Block List
e. http://dsbl.org/ - DSBL
f. http://sa-hil.habeas.com - Habeas
g. http://bl.spamcop.net - SpamCop
h. http://www.mail-abuse.org - MAPS (4 separate lists)
i. http://bondedsender.org - BSP
Each match will give a few points towards the score of a potential spam – several matches, or one match and points from other checking techniques will label the mail as spam. This reduces overblocking, since many servers listed on the blacklists are innocent, having been used in the past to relay spam without the knowledge of the owners.
7) Language is tested for obfuscations and particular words or strings of words. “Vi.ag.ra”, “V*I*A*G*R*A”, “v1agr4” and other such possibilities are recognised, and will actually score the mail higher than if just the plain word “Viagra” is present (although not by much in this case). Words that appear in most spam, like “mortgage”, “enlargement” and “medication” are checked in similar ways with every possible combination of letters, numbers and symbols that could be used to make the word look right but normally avoid scanners. Higher points for obfuscated words means that genuine mail that happens to mention one of these subjects is less likely to be tagged as spam.
8) The style of the email is checked too – a large amount of images or HTML (rather than plain text) is recognised and awarded points. Coloured Fonts, Large Fonts, Block Capitals, Large gaps in content, lots of non-words (such as strings of punctuation symbols) and excessive exclamation marks are spotted and scored.
There are many other small checks, too many to list in detail. Checks on sender addresses – such as the use of a Polish (.pl) domain, or numbers in the username (before the @) – will add a few points each. Invalid email addresses, no sender address, blank subjects, many CC recipients, long spaces in subject, foreign language (such as Korean) text and lots of other checks are all carried out, and will all add points.
With enough points, the email is tagged, a certain amount more, the email is blocked entirely. Somewhere in between, the email has enough points to be reported to the Razor database. The amount of points (no matter how many) is also used to update the Bayesian database with keywords that denote future matches as spam or not spam
The current solution to the increase in ‘backscatter’ type bounce spam (genuine bounces from faked-sender spam) uses SPF records on the domain. This is a DNS TXT record (like a CNAME or A record) which lists every address that email from that domain should come from. Some servers – especially the larger company servers - check these SPF records and reject email if the source IP does not match.
For example, knowall.net has a TXT record of: “v=spf1 ip4:80.45.110.33/27 ip4:80.46.97.80/28 ~all”
In this way, the spam email is rejected on connection, which does not generate a bounce message sent to the faked sender address. It also helps to reduce overall spam levels by ensuring the sender is a genuine source.
Cerberus
Spam Filter