Japanese

top report presentation (PDF) article (Ja.) white list black list watch tool applyers Q&A blog (Ja.) links history contact
       : Be sure to read before applying.

Study Report of an Anti-spam System with a 99% Block Rate

-- The Selective SMTP Rejection (S25R) System --

Last update:  Jul 15, 2021
Publication:  Jun 26, 2004
Author:  ASAMI Hideo (Japan)
Location of the original:  http://www.gabacho-net.jp/en/anti-spam/
Author's contact address:  <webmaster@gabacho-net.jp>

Contents
Abstract
1. Conventional anti-spam countermeasures
2. Concept of the S25R anti-spam system
3. Client restriction rules
   3.1. Generic rules
   3.2. Black list
   3.3. White list
   3.4. Other filters
4. Statistics
5. How to operate the S25R anti-spam system in large sites
6. To operate the S25R anti-spam system in all over the Internet
7. How to slip through the S25R anti-spam system for spammers
Conclusion
Appendix A. Postfix configuration
Appendix B. Script for extracting rejection records

Abstract
   Mail servers which perform open relay (third party relay) have gotten fewer, so recently spammers have been distributing spam directly from many end-user's computers which are connected to high-rate end-user's subscriber lines such as ADSLs or cable networks and infected by bots. The open relay black lists have already been useless. An effective anti-spam countermeasure is to accept SMTP accesses from mail relay servers but reject those from end-user's subscriber lines. Either of them can be distinguished by a reverse name of a client's IP address. Using this system, I have succeeded in blocking nearly 99% of abuse mail including spam and virus mail without using any additional software other than Postfix.

 1. Conventional anti-spam countermeasures
   There have been anti-spam countermeasures as follows, however, all of them have been broken by spammers.
  1. Sender domain check
       Spammers often use a fake sender address so that they can avoid receiving complaint replies. So some mail systems take a countermeasure refusing a message with an inexistent sender domain. However, spammers have broken this countermeasure by abusing existent domains of innocent others. Needless to say, reversals of spam rush to innocent domains. A great deal of trouble!

  2. Content check
       There is a countermeasure refusing a message which includes specific words (such as "make money"). However, spammers have broken this countermeasure by modifying words (such as "m-a-k-e m-o-n-e-y"), inserting invalid HTML tags into a word, or encoding a whole message body into the BASE64 code.

  3. Open relay prohibition
       Spammers have abused open relay by mail servers on the Internet in order to distribute a lot of spam. So a countermeasure prohibiting open relay has spreaded. Additionally, some organizations have offered an open relay black list (ORBL) and have suggested to refuse messages from mail relay servers listed in it. However, spammers have been distributing a lot of spam directly to recipient's domains controlling many end-user's computers infected by bots, without relying on open relay, since end-user's high-rate subscriber lines such as ADSLs or cable networks became widely available.
       When we receive spam from such an end-user's subscriber line and input its IP address into an ORBL site, it answers that the host is not an open relay server. It is quite natural that evil spam transmission computers don't perform open relay because they only originate SMTP accesses but never receive them. The ORBLs, which expose mail system administrators who don't have security knowledge nor malice, seem to be tolerant of real wrongdoers. Sigh...

  4. Check of a client's IP address or domain name
       There is also a countermeasure checking an IP address of a client originating an SMTP access. However, even if we list a spammer's IP address on a black list, the spammer sends out spam from another IP address. Then the black list will get fat unlimitedly.
       We can also list either an IP address block (such as "223.45.67.*") or a domain name gotten by reverse lookup (such as "*.example.com"). However, legitimate mail relay servers may be refused getting involved in the black list.
       Consequently, mail system administrators may not want to use this countermeasure except when they get mail bombs, although it is close to my idea.
 2. Concept of the S25R anti-spam system
   The concept of the Selective SMTP Rejection (S25R) anti-spam system, which I invented, is so simple that a mail server accepts SMTP accesses from mail relay servers but rejects direct SMTP accesses from end-user's computers.
   While most of the legitimate mail is sent through mail relay servers of ISPs (Internet Service Providers) or organizations, most of the spam of nowadays is sent directly from end-user's computers connected to end-user's subscriber lines such as ADSLs or cable networks and infected by bots. Spam sent through mail relay servers is kept rather little because performance of a mail relay server effects as a bottleneck.
   Additionally, many viruses of nowadays have their own SMTP engines and distribute copies of themselves directly to destination domains. Viruses which go through a mail relay server are blocked if it has an anti-virus gateway.
   Because of the circumstances mentioned above, it is good policy to receive mail only from mail relay servers so we can reduce receipt of spam and virus mail.
   You might think it is very hard to distinguish between SMTP accesses from mail relay servers and those from end-user's computers, however, I invented a surprisingly easy means. As discussed in the next section, it is a means looking at a feature of an FQDN (Fully Qualified Domain Name) gotten by reverse lookup from an IP address. It can be easily actualized using Postfix.
   A mail server which acts as an MX (Mail eXchanger), namely a mail entrance from the outside, should perform such distinction. That is, an MX shouldn't accept a message before checking it but should reject probable spam. A spammer's computer will get a response code meaning failure before sending message contents. Therefore, wasteful traffic on the Internet will reduce. Moreover, fewer wrong reversals will rush to victims whose domain name is abused for a fake sender address. Meanwhile, as our mail servers return the response code meaning "try again later" when blocking, we can detect legitimate mail relay servers mistakenly blocked and exempt them. If this anti-spam system spreads over the Internet, spammers will realize that spam messages fail one and all.

 3. Client restriction rules
   Based on a feature of an FQDN of a client that originates an SMTP access, we can presume whether the client is a mail relay server or it is a computer connected to an end-user's subscriber line. The rules for restricting clients by this means consist of generic rules, a black list, and a white list (permission list). The generic rules trap most of the end-user's subscriber lines. The black list traps end-user's subscriber lines which slip through the generic rules. It also traps evil mail transmission servers. The white list exempts legitimate mail relay servers from mistaken rejection by the generic rules or the black list. Although this countermeasure cannot be perfect, it is quite available.

 3.1. Generic rules
   Most of the well-managed mail relay servers have their FQDNs which can be gotten by reverse lookup. IP addresses without a reverse FQDN are mostly of end-user's subscriber lines such as dial-up, ADSLs, or cable networks, because end-user's subscriber lines are far many than mail relay servers. Therefore, we can reject SMTP accesses from IP addresses without a reverse FQDN excluding a few exceptions.
   Rather many IP addresses of end-user's subscriber lines have their reverse FQDNs, however, the feature of them is different from that of mail relay server's names. Reverse FQDNs of end-user's subscriber lines often include rather many numerals for convenience of name assignment to many IP addresses, while mail relay server's names generally include few numerals because any network administrators prefer names easy to remember. So we can distinguish reverse FQDNs of end-user's subscriber lines from those of mail relay servers by using regular expressions. Then we can reject SMTP accesses from IP addresses which are presumed to be of end-user's subscriber lines excluding a few exceptions.
   The generic protection rules which I invented are explained below. These rules consist of only six regular expressions to check a client's FQDN as well as a specification for rejecting clients without a reverse FQDN. These rules are devised so that our mail servers seldom trap mail relay server's FQDNs.
  1. [Rule 0] Reverse lookup failure
       Reverse lookup failure means not only that an FQDN cannot be retrieved from an IP address but also that a result of forward lookup from a reverse FQDN is not consistent with the former IP address.
       Most of the hosts whose reverse lookup failes are end-user's computers. It is quite natural because end-user's computers are far many than mail servers on the Internet.

  2. [Rule 1] The bottom-level name (at left-end) of a reverse FQDN includes two or more numeral strings devided by a string of any characters but numerals.
    Examples:  220-139-165-188.dynamic.hinet.net
    evrtwa1-ar3-4-65-157-048.evrtwa1.dsl-verizon.net
    a12a190.neo.rr.com
    Regular expression of Postfix:  /^[^.]*[0-9][^0-9.]+[0-9].*\./
       FQDNs which are trapped by this rule are the most among those of end-user's subscriber lines. Although I have found some mail relay servers which are trapped by this rule, they are not so many.
       The part of the regular expression "\." at the end matches a dot. It is necessary to avoid trapping every IPv6 address.

  3. [Rule 2] The bottom-level name of a reverse FQDN includes a sequence of five or more numerals.
    Examples:  YahooBB220030220074.bbtec.net
    pcp04083532pcs.levtwn01.pa.comcast.net
    Regular expression of Postfix:  /^[^.]*[0-9]{5}/
       I decided to trap a sequence of file or more numerals because I had found a mail relay server's terminal host name including a sequence of four numerals at longest. After that, I have found mail relay servers whose terminal host name includes a sequence of five or more numerals, however, they are very few.

  4. [Rule 3] Excluding the top three layers, the bottom-level or second bottom-level name of a reverse FQDN begins with a numeral.
    Examples:  398pkj.cm.chello.no
    host.101.169.23.62.rev.coltfrance.com
    Regular expression of Postfix:  /^([^.]+\.)?[0-9][^.]*\.[^.]+\..+\.[a-z]/
       I decided to exclude the top three layers from check so that an FQDN such as "smtp.246.ne.jp" (this is existent) is not trapped by this rule.
       The part of the regular expression "\.[a-z]" at the end matches the head of a top-level domain name. It is necessary to avoid trapping every dotted decimal notation of an IPv4 address.

  5. [Rule 4] The bottom-level name of a reverse FQDN ends with a numeral, and the second bottom-level name includes two or more numeral strings devided by a hyphen.
    Examples:  wbar9.chi1-4-11-085-222.dsl-verizon.net
    m226.net81-66-158.noos.fr
    Regular expression of Postfix:  /^[^.]*[0-9]\.[^.]*[0-9]-[0-9]/
       An FQDN "mail1.1-2-3.co.jp" (this domain name is existent but the terminal host name is fictitious) will be trapped by this rule. However, I didn't decide to exclude the top three layers from check because some end-user's subscriber lines would be missed if I did so.

  6. [Rule 5] Both of the bottom-level and second bottom-level names of a reverse FQDN end with a numeral, provided that the FQDN is five or more layered.
    Examples:  m500.union01.nj.comcast.net
    d5.GtokyoFL27.vectant.ne.jp
    Regular expression of Postfix:  /^[^.]*[0-9]\.[^.]*[0-9]\.[^.]+\..+\./
       I decided to exclude the top three layers from check so that an FQDN such as "mail1.number1.co.jp" (this domain name is existent but the terminal host name is fictitious) is not trapped by this rule. An FQDN "mail1.gate1.example.co.jp" will be trapped by this rule, however, mail relay servers with such an FQDN are rare.

  7. [Rule 6] The bottom-level name of a reverse FQDN begins with "dhcp", "dialup", "ppp" or a name of the DSL family (such as "dsl, "adsl" or "xdsl") and includes a numeral.
    Examples:  dhcp0339.vpm.resnet.group.upenn.edu
    dialupM107.ptld.uswest.net
    PPPbf708.tokyo-ip.dti.ne.jp
    dsl411.rbh-brktel.pppoe.execulink.com
    adsl-1415.camtel.net
    xdsl-5790.lubin.dialog.net.pl
    Regular expression of Postfix:  /^(dhcp|dialup|ppp|[achrsvx]?dsl)[^.]*[0-9]/
       A few FQDNs are trapped by this rule only. There are almost no mail relay servers with such a terminal host name.

    Note   The regular expression part "adsl" has been changed into "[achrsvx]?dsl" since September 2007.
   When our mail server rejects SMTP accesses by those rules, it should return the response code "450" meaning "try again later". Then, making a white list (permission list) discussed below, we can exempt legitimate mail relay servers which are trapped by any of those rules.
   Appendix A. shows a file for Postfix to configure those rules together with a black list and a white list disscussed below.

 3.2. Black list
   The black list is used to reject end-user's subscriber lines which slip through the generic rules discussed above. It is also used to reject evil mail transmission servers.
   FQDN features which slip through the generic rules are as follows:
  1. A terminal host name includes a hexadecimal number.
       I found an FQDN "ACBBD419.ipt.aol.com". Although human observation can guess that this terminal host name shows a hexadecimal number, this FQDN slips through the generic rules. Nearly 4% of 8-digit hexadecimal numbers slip through both of the rules 1 and 2 among the generic rules, while nearly 96% of them are trapped either of the rule 1 or 2.
       To round up all host names which include a hexadecimal notation of an IP address, you may consider appending a regular expression "/^[^.]*[0-9a-f]{8}/" to the generic rules. However, if you do so, you shouldn't wonder why "speedfeed1" is trapped while "rapidfeed1" isn't.

  2. A terminal host name includes letters showing a number.
       I found FQDNs "user-0cetcbr.cable.mindspring.com" and "user-vc8fldi.biz.mindspring.com". The letters which seems to be random probably shows a number. Both of those examples slip through the generic rules.

  3. An FQDN looks as if it is of a mail relay server.
       I received spam from a host whose FQDN doesn't include numerals (the FQDN had the same syntax as "host.example.co.uk"). The host seemed to be dedicated to spam transmission because it didn't accept an SMTP access. Researching some host names under the domain name, I guessed that the domain assigned host names as the users liked. Spammers who uses such a service can slip through the generic rules.

  4. A representative mail server of a domain sends spam.
       Some corporations or Internet sites that have their own domain names send spam for advertisement. The FQDNs of their mail transmission servers generally slip through the generic rules.
   When we list an ISP's domain name on the black list, we should take care to avoid refusing mail relay servers in the same domain. When we want to make a denial condition from an FQDN "ACBBD419.ipt.aol.com" for example, a regular expression "/\.ipt\.aol\.com$/", which includes the subdomain name, may be good. As for FQDNs "user-0cetcbr.cable.mindspring.com" and "user-vc8fldi.biz.mindspring.com", a regular expression "/^user.+\.mindspring\.com$/", which includes a common substring, may be good.
   When we want to refuse a specific host, we can describe a regular expression such as "/^host\.example\.com$/". If we judge that the whole domain is suspicious, we may describe a regular expression as "/\.example\.com$/".

 3.3. White list
   The white list (permission list) is required to exempt legitimate mail relay servers which are trapped by the generic rules or the black list. The white list must be specified prior to the generic rules and the black list.
   Records of rejection against legitimate mail relay servers can be found from mail logs. If our mail server returns a response code "450" when it refuses a client by the generic rules or the black list, legitimate mail relay servers will retry mail transfer over several days. If each rejection record shows the same client's IP address, HELO address, sender address and recipient address at intervals of one minute or longer, it may show an access from a legitimate mail relay server. When we judge that the client is a legitimate mail relay server, we can receive the message by listing a permission condition on the white list. Most of the spam transmitters and viruses don't retry, otherwise they retry for a while at intervals of several seconds, because much computer resource is required to retry in the same way as a legitimate mail relay server does.
   To say for your information, Postfix kicks out a client by default after letting a client send a HELO command, a MAIL FROM command and a RCPT TO command even if it must be surely refused by client restriction. This action is helpful for us to find the sender address, the recipient address and the HELO address of a refused message from mail logs.
   Accesses from legitimate mail relay servers are rejected in cases as follows:
  1. Reverse lookup failure
       There are some mail relay servers whose IP address doesn't have a reverse FQDN.

  2. A reverse FQDN corresponds to multiple IP addresses.
       Occasionally, multiple IP addresses are gotten by forward lookup from a reverse FQDN. In such case, Postfix judges reverse lookup failure unless the first IP address gotten by forward lookup is consistent with the former IP address.

  3. A host name is trapped by the generic rules.
       There are a few host names which are trapped by the generic rules. As an example, a reversal of a message which I sent to a user of the "hotmail.com" domain came from a host named "mc1-s3.bay6.hotmail.com".

  4. An end-user's subscriber line is used.
       Some small organizations, personal Internet sites and mail magazine distributors operate their mail servers using end-user's cheap subscriber lines. Even though their mail servers have their own domain names, the reverse FQDNs of the subscriber lines are administered by ISPs, so they are trapped by the generic rules.
   We can permit an IP address which has a reverse FQDN by a regular expression such as "/^223-123-45-67\.example\.net$/", and permit an IP address without a reverse FQDN by a regular expression such as "/^223\.123\.45\.67$/".
   A mail server using an end-user's subscriber line can be permitted safely in the way mentioned above as far as it has a fixed IP address. Meanwhile it is risky to exempt a mail server which uses a floating IP address and the dynamic DNS, because an IP address which is assigned for a mail server at a time may be assigned for a computer infected by a bot at another time! I wish such a mail server sends out mail through an ISP's mail relay server, otherwise it switches over to the fixed IP address service.

 3.4. Other filters
   Although most of the spam messages are trapped by the client restriction, a few spam messages slip through it. To trap them, other mail filters sometimes take effect.
  1. HELO command check
       Even though the HELO command must introduce a client host's FQDN, some spam transmission programs and viruses announce with the HELO command a destination server's IP address or a recipient's domain name. Legitimate mail relay servers never do so unless they are infected with a virus. If a client in another network announces with the HELO command your mail server's IP address or your domain name, you don't need to hesitate about kicking it out returning a response code meaning "fatal".
       Those who make such SMTP engines which behave wrongly must be ignorant about SMTP rules. It's silly of them to offer by themselves a clue for sorting out their abuse mail!
       To say for your information, in order to reject mail by HELO command check using Postfix, we need to specify the keyword "REJECT". We cannot make a client retry mail transfer by returning the response code "450".

  2. Sender domain check
       It is a traditional countermeasure to check the existence of a sender's domain by the DNS. It seldom takes effect because many spammers of nowadays abuse existent domains, however, it is still worth specifying.
       To say for your information, Postfix returns a response code "450" by default when rejecting a client by the sender domain check. We must not change this default because DNS lookup may temporarily fail even if the domain name is existent.

  3. Content check
       It is also a traditional countermeasure to check a mail header and a message body and find words which are often included in spam. It occasionally takes effect. However, labor for maintaining content check filters doesn't pay because spammers have already devised many countermeasures to slip through content check.
       To say for your information, in order to reject mail by content check using Postfix, we need to specify the keyword "REJECT". We cannot make a client retry mail transfer by returning the response code "450".
 4. Statistics
   Here are statistics of block rates of abuse mail including spam and virus mail which came to my mail server during one month in April 2004. The block rates indicate percentages against 567 IP addresses of clients.
   Each increase of the block rates by the rule 2 to 6 among the generic rules doesn't indicate the percentage of all of accesses trapped by each rule, but indicates the percentage of accesses which were trapped by each rule after slipping through the preceding rules. The increase of the block rate by the black list doesn't indicate the percentage of all of accesses trapped by it, but indicates the percentage of accesses which couldn't be trapped by the generic rules. The increase of the block rate by the HELO command check indicates the percentage of accesses which were trapped by it after slipping through the client restrictions.
   There were 5 messages (0.9%) which couldn't be blocked by client restriction and other countermeasures used together. 4 of them were addressed to unknown recipients. I received only one message, which was spam sent from a mail relay server of a legitimate corporation. The mail relay server seemed to have been infected with a bot.
   Meanwhile, during that period, I found sometimes rejections against legitimate messages, all of which I successfully received by listing 5 entries on the white list.
   At the end of April 2004, my black list had 17 entries, and my white list had 9 entries.

Table 1. Statistics of abuse mail block rates at my mail server (April 2004)
rejection conditionmatch rateblock rate increasecumulative block rate
[Rule 0] Reverse lookup failure38.4%38.4%38.4%
[Rule 1] The bottom-level name (at left-end) of a reverse FQDN includes two or more numeral strings devided by a string of any characters but numerals.44.5%44.5%82.9%
[Rule 2] The bottom-level name of a reverse FQDN includes a sequence of five or more numerals.10.2%7.6%90.5%
[Rule 3] Excluding the top three layers, the bottom-level or second bottom-level name of a reverse FQDN begins with a numeral.12.3%6.5%97.0%
[Rule 4] The bottom-level name of a reverse FQDN ends with a numeral, and the second bottom-level name includes two or more numeral strings devided by a hyphen.3.4%0.9%97.9%
[Rule 5] Both of the bottom-level and second bottom-level names of a reverse FQDN end with a numeral, provided that the FQDN is five or more layered.8.5%0.3%98.2%
[Rule 6] The bottom-level name of a reverse FQDN begins with "dhcp", "dialup", "ppp" or a name of the DSL family and includes a numeral.7.6%0%98.2%
Blacklisted2.6%0.5%98.7%
The HELO command announces a destination server's IP address or a recipient's domain name.9.3%0.4%99.1%
Sender domain lookup failureunknown0%99.1%
Content checkunknown0%99.1%

Note   The false positive rate before whitelisting is estimated at about 13% in counting the sender hosts which have sent me legitimate mail. The applyers of large sites says that they need to whitelist about 1,000 items but the frequency of whitelisting gets reduced in a period of two weeks to one month. The false positive rate gets reduced as whitelisting proceeds.

 5. How to operate the S25R anti-spam system in large sites
   I developed the S25R anti-spam system at my own risk by using my personal mail server. When large sites such as organizations or ISPs introduce this system, it will be an important problem to control risks in operation.
   However greatly a mail system administrator burns with anger at spammers, he/she shouldn't be absorbed in bringing a spam block rate close to 100% like I do. Even if a user reports that he/she has received spam from a certain host, legitimate mail may come from the same host. We should carefully investigate damage by spam and behavior of a client host before listing the host on the black list.
   It is more important to avoid risk of recipients missing mail which must be received. It is unpardonable to leave legitimate hosts retrying. I make my white list watching rejection records by using a script which extracts rejection records from mail logs (see Appendix B.). Addittionally, I made an improved script which facilitates finding regular retries. It might be useful for large sites.
   Maintenance of a white list can be automated provided that we can allow a risk of mistakenly permittig a little bit of spam. A certain applyer of the S25R system has developed an automated whitelisting system. And furthermore, systems which automatically permit legitimate hosts by combining the S25R system with the greylisting (a method of permitting hosts which retry) or the tarpitting (response delay) have been contributed by Mr. SATOH Kiyoshi. For further information, refer to the links page:
http://www.gabacho-net.jp/en/anti-spam/links.html

6. To operate the S25R anti-spam system in all over the Internet
   If the S25R anti-spam system gets support in all over the Internet, it will become more reliable by agreement with a rule to distinguish mail relay servers from end-user's subscriber lines. In order to avoid trapping mail relay servers, I suggest a rule that terminal host names of end-user's subscriber lines begin with a numeral, include a sequence of five or more numerals, or include two numeral strings devided by a string of any characters but numerals.
   Additionally I suggest that all domains obey the rules as follows:
  1. IP addresses of end-user's subscriber lines have reverse FQDNs suitable for an end-user's subscriber line's name, otherwise they don't have reverse FQDNs.
  2. IP addresses of mail relay servers necessarily have reverse FQDNs suitable for a mail relay server's name. A reverse FQDN must not correspond to multiple IP addresses, or verification of reverse lookup may fail.
  3. In case a user operates his/her own mail relay server using an end-user's subscriber line, an ISP provides for the user a reverse FQDN suitable for a mail relay server's name, otherwise the ISP permits mail from the user's mail server to go through the ISP's mail relay server.
  4. In case a user operates his/her own mail server, he/she doesn't use a floating IP address and the dynamic DNS, otherwise he/she makes the mail server send out mail through an ISP's mail relay server.
 7. How to slip through the S25R anti-spam system for spammers
   Spammers can have countermeasures as follows in order to slip through the S25R anti-spam system. However, in any way, it seems to be very hard for spammers to go on to distribute a lot of spam at a low cost. That is, this anti-spam system is considered invulnerable.
  1. Sending spam through an ISP's mail relay server
       Spam sent through an ISP's mail relay server is not trapped by this anti-spam system because mail system administrators of recipient's domains don't blacklist the mail relay server as far as legitimate mail may go through it. However, spammers cannot distribute so much spam in this way. If a spammer overloads a mail relay server, he/she may be punished by the ISP. Moreover, ISPs may make their mail relay servers refuse a message with too many destinations or too frequent requests from a client. Therefore, spammers can only send a little bit of spam modestly. Does the mass spam distribution business pay in this way?

  2. Pretending to be a legitimate mail relay server
       If a spam transmission computer retries transfer against a response code "450" in the same way as a legitimate mail relay server does, a mail system administrator of a recipient's domain may mistake the spam transmission computer for a legitimate mail relay server, and then, he/she may list it on a white list. However, he/she will unlist it when the message turns out to be spam. Then the spammer will not be able to send spam again from the same computer to that domain. That is, in spite of spending computer resource for a mail queue for resending, it is hard to raise the reaching rate of spam. Does the mass spam distribution business pay in this way?

  3. Using an ISP which harbors spammers
       An ISP which intends to harbor spammers may provide for spammers FQDNs which seem to be of a legitimate mail relay server. However, the domain name of such an ISP will be soon blacklisted in many domains. Therefore, such an ISP will not endure. Sorry about that!

  4. Having an own domain name
       If a spammer can operate the DNS, he/she can use an FQDN which seems to be of a legitimate mail relay server by having an own domain name. However, the result will be the same as in case of using an ISP which harbors spammers. Attackers need to pay to change their domain names while defenders don't need to pay to blacklist domain names. I pity spammers!
 Conclusion
   It is not helpful at all to vent our anger at spammers for reducing spam. Spammers won't stop spamming because it pays. In order to put an end to spamming, we should rather keep on returning silently a response code for rejection in a way such that attackers need to pay while defenders don't need to pay.
   You can introduce the S25R anti-spam system at once free of charge as far as you are ready for controlling a risk of a small side effect by this system. Moreover, this system can block nearly 99% of spam or virus mail. The effectiveness of this system has been verified by my personal mail server. If this system spreads all over the Internet, receipt of spam will fall off drastically even though it will not be extinct.
   I don't claim any exclusive rights about my idea. And, if you invent a new means based on my idea, I hope you contribute it to the Internet world without claiming exclusive rights.
   I expect comments and contribution from the readers.

 Appendix A. Postfix configuration
   Here is a configuration for restricting clients using postfix.
   Several practical white list entry examples are picked up from the white list information page:
http://www.gabacho-net.jp/en/white-list.html
   Several practical black list entry examples which have often taken effect are picked up from those which I have listed in my mail server.
   The example FQDNs described in the black list and the generic rules are of the clients which had actulally tried to send spam or virus mail to my mail server. I don't mean that those domains are evil, but I just mean that those IP addresses have been used by malicious or careless users.

Lines to be added into the /etc/postfix/main.cf file 
smtpd_client_restrictions =
  permit_mynetworks,
  check_client_access regexp:/etc/postfix/white_list
  check_client_access regexp:/etc/postfix/rejections

smtpd_helo_required = yes

smtpd_helo_restrictions =
  permit_mynetworks,
  reject_invalid_hostname,
  check_helo_access regexp:/etc/postfix/helo_restrictions

smtpd_sender_restrictions =
  permit_mynetworks,
  reject_non_fqdn_sender,
  reject_unknown_sender_domain

Note   Though the former configuration had had one client restriction configuration file, it has been divided into a white list file and a rejection condition file. This way makes the file maintenance easier. In addition, this way enables us to specify any rejection message for reverse lookup failure. The reject_unknown_client specification, which returns the inconsiderate message "cannot find your hostname", has become unnecessary.

Contents of the /etc/postfix/white_list file 
# S25R client permission specifications for Postfix
# Contributed by ASAMI Hideo (Japan), Jun 2004; Jul 2007
# Refer to: http://www.gabacho-net.jp/en/anti-spam/
#
# To use this file, add following lines into the /etc/postfix/main.cf file:
#
# smtpd_client_restrictions =
#   permit_mynetworks,
#   check_client_access regexp:/etc/postfix/white_list
#   check_client_access regexp:/etc/postfix/rejections
#
# where "white_list" is the name of this file.
#
# *** WHITE LIST ***
#
# When you find a legitimate mail relay server which is rejected by the
# rejection specification written in the /etc/postfix/rejections file, write
# down here a permission specification taking a leaf from the following
# examples.
#
#/^223-123-45-67\.example\.net$/                OK
#/^223\.123\.45\.67$/                           OK
# 
# Practical examples:
#
# mail-gx0-f21.google.com, etc.
/\.google\.com$/                                OK
#
# mc1-s3.bay6.hotmail.com, etc.
/\.hotmail\.com$/                               OK
#
# h04-a1.data-hotel.net, etc.
/\.data-hotel\.net$/                            OK
#
# web10902.mail.bbt.yahoo.co.jp
/\.yahoo\.co\.jp$/                              OK
#
# web35509.mail.mud.yahoo.com
/\.yahoo\.com$/                                 OK
#
# n2.59-106-41-68.mixi.jp, etc.
/\.mixi\.jp$/                                   OK
#
# mta12.m2.home.ne.jp, etc.
/\.m2\.home\.ne\.jp$/                           OK
#
# mmrts006p01c.softbank.ne.jp, etc.
/\.softbank\.ne\.jp$/                           OK
#
# imt1omta04-s0.ezweb.ne.jp, etc.
/\.ezweb\.ne\.jp$/                              OK
#
# bay-w1-inf5.verisign.net, benicia-w2-inf30.verisign.net
/\.verisign\.net$/                              OK

Contents of the /etc/postfix/rejections file 
# S25R client rejection specifications for Postfix
# Contributed by ASAMI Hideo (Japan), Jun 2004; Jul 2007
# Refer to: http://www.gabacho-net.jp/en/anti-spam/
#
# To use this file, add following lines into the /etc/postfix/main.cf file:
#
# smtpd_client_restrictions =
#   permit_mynetworks,
#   check_client_access regexp:/etc/postfix/white_list
#   check_client_access regexp:/etc/postfix/rejections
#
# where "rejections" is the name of this file.
#
# *** BLACK LIST ***
#
# When you find a UCE sender's FQDN which is not rejected by the generic
# protection rules specified below, insert here a denial specification taking
# a leaf from the following practical examples. You should specify a subdomain
# name or a substring together with the domain name if possible so that you can
# avoid rejecting legitimate mail relay servers in the same domain.
#
# pr86.internetdsl.tpnet.pl
# fq217.neoplus.adsl.tpnet.pl
# pa148.braniewo.sdi.tpnet.pl
/\.(internetdsl|adsl|sdi)\.tpnet\.pl$/          450 domain check, be patient
#
# user-0cetcbr.cable.mindspring.com
# user-vc8fldi.biz.mindspring.com
/^user.+\.mindspring\.com$/                     450 domain check, be patient
#
# c9531ecc.virtua.com.br (hexadecimal used)
# c9066a60.static.spo.virtua.com.br (hexadecimal used)
/^[0-9a-f]{8}\.(.+\.)?virtua\.com\.br$/         450 domain check, be patient
#
# catv-5984bdee.catv.broadband.hu (hexadecimal used)
/\.catv\.broadband\.hu$/                        450 domain check, be patient
#
# Edc3e.e.pppool.de
# BAA1408.baa.pppool.de
/[0-9a-f]{4}\.[a-z]+\.pppool\.de$/              450 domain check, be patient
#
# pD9EB80CB.dip0.t-ipconnect.de (hexadecimal used)
/\.dip[0-9]+\.t-ipconnect\.de$/                 450 domain check, be patient
#
# pD9E799A1.dip.t-dialin.net (hexadecimal used)
/\.dip\.t-dialin\.net$/                         450 domain check, be patient
#
# ool-43511bdc.dyn.optonline.net (hexadecimal used)
/\.dyn\.optonline\.net$/                        450 domain check, be patient
#
# rt-dkz-1699.adsl.wanadoo.nl
# c3eea5738.cable.wanadoo.nl (hexadecimal used)
/\.(adsl|cable)\.wanadoo\.nl$/                  450 domain check, be patient
#
# ACBBD419.ipt.aol.com (hexadecimal used)
/\.ipt\.aol\.com$/                              450 domain check, be patient
#
# *** GENERIC PROTECTION ***
#
# [rule 0]
/^unknown$/                         450 reverse lookup failure, be patient
#
# [rule 1]
# ex.: evrtwa1-ar3-4-65-157-048.evrtwa1.dsl-verizon.net
# ex.: a12a190.neo.rr.com
/^[^.]*[0-9][^0-9.]+[0-9].*\./                  450 S25R check, be patient
#
# [rule 2]
# ex.: pcp04083532pcs.levtwn01.pa.comcast.net
/^[^.]*[0-9]{5}/                                450 S25R check, be patient
#
# [rule 3]
# ex.: 398pkj.cm.chello.no
# ex.: host.101.169.23.62.rev.coltfrance.com
/^([^.]+\.)?[0-9][^.]*\.[^.]+\..+\.[a-z]/       450 S25R check, be patient
#
# [rule 4]
# ex.: wbar9.chi1-4-11-085-222.dsl-verizon.net
/^[^.]*[0-9]\.[^.]*[0-9]-[0-9]/                 450 S25R check, be patient
#
# [rule 5]
# ex.: d5.GtokyoFL27.vectant.ne.jp
/^[^.]*[0-9]\.[^.]*[0-9]\.[^.]+\..+\./          450 S25R check, be patient
#
# [rule 6]
# ex.: dhcp0339.vpm.resnet.group.upenn.edu
# ex.: dialupM107.ptld.uswest.net
# ex.: PPPbf708.tokyo-ip.dti.ne.jp
# ex.: dsl411.rbh-brktel.pppoe.execulink.com
# ex.: adsl-1415.camtel.net
# ex.: xdsl-5790.lubin.dialog.net.pl
/^(dhcp|dialup|ppp|[achrsvx]?dsl)[^.]*[0-9]/    450 S25R check, be patient

Note   The former rejection messages had been "domain UCE-blacklisted" for the black list and "may not be mail exchanger" for the generic rules. They have been changed into ones including the phrase "be patient" to avoid worrying good senders who receive a deferment warning message.

Sample of the /etc/postfix/helo_restrictions file 
# Illegal HELO command blocking specification
# Provided that your mail server's IP address is 223.12.34.56 and its
# acceptable domain name is "example.com", specify as follows:
#
#/^\[?223\.12\.34\.56\]?$/                      REJECT
#/^(.+\.)?example\.com$/                        REJECT

Appendix B. Script for extracting rejection records
   Here is a shell script for extracting rejection records from Postfix mail logs. If your computer combines a web server with a mail server, you can easily watch rejection records with a web browser by means of installing this script together with a password under a directory under the cgi-bin directory. If necessary, alter the log file names in the script according to your system environment or a period of time you want to watch.

Note   An improved shell script is published at:
http://www.gabacho-net.jp/en/anti-spam/log-sorting-script.html
It sorts rejection records so that retry accesses are arranged in a sequence.

(Plain text version; long lines are folded)
#!/bin/sh
echo "Content-Type: text/plain"
echo
echo "Mail rejection log"
echo
cat /var/log/maillog.4 /var/log/maillog.3 /var/log/maillog.2 \
    /var/log/maillog.1 /var/log/maillog | \
grep 'reject:' | \
gawk '
BEGIN {
  count=0
}
{
  printf "%s %2d %s %s\n", $1, $2, $3, substr($0, match($0, /reject:/))
  ++count
}
END {
  print "\ncount =", count
}
'

(HTML <PRE> tag version; long lines should be horizontally scrolled)
#!/bin/sh
echo "Content-Type: text/html"
echo
echo "<html><body><pre>"
echo "Mail rejection log"
echo
cat /var/log/maillog.4 /var/log/maillog.3 /var/log/maillog.2 \
    /var/log/maillog.1 /var/log/maillog | \
grep 'reject:' | \
gawk '
BEGIN {
  count=0
}
{
  printf "%s %2d %s %s\n", $1, $2, $3, substr($0, match($0, /reject:/))
  ++count
}
END {
  print "\ncount =", count
}
' | \
gawk '
{
  gsub(/</, "\\&lt;")
  gsub(/>/, "\\&gt;")
  print
}
'
echo "</pre></body></html>"
top report presentation (PDF) article (Ja.) white list black list watch tool applyers Q&A blog (Ja.) links history contact