regex - Logstash grok filter to tag received and bounced messages

Question

Sthg makes me crazy, I would like to parse Postfix logs to know the status of emails, here is what I tried so far :

input {
   file {path => "/var/log/mail.log"}
}

filter {
    kv {
        trim => "<>"
    }

    if [message] =~ /[ "status=bounced" ]/ {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => {"message" => "%{SYSLOGBASE} (?<QID>[0-9A-F]{10}): %{GREEDYDATA:message}"}
            add_tag => "bounce"
        }
    }

}
output {
   if "bounce" in [tags] {
      stdout { codec => rubydebug }
   }
}

---

Example of mail.log :

Jul 26 04:18:34 mx12 postfix/cleanup[20659]: 3mfHGL1r9gzyQP: message-id=<3mfHGL1r9gzyQP@www.mydomain.fr>

Jul 26 04:18:34 mx12 postfix/smtp[20662]: 3mfHGL1r9gzyQP: to=, relay=127.0.0.2[127.0.0.2]:25, delay=0.53, delays=0.13/0/0.23/0.16, dsn=2.0.0, status=sent / bounced

---

Result 1 :

I send an email to an existing email address, the status in mail.log is :

sent (250 ok) : OKAY

But here is what Logstash tells :

.. and I see that for every message generated by each postfix program (qmgr, smtp, qmgr again..). In other words, for all messages that even not contain "status=bounced".

Then I also tried :

   if [message] =~ /[ "bounced" ]/ {
     mutate {add_tag => [ "bounce" ]}
  }

  if [message] =~ /[ "message-id", "(.*)@www.mydomain.fr" ]/ {
      mutate { add_tag => [ "send" ] }
  }
  grok {
       match => {"message" => "%{SYSLOGBASE} (?<QID>[0-9A-F]{10}): %{GREEDYDATA:message}"}
  }

Result 2 : Logstash add here always 2 tags : bounce + send :(

---

Result expected :

What I try to do is exactly this config file, but it was made with an old version of Logstash ("grep" for example is not available now), but this is exactly what I try to make working :

http://tales.itnobody.com/2013/07/using-logstash-to-log-smtp-bounces-like-a-boss.html

In one word :

1. Any entries with a DSN – RECORD: QID, dsn
2. Any entries matching message-id=< hashRegex > – RECORD: QID, message-id

As follow :

    output{
       if "bounce" in [tags] {
           exec {
                command => "php -f /path/LogDSN.php %{QID} %{dsn} &"
           }
       }

       if "send" in [tags] {
           exec {
                command => "php -f /path/LogOutbound.php %{QID} %{message-id} &"
           }
       }
     }

But there is a problem in my filter, that makes me crazy,

Any idea ??

Answer 1

I have found the problem.

It's coming from this test:

if [message] =~ /[ "bounced" ]/ {
  mutate {add_tag => [ "bounce" ]}
}

The regex is the part between the /, so your regex is evaluated like that : https://regex101.com/r/eaB5jp/2

So all your lines will match and get the tag.

In order to work, the test should be:

if [message] =~ /bounced/ {
  mutate {add_tag => [ "bounce" ]}
}