Haraka and recipient verification

[sriccio]

Hello Haraka community,

I already use Haraka for some different projects and I love it.

For a new incoming mail filtering gateway project, I decided to use Haraka instead of postfix and I have a question about recipient verification.

The project is quite simple, I have a few mail servers of different kind and would like to use the filtering gateway I'm building in front of them.

To begin with, I used relay_force_routing to declare which domains should be routed to which target server

Then, one of the important component I need for this project is to drop the recipient if the target RCPT TO does not match an existing mailbox on the target server for the domain.

Like for example a mail to [email protected] would be rejected if validtargetdomain.com is a ... valid target domain, but the mailbox doesn't exists on the target server.

At first I thought the haraka-plugin-recipient-routes plugin would fulfil this, and it partly does.
Somehow it feels like this plugin does almost the same thing than relay_force_routing but with more granularity (at mailbox level instead of domain) or I am missing something here ?

Still, I wrongly imagined that the plugin would verify that the mailbox really exists by first connecting to the target server and checking if the mailbox exists, by doing a MAIL FROM / RCPT TO to do the check.

That's what I currently use with another setup using postfix, using the address verification procedure:
https://www.postfix.org/ADDRESS_VERIFICATION_README.html#recipient

I'm looking to achieve the same with Haraka. Before reinventing the wheel, Is that something already existing in a plugin ?

I guess if it doesn't exists yet, maybe I should write a plugin, maybe using haraka-plugin-recipient-routes as base and add on-the-fly recipient verification like postfix is doing with its adress verification feature (with some local caching, maybe in redis ?).

I would really appreciate any comments, hints, questions :)

Thanks a lot.
Kind regards

[msimerson]

Somehow it feels like this plugin does almost the same thing than relay_force_routing but with more granularity (at mailbox level instead of domain) or I am missing something here?

You have exactly described what haraka-plugin-recipient-routes does. It validates full email addresses.

Still, I wrongly imagined that the plugin would verify that the mailbox really exists by first connecting to the target server and checking if the mailbox exists, by doing a MAIL FROM / RCPT TO to do the check.

That is correct. That plugin expects the user to populate either files or a redis DB with the valid email addresses. This was a purposeful design choice because the SMTP next hop was a personal appliance that was "thinly" connected to the internet. Think: user devices with non-static IPs. Doing a SMTP callout to verify users would have been problematic.

Extending the plugin to support learning of email addresses, caching the results to redis, and then having some robust logic for expiring / re-testing email address so the validation data doesn't get too stale would be a good approach.

[sriccio]

Hello @msimerson,

Thanks a lot for the feedback.

I'm going to give a try at extending the plugin. I'm not a JS/Node king though. I think I can handle the redis caching part, but I'm not sure yet how to handle the on-the-fly probe to the destination server.

For example not sure yet if there is already some built-in Haraka function I could re-use to trigger a connect + EHLO + MAIL FROM + MAIL TO the host and grab the result, or if I should directly use some NodeJS lib for this.

Kind regards

[sriccio]

For example, to handle the expiration, I could rely on redis EXPIRE thingy.

https://redis.io/commands/expire/

When adding a key I would set a (configurable in .ini) expiration time, also the expiration time could dependi if it's a negative or positive result.
It would be then clean itself with redis passive and active expiration (see "How Redis expires keys" on expire doc page).

What do you think about it?

[msimerson]

Using the redis expire keys is exactly how I would do it. What the positive and negative cache times should be will have a number of tradeoffs and I have no experience there. I'd start by using the values that postfix uses, as they have already seen real world usage.

For smtp client code, have a look at using ./smtp_client.js in the project and the callers that are using it.

[sriccio]

Thanks for confirming I'm heading in the right direction with the redis caching and for the hint about smtp_client.js.

Now I'm asking myself if I should add this directly to the existing haraka-plugin-recipient-routes plugin or create a new plugin, let's say haraka-plugin-recipient-routes-probe or something like this that would be an extension that only handle the probing and only populate entries in the redis db used by haraka-plugin-recipient-routes.

I would say the second option is better, but I'm not sure I can achieve it without any modification of haraka-plugin-recipient-routes

[sriccio]

Hello,

So I'm working on an attempt at using smtp_client to probe the target mx server for the recipient address, but I'm facing an issue where the smtp_client seems to be running async so my function returns a result before the probing has ended.
That must be because I'm a noob at NodeJS.

Here I'm calling a function that should return true or false if the recipient exists on target server
https://gist.github.com/sriccio/fd79e3040bc38c514fbb404d38628af7#file-index-js-L129

In the called function I use smtp_client to do the probe, but it returns the value of success without waiting for a result
https://gist.github.com/sriccio/fd79e3040bc38c514fbb404d38628af7#file-index-js-L63

Maybe do you have a little hint about what I'm doing wrong here... Learning NodeJS on-the-fly here.

Other than that, on the target server, I can see the probe at work, that's already something :)

VERY ALPHA CODE WARNING:
These are first attempts, a lot of stuff that should be dynamic is hardcoded atm, for the tests :)

Kind regards

[msimerson]

That must be because I'm a noob at NodeJS.

Yep. There are two types of functions in async languages, sync and async. Any time a function does an asynchronous event in node, you have to await the results of the function. In older versions of JS, that was done with callbacks. Nowadays it's done with the async/await keywords.

[sriccio]

Hello :)

Yeah, yesterday I've read a bit about callbacks, promises, async/await. I'm still a bit lot yet.

I'm trying to figure out how to call the smtp_client.get_client_plugin function from my probe_recipient function and have my function to wait for a result (when it receive a 250 on RCPT TO) to return true or false to the calling function.

I'm not yet 100% confident with how to do it the cleanest way possible :)

[sriccio]

Hello @msimerson,

I came up with a first version of the plugin, if you want to take a look at it :)

https://github.com/sriccio/haraka-plugin-recipient-routes-probe

It probably lacks some stuff but it is working quite well in my setup for now.