Google / Gmail. Seems like I have to disable harvesting -otherwsie we block gmail. Is there no other way around this ?

Problem reported by Curtis Kropar www.HawaiianHope.org - Today at 12:43 AM

Submitted

I am just SOO irritated recently, and its not just the spammers, but also clients that ultimately shoot themselves in the foot with Google and gMail.

Is there any way around this or do we just have to disable the SMTP Harvesting security ?

We have our server set to block IP Addresses that send too many emails to accounts that don't exist on our server - "Harvesting" right ?\

I constantly get users complaining that they are not getting emails from gMail. So I try to figure out what IP Addresses gMail is using to send from and see if "it" is blocked... well it is not an "it" IP address. This is the current list of all of the IP address RANGES that google uses.

https://www.gstatic.com/ipranges/goog.json

I simply cant keep up. This list represents tens of millions of IP Addresses that google uses.

What has been happening the last 3 months now - is a domain admin deleted the email address of a person that was with the company for 25+ years. So there are thousands of people that have this persons email address and they were also part of multiple legitimate email lists from google groups, many of them state related (homeless services).

So now that email address does not exist on our servers, BUT all of these people and email lists still try to send emails to that person. SOOOOO many from gMail that gMail IP Addresses start getting banned by our server. And that ban is server wide, not just a single domain. Yes ?

So our list of 30 or so non profits that we work with and provide hosting for are now not getting emails from various state agencies that use google groups as their mailing lists. The non profits essentially banned their own inbound email because they killed off previously heavily used email addresses and our server now counts all of the legitimate inbound email as harvesting- banned.

So what did the the non profits do to get around this ? Now they created email addresses on gmail - "UserName.Agency@gmail.com" and are having those email addresses forward email to our servers that have a matching username.

This of course just increases the traffic load from google. And if the agency kills off a user on our server without killing off the matching gmail account they created (it happened) we get google and gmail banned on yet another IP Address. Meanwhile the non profits are criticizing us that we are blocking their email from state agencies.

It seems like If I want my users to get email from google / gMail / google groups, etc, my only recourse is to disable that "feature" of banning spammers - sending to email addresses that do not exist.

I don't know how else to fix this.

Any ideas ?

www.HawaiianHope.org - Providing technology services to non profit organizations, low income families, homeless shelters, clean and sober houses and prisoner reentry programs. Since 2015, We have refurbished over 11,000 Computers !

2 Replies

Reply to Thread

Douglas Foster Replied

Today at 4:35 AM

Here is my solution:

I have an incoming gateway based on SmarterMail + Declude

Recipient Verification is disabled in Declude.

A Declude custom filters calls a Python script.

The python script parses the .HDR file and performs Recipient Autentication.

Messages with no valid recipients are flagged with a bitmap result, but some basic message metadata is logged to a database.

The bitmap result triggers a Declude DELETE action.

Declude is configured to stop processing on first delete.

My first attempt used the SmarterMail API to check for invalid recipients, but the lookup volume was high. My Administrative log was filling with login-logout messsages, which was a clue that I was creating a DoS attack on my server. The solution was to extract valid users to a database table and do the lookups against the database list. This is about 65% of all incoming messages.

I have tried reviewing the invalid sender metadata to determine who should and should not be blocked, but the data is overwhelming. But it blocks the spammers while minimizing the processing cost for discard their messages.

Douglas Foster Replied

Today at 7:07 AM

(My code available to anyone on request - just send a private message with your email address so we can go offline)

The problem with your approach is that it does not distinguish between:

legitimate traffic to terminated accounts, (optimal treatment is unsubscribe),
spam to terminated accounts, and
directory harvesting attacks to non-existent accounts.

Additionally, you have the problem that some of the directory harvesting comes from infrastructure IP addresses, such as SendGrid. That is why rapid discard makes sense, but blocking is more difficult.

In my environment, terminated employees are initially disabled but not removed from SmarterMail. My account extract process marks the corresponding database entry as invalid without removing it. That begins to build a list of formerly-valid accounts that could be used for this analysis in the future. But since we still get email for employees terminated 10 years ago, the former-employee list will never be complete.

An aside on Google Groups:

I have a recurring problem with fake GoogleGroups being used as an open relay attack vector, so I quarantine GoogleGroups by default and only allow traffic after I have confirmed that a real user intended to subscribe to a group. It would also be appropriate to verify with the domain owner that group subscription is consistent with organizational policy for their staff.

Back to Community Threads

Please leave this box unchecked

Reply to Thread

Enter the verification text