Unacceptable email security holes exploited by criminals for profit.
June 3, 2023
by Andrew G. Watters
Note: rather than embarrassing anyone at UCLA, the purpose of this page is to protect the public, as there is no other writeup on the internet about these extremely sophisticated email scams.
After months of emails to the UCLA I.T. department, CISO, and even the Chancellor asking them to fix this, I finally had the last straw happen-- yet another scam email from the UCLA email server. This one wasn't particularly upsetting, it just pissed me off that it used Salesforce CRM to track replies...in other words, the offshore criminals who sent it will have a proper dashboard and analytics! Useful for tracking which victims actually engage with the email. Here it is:
Return-Path:
Delivered-To: raellic@mail.andrewwatters.com
Received: from mail.andrewwatters.com
by mail.andrewwatters.com with LMTP
id U8wHC7EBe2SQjzsAHMUQ+Q
(envelope-from )
for ; Sat, 03 Jun 2023 02:02:41 -0700
Received-SPF: Softfail (mailfrom) identity=mailfrom; client-ip=169.232.46.176; helo=tx2-csb1.smtp.ucla.edu; envelope-from=adnoc=adnocaevendors.org__4acntnu18tre87ni.u1sjly5f76jwz5e6@9nq5h7incjr651do.chhjr.8e-qy85eac.um9.bnc.salesforce.com; receiver=
DMARC-Filter: OpenDMARC Filter v1.4.2 mail.andrewwatters.com 45CBAC00E69D
Authentication-Results: mail.andrewwatters.com; dmarc=none (p=none dis=none) header.from=adnocaevendors.org
Authentication-Results: mail.andrewwatters.com; spf=fail smtp.mailfrom=9nq5h7incjr651do.chhjr.8e-qy85eac.um9.bnc.salesforce.com
DKIM-Filter: OpenDKIM Filter v2.11.0 mail.andrewwatters.com 45CBAC00E69D
Received: from tx2-csb1.smtp.ucla.edu (tx2-csb1.smtp.ucla.edu [169.232.46.176])
by mail.andrewwatters.com (Postfix) with ESMTPS id 45CBAC00E69D
for ; Sat, 3 Jun 2023 02:02:35 -0700 (PDT)
Received: from smtp08-lo2-sp1.mta.salesforce.com (smtp08-lo2-sp1.mta.salesforce.com [161.71.6.231])
by mx-csb1.smtp.ucla.edu (8.15.2/8.15.2/Debian-8) with ESMTPS id 3538sFDd015960
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT)
for ; Sat, 3 Jun 2023 01:54:18 -0700
Authentication-Results: mx2-lo2-sp1.mta.salesforce.com x-tls.subject="/C=US/ST=California/L=San Francisco/O=salesforce.com, inc./OU=0:app;1:lo2;2:lo2-sp1;3:um9;4:prod/CN=um9-app1-44-lo2.ops.sfdc.net"; auth=pass (cipher=ECDHE-RSA-AES256-GCM-SHA384)
Received: from [10.161.225.40] ([10.161.225.40:51494] helo=um9-app1-44-lo2.ops.sfdc.net)
by mx2-lo2-sp1.mta.salesforce.com (envelope-from )
(ecelerity 4.4.0.19839 r(msys-ecelerity:tags/4.4.0.0^0)) with ESMTPS (cipher=ECDHE-RSA-AES256-GCM-SHA384
subject="/C=US/ST=California/L=San Francisco/O=salesforce.com, inc./OU=0:app;1:lo2;2:lo2-sp1;3:um9;4:prod/CN=um9-app1-44-lo2.ops.sfdc.net")
id A8/85-39597-4BFFA746; Sat, 03 Jun 2023 08:54:12 +0000
Date: Sat, 3 Jun 2023 08:54:12 +0000 (GMT)
From: Abu Dhabi National Oil Company
Sender: noreply@salesforce.com
Message-ID: <2Tg8z000000000000000000000000000000000000000000000RVO62C00cPmCX-Y3SNCHAkA6WuoJlw@sfdc.net>
Subject: "Invitation for "Consulting And Specialty Services"
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_16103_362624665.1685782452894"
X-Priority: 3
X-SFDC-LK: 00D8e000000Qy85
X-SFDC-User: 0058e000001ZPh3
X-Sender: postmaster@salesforce.com
X-mail_abuse_inquiries: http://www.salesforce.com/company/abuse.jsp
X-SFDC-TLS-NoRelay: 1
X-SFDC-Binding: 82eopsuF2FE0HK7B
X-SFDC-EmailCategory: singleEmailMessage
X-SFDC-Interface: internal
X-Probable-Spam: no
X-Spam-Hits: 3.547
X-Spam-Score: ****
X-Spam-Report: DCC_CHECK,FROM_FMBLA_NEWDOM14,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MISSING_HEADERS,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
X-Scanned-By: MIMEDefang 2.79 on 169.232.46.172
------=_Part_16103_362624665.1685782452894
Content-Type: multipart/alternative;
boundary="----=_Part_16102_984150457.1685782452894"
------=_Part_16102_984150457.1685782452894
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Greetings of the day,
We are inviting your esteemed company for vendor registration and intending partners for Abu Dhabi National Oil Company (ADNOC) 2023/2024 projects.
These projects are open for all companies around the world, if you have intention to participate in the process, please confirm your interest by asking for Vendor Questionnaire and EOI.
We appreciate your interest in this invitation, and look forward to your early response.
Kind Regards,
Mr. Mohamed Ghazi B.
Senior Project Manager
[snip]*
*HTML code deleted for readability.
This one follows years of fake emails from UCLA seeking donations and funding for various programs. Why did this one irritate me so much? Because it breaks my heart having to block the UCLA email server, which I just did. But let's start at the beginning.
In 1998, I graduated high school and went to UCLA. I was so proud to have my own email address; I was the earliest adopter among all my friends, and I proudly listed awatters@ucla.edu on my personal web page (long since lost to history). The coolest part about the UCLA email system back then was that you didn't have to have any subdomains on the UCLA domain. In other words, it was a very easy to type and easy to remember address.
I graduated in 2002. I forgot exactly what happened, but at some point in the 2007 time frame, I think I requested that my original UCLA address forward to my current email address, andrew@andrewwatters.com. I'm not sure why they allow this with hundreds of thousands of alumni, or why they can, but that's not the point. The point is that I enjoy having a UCLA alumni address that is identical to my undergraduate address, even though no one has ever used it to send me legitimate email in the last 20+ years (lol).
In 2022, when I fully transitioned my email to my own system after several years of using a mix of my own equipment and corporate Gmail, I noticed something unusual: (1) I get a lot of emails purporting to be from my own UCLA address to myself, and (2) there are a lot of fundraising emails claiming to be from new UCLA programs and seeking donations. (1) is impossible since my UCLA email is a forwarding-only address, and (2) turns out to be ongoing scams that were previously blocked on Gmail, but which manage to get through to me because I take the "raw and uncut" view on email. I don't have a spam folder specifically to avoid other lawyers' typical b.s. excuses that some critically important email "must have gone to his spam folder." That does not happen; either they get a bounce message or the message is delivered to my inbox, which is the intended behavior. I will never go back to Gmail for this and other reasons, such as the fine-grained customization and control that are possible when running my own system.
Anyway, I reached out to the CISO of UCLA and complained about this in Fall 2022. When the problem wasn't fixed in three months, I reached out again in March 2023. I had a little fun at his expense by spoofing an email from the Chancellor of UCLA to the CISO stating that the CISO was being fired:
[ps354511]$ openssl s_client -starttls smtp -connect mx.smtp.ucla.edu:25
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
verify return:1
depth=0 C = US, ST = California, O = "University of California, Los Angeles", CN = mx-asm1.smtp.ucla.edu
verify return:1
---
Certificate chain
0 s:C = US, ST = California, O = "University of California, Los Angeles", CN = mx-asm1.smtp.ucla.edu
i:C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
1 s:C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
3 s:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGwTCCBamgAwIBAgIQSJTtKdXF02jQP2oaG01uVTANBgkqhkiG9w0BAQsFADB2
MQswCQYDVQQGEwJVUzELMAkGA1UECBMCTUkxEjAQBgNVBAcTCUFubiBBcmJvcjES
MBAGA1UEChMJSW50ZXJuZXQyMREwDwYDVQQLEwhJbkNvbW1vbjEfMB0GA1UEAxMW
SW5Db21tb24gUlNBIFNlcnZlciBDQTAeFw0yMjEyMDUwMDAwMDBaFw0yNDAxMDUy
MzU5NTlaMHIxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMS4wLAYD
VQQKEyVVbml2ZXJzaXR5IG9mIENhbGlmb3JuaWEsIExvcyBBbmdlbGVzMR4wHAYD
VQQDExVteC1hc20xLnNtdHAudWNsYS5lZHUwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC8N9Od0VmXahnBt1A3DXSS7AN6WpD1L238+06Je0eMcffmrq2t
z2NMgnSInc9iwDv/C7DKcFxhBEkGRoRADkQYLgqeDP6YY8pUVRWUk4ymcB1gijFv
6zeKOSFXWPPDMqzyllGAy6Ymc4J5EGIxJle9/S9gUxiO7gCz+Btalb7Ibrd7g6li
p01RtjAqm5ljI7GWEDrwGjdLAmFnhD+rU61Y79rurPGlkHNFqCmddQ4gKz2ZqgdU
N3euFx2X14cHf4fKygEy/J7iDBrony3U1uFNX+/Te6xULW3LanKvcTk7ekUT/GpJ
jMTYjr3JORS1z5eBU30xQ49VIAAQJxwvedOfAgMBAAGjggNNMIIDSTAfBgNVHSME
GDAWgBQeBaN3j2yW4luHS6a0hqxxAAznODAdBgNVHQ4EFgQUnIopFSzlJlgmNmpy
FqJjS+z5yLAwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMGcGA1UdIARgMF4wUgYMKwYBBAGuIwEEAwEB
MEIwQAYIKwYBBQUHAgEWNGh0dHBzOi8vd3d3LmluY29tbW9uLm9yZy9jZXJ0L3Jl
cG9zaXRvcnkvY3BzX3NzbC5wZGYwCAYGZ4EMAQICMEQGA1UdHwQ9MDswOaA3oDWG
M2h0dHA6Ly9jcmwuaW5jb21tb24tcnNhLm9yZy9JbkNvbW1vblJTQVNlcnZlckNB
LmNybDB1BggrBgEFBQcBAQRpMGcwPgYIKwYBBQUHMAKGMmh0dHA6Ly9jcnQudXNl
cnRydXN0LmNvbS9JbkNvbW1vblJTQVNlcnZlckNBXzIuY3J0MCUGCCsGAQUFBzAB
hhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMCAGA1UdEQQZMBeCFW14LWFzbTEu
c210cC51Y2xhLmVkdTCCAYAGCisGAQQB1nkCBAIEggFwBIIBbAFqAHcAdv+IPwq2
+5VRwmHM9Ye6NLSkzbsp3GhCCp/mZ0xaOnQAAAGE407wqQAABAMASDBGAiEAut8j
fYu2kI1piQo160upin9bE7Be8/JFgfbmmWAhM0oCIQDuieyRvAQ7eZ5n4VJihO+Q
fMh+S5wb4WGs3nIvPZ3F4gB3ANq2v2s/tbYin5vCu1xr6HCRcWy7UYSFNL2kPTBI
1/urAAABhONO8H4AAAQDAEgwRgIhAKEN3gP+OXR+cuaeQCurbp6JB+M+bv0t4xcX
sBeFKVo6AiEAveUvYSAf2WB29S+BakyARkrpWI5uS3RxHndOI1q874YAdgDuzdBk
1dsazsVct520zROiModGfLzs3sNRSFlGcR+1mwAAAYTjTvBBAAAEAwBHMEUCIAb5
+rLnJm0vXtxY+5zENsvm4Ry4jI2MGJcW/pXk1XLnAiEAgAcodKpqTfJvpaChZojU
KfllLuJHM3eq/2bBWoavssUwDQYJKoZIhvcNAQELBQADggEBAFiM8V2xoifgmmGf
KcH8KqAZyNd8EvE2C5LadS0if/ySNOlEdhz6oquU1+do2XeK0/BGqL5H81Zy/hFp
6u8oR+UeMlQGf3RVWWZVlP50/8xJ6Gt6XCxAxnNrlZ8UDuevAJIDf4/Aasrlt/uO
jam6Gg7KEWe9cU669WdZyYvEQWxgvs4/8lL+Sas8S5Fx8eL193gXhUnOANTtp3IT
QC82xjVWrkCEpioSD/LxEsgM75JbnOepDwjna3oBeOwbmI0k8+kspLqvksFN16Md
rj1qMYoNOMC4R4jgOW9MdKQs1u+m9Sd2dkAQngCJ9d0ucFNBNFLoYpF6VGm8ZjH3
DmcHg+Y=
-----END CERTIFICATE-----
subject=C = US, ST = California, O = "University of California, Los Angeles", CN = mx-asm1.smtp.ucla.edu
issuer=C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 6907 bytes and written 467 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: D6BD0CFA8558F39BC9A45EB11495848FC910019CCE447A81BBCF1EB2C0878FAA
Session-ID-ctx:
Master-Key: 3899C88D99B0DFE2E5E3B3083501085422F89F580FF3FC0B5A84602E1ACCB799271815252E1B6EEF9AE4EF03A0B18082
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 1 (seconds)
TLS session ticket:
0000 - 14 a0 25 27 12 0d f9 80-cd 5f 5c fd 16 40 b1 c6 ..%'....._\..@..
0010 - 24 11 c1 a7 28 72 be 51-89 88 bd 57 79 08 cf c7 $...(r.Q...Wy...
0020 - 0e 9a 4c 4c e7 3a c5 f1-96 c0 57 14 95 29 89 c2 ..LL.:....W..)..
0030 - 85 b3 80 73 2c 6b 01 77-55 a6 ad b4 5a 9f 7e 05 ...s,k.wU...Z.~.
0040 - bc f7 77 cb 69 de 35 bd-d6 2b ce 19 f3 49 fd 21 ..w.i.5..+...I.!
0050 - 48 18 4a 50 93 90 dc 74-8c 2f 17 87 b8 69 e5 bf H.JP...t./...i..
0060 - b3 6d d4 73 48 c7 2f b9-8a bf 37 47 a5 35 42 68 .m.sH./...7G.5Bh
0070 - df e0 ec 6e b8 17 b4 a3-c1 ed 9e e1 16 f7 fb ce ...n............
0080 - 50 96 e2 27 a9 a2 ac 9c-ad 03 3c 7f 9e bb 4d 6b P..'......<...Mk
0090 - e2 4d 83 e6 df c8 5a 20-0b ff fb cb 90 26 5b 5f .M....Z .....&[_
00a0 - 27 f6 b6 73 a2 aa 1b c2-fc 5d 9a 0d d9 12 1a 3e '..s.....].....>
00b0 - 89 0f 54 26 ee 2c e0 69-26 a8 a0 c5 ee 94 e8 81 ..T&.,.i&.......
Start Time: 1679434834
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: yes
---
250 HELP
helo ps354511.dreamhostps.com
250 mx-asm1.smtp.ucla.edu Hello ps354511.dreamhostps.com [69.163.233.88], pleased to meet you
mail from: chancellor@ucla.edu
250 2.1.0 chancellor@ucla.edu... Sender ok
rcpt to: chancellor@ucla.edu
250 2.1.5 chancellor@ucla.edu... Recipient ok
rcpt to: djshaw@ucla.edu
250 2.1.5 djshaw@ucla.edu... Recipient ok
data
354 Enter mail, end with "." on a line by itself
From: Gene Block
To: David Shaw
Subject: Notice of Personnel Action
Dear David,
Thank you for your service to the UCLA community as Chief Information Security Officer.
As part of a normal restructuring of operations, your position is being eliminated, effective immediately.
Please refer to our assigned counselor in the event you feel this decision is not optimal for the UCLA community and yourself. I wish you the best in your future endeavors.
Best,
Gene Block
Chancellor
.
250 2.0.0 32LLeW4W025827 Message accepted for delivery
quit
221 2.0.0 mx-asm1.smtp.ucla.edu closing connection
closed
[ps354511]$
That was the only way to get my point across, and I made my point. But nothing was done. A couple months later, in May 2023, the UCLA email server still does not enforce HELO restrictions or have proper SPF or DMARC settings. As a result, anyone can connect to it, pretend to be a UCLA sender, and send within UCLA or externally. Check this email out from a couple days ago:
Return-Path:
Delivered-To: raellic@mail.andrewwatters.com
Received: from mail.andrewwatters.com
by mail.andrewwatters.com with LMTP
id eTrhDtH9eGTIqzoAHMUQ+Q
(envelope-from )
for ; Thu, 01 Jun 2023 13:21:37 -0700
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=169.232.46.187; helo=tx1-csb2.smtp.ucla.edu; envelope-from=chancellor@ucla.edu; receiver=
DMARC-Filter: OpenDMARC Filter v1.4.2 mail.andrewwatters.com ED479C00E6A6
Authentication-Results: mail.andrewwatters.com; dmarc=none (p=none dis=none) header.from=ucla.edu
Authentication-Results: mail.andrewwatters.com; spf=pass smtp.mailfrom=ucla.edu
DKIM-Filter: OpenDKIM Filter v2.11.0 mail.andrewwatters.com ED479C00E6A6
Received: from tx1-csb2.smtp.ucla.edu (tx1-csb2.smtp.ucla.edu [169.232.46.187])
by mail.andrewwatters.com (Postfix) with ESMTPS id ED479C00E6A6
for ; Thu, 1 Jun 2023 13:21:35 -0700 (PDT)
Received: from ps354511.dreamhostps.com (ps354511.dreamhostps.com [69.163.233.88])
by mx-csb2.smtp.ucla.edu (8.15.2/8.15.2/Debian-8) with SMTP id 351KGBSF014314
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Thu, 1 Jun 2023 13:16:33 -0700
Date: Thu, 1 Jun 2023 13:16:33 -0700
Message-Id: <202306012016.351KGBSF014314@mx-csb2.smtp.ucla.edu>
From: Chancellor Gene Block
To: UCLA Network Operations Center ,
"Andrew G. Watters"
Subject: Notice of personnel action
X-Probable-Spam: no
X-Spam-Hits: 2.039
X-Spam-Score: **
X-Spam-Report: MISSING_DATE,SPF_HELO_NONE,SPF_NEUTRAL,T_SCC_BODY_TEXT_LINE
X-Scanned-By: MIMEDefang 2.79 on 169.232.46.184
Dear UCLA NOC,
Your positions have been eliminated through a normal restructuring of operations.
Please discontinue your services to the UCLA community immediately, and lock the door on the way out.
Best,
Gene Block
Chancellor
UCLA
The sad part is that the NOC acted like this was novel, which made me concerned that complaints are not being taken seriously. I subsequently received an email from an I.T. department supervisor indicating that the issue was being looked at. While I appreciate that running a large email system at the finest public university in the world must be extremely difficult, these security holes are unacceptable because they enable massive wire fraud and other scams. Some of these scams are so diabolical that I probably haven't even thought of what they might do-- and I'm freaking diabolical (when I want to be). Bottom line, the issues damage the UCLA brand and cause victims to lose money.
I am tired of this, and I shouldn't have to block the entire UCLA domain as I have done. Unfortunately, publicly shaming the UCLA I.T. department is the only solution at this point. Happy forging!
Update November 27, 2024: They finally seem to have fixed this. Only took a year and several months and them switching to Google for email. Lol.
[raellic@andys-workstation ~]$ telnet smtp.google.com 25
Trying 2607:f8b0:4004:c19::1a...
Connected to smtp.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP d75a77b69052e-466c4227dd5si803471cf.446 - gsmtp
helo ps2600.dreamhostps.com
250 mx.google.com at your service
mail from: chancellor@ucla.edu
250 2.1.0 OK d75a77b69052e-466c4227dd5si803471cf.446 - gsmtp
rcpt to: security@ucla.edu
250 2.1.5 OK d75a77b69052e-466c4227dd5si803471cf.446 - gsmtp
data
354 Go ahead d75a77b69052e-466c4227dd5si803471cf.446 - gsmtp
From: Gene Block chancellor@ucla.edu
To: UCLA I.T. Security security@ucla.edu
CC: Andrew Watters andrew@andrewwatters.com
Date: November 27, 2024 14:42:59 PST
Message-ID: 289478924782@ucla.edu
Subject: Notice of Personnel Action
Thank you all for your service to the UCLA community.
As part of a normal restructuring of operations, your positions are being eliminated, effective immediately.
Please refer to our assigned counselor in the event you feel this decision is not optimal for the UCLA community and yourself. I wish you the best in your future endeavors.
Best,
Gene Block
Chancellor Emeritus
.
550-5.7.26 Your email has been blocked because the sender is unauthenticated.
550-5.7.26 Gmail requires all senders to authenticate with either SPF or DKIM.
550-5.7.26
550-5.7.26 Authentication results:
550-5.7.26 DKIM = did not pass
550-5.7.26 SPF [ucla.edu] with ip: [2001:550:2:b::56:2] = did not pass
550-5.7.26
550-5.7.26 For instructions on setting up authentication, go to
550 5.7.26 https://support.google.com/mail/answer/81126#authentication d75a77b69052e-466c4227dd5si803471cf.446 - gsmtp
Connection closed by foreign host.
Last thing I need to do is try this from the UCLA campus. That way I'm on their network and SPF should work properly.