Random Linux and macOS tips and tricks

This problem rarely happens, but sometimes your web interface for your Asus iKVM or Supermicro IPMI will show "session expired" immediately, and then log you out immediately. I checked, and apparently this is due to the BMC running out of space when the System Event Log reaches the capacity of the BMC's limited storage. The problem is what to do when this happens when you are not on-site. Here's your solution: issue IPMI commands to the BMC from within your network to reset the BMC, and then boot the machine like you were trying to do. This achieves several goals, starting with not having to go on site and unplug the power cables (lol). First, log in to your network where you can see the BMC IP address on your LAN. You should already have configured an IP for the BMC, but if not, use Nmap to determine which IP is the BMC. You should get Nmap output showing these ports from an Asus BMC:

PORT      STATE    SERVICE
22/tcp    open     ssh
80/tcp    filtered http
427/tcp   open     svrloc
443/tcp   open     https
49154/tcp open     unknown

ipmitool -U admin -P [IPMI password] -H 192.168.1.6 bmc info

ipmitool -U admin -P [IPMI password] -H 192.168.1.6 bmc reset cold

Device ID                 : 32
Device Revision           : 1
Firmware Revision         : 1.32
IPMI Version              : 2.0
Manufacturer ID           : 2623
Manufacturer Name         : Unknown (0xA3F)
Product ID                : 3971 (0x0f83)
Product Name              : Unknown (0xF83)
Device Available          : yes
Provides Device SDRs      : yes

ipmitool -U admin -P [IPMI password] -H 192.168.1.6 power on

ssh -L 8484:192.168.1.7:22 [username]@www.andrewwatters.com

Add to /etc/postfix/header_checks:

/^From:.{0,2}((Andrew Watters)|(Andrew G\. Watters)|("Andrew Watters")|("Andrew G\. Watters"))(?!((.{0,3}andrew@andrewwatters\.com.{0,3})|(.{0,3}andrew@raellic\.com.{0,3}))).{0,256}$/ REDIRECT raellic@mail.andrewwatters.com

This is a "negative look-ahead" that matches my name when NOT followed by my actual email addresses. It redirects scam emails claiming to be me to my own account, so that the messages don't even go to my employees. This has been an issue recently and I finally resolved it, I think. You also should change /etc/postfix/main.cf to only apply this header check to the main message, not any attachments. That way you can forward the scam messages to warn people or to abuse@google.com:

header_checks = pcre:/etc/postfix/header_checks
nested_header_checks =
mime_header_checks =

sudo find /var/log/httpd -name "andrew*" -exec grep -P "GET \/ " {} \; | grep -v "bot" | grep "[0-3][0-9]\/Feb" | sort -t ' ' -k 4 | awk -F ' ' 'NR>0{arr[$1]++}END{for (a in arr) print a, arr[a]}' | sort -t ' ' -k 1

Super handy for figuring out which anonymous visitors visit the index page of your website the most. This one finds all unique visitors for February and shows view counts. Change as needed.

curl https://www.nytimes.com/2023/08/06/world/europe/putins-forever-war.html | lynx -stdin

montage -tile 3x11 -verbose -geometry 30% -border 10 -label '%f %wx%h %C %Q\%\n%[EXIF:Model]' *.JPG sheet.jpg

This takes a folder full of images, resizes them to 30% of original size, labels them with filename, dimensions, compression algorithm, quality level, and camera make/model, and then creates a tiled contact sheet. Voila:

cat ../MIOG-pages.txt | awk -v RS="<page>" 'NR > 1 { print RS $0 > "MIOG-" (NR-1) ".php"; close("MIOG-" (NR-1) ".php") }'

This splits a file based on chunks of text that start with "<page>" and creates as many individual files as there are pages. Very handy!

When sending email to people who don't disclose their email addresses online, I often will guess their address and then test it with Telnet to port 25 of their email server. This has several downsides, such as using plaintext that anyone can see and that leaves a permanent record of what data were transmitted. So now I use OpenSSL to do this, which has the added benefit of giving security and configuration information about the subject email server. Check out this session, in which I send email claiming to be from a virtual private server to my own email server:

awatters@Andrews-MBP-3 ~ % openssl s_client -starttls smtp -connect mail.andrewwatters.com:25
CONNECTED(00000005)
depth=3 C = PL, O = Unizeto Technologies S.A., OU = Certum Certification Authority, CN = Certum Trusted Network CA
verify return:1
depth=2 C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA
verify return:1
depth=1 C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com RSA SSL subCA
verify return:1
depth=0 CN = andrewwatters.com
verify return:1
---
Certificate chain
 0 s:/CN=andrewwatters.com
   i:/C=US/ST=Texas/L=Houston/O=SSL Corporation/CN=SSL.com RSA SSL subCA
 1 s:/C=US/ST=Texas/L=Houston/O=SSL Corporation/CN=SSL.com RSA SSL subCA
   i:/C=US/ST=Texas/L=Houston/O=SSL Corporation/CN=SSL.com Root Certification Authority RSA
 2 s:/C=US/ST=Texas/L=Houston/O=SSL Corporation/CN=SSL.com Root Certification Authority RSA
   i:/C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Trusted Network CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIIrTCCBpWgAwIBAgIQb39rYSJwY8v6fGWwn1ukwDANBgkqhkiG9w0BAQsFADBp
MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hvdXN0b24x
GDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjEeMBwGA1UEAwwVU1NMLmNvbSBSU0Eg
U1NMIHN1YkNBMB4XDTIzMDEwNTE1MjI1M1oXDTI0MDEwMzE1MjI1M1owHDEaMBgG
A1UEAwwRYW5kcmV3d2F0dGVycy5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw
ggIKAoICAQC4guqvUsdlj2fOPLNEC9BIOaplOmZiZrlqay+8sPX/92G6SN0Vsdkd
7ekmJ/v3JBBC2ugEn2UTgMNz+Ec31rA0/Q1u+da3+apZpjJ2mDu2a0Ap9FllVhF9
YDj3wY/xJU3kz3jzd2SWKrlo0EQnU0bClZ/1Rp4QU+WYUfBtOTHCj5sMhovTAvVl
XM6nEHLK98omvwch/A0vza5m8Uq4xbjNE4wYe6AbNlLX+PfLO6y1cHbKOroZGrdN
k0kjjw433fZ/lS3uOFbLzC1P5dlyz+oq11wXQmf38hFsXO3mlPL9MzBY4w0jTi7C
iT/RhlU+YrTrWIDq8lTnFvvoJ5qe1/mGRZNJD+pkKaGRsNAVgEzrbUKRz2scQmnc
e/1saj6HQKzAi1Pzmv2W58EgXbnZv2DyWuK6E7d2EmiBm1kUh/hlCSPfkcSPmTNt
lgO5U0rGYS1Deu62S1ygtBCT1JuEixOPQJz+YOQfGXASrFzMJjqEmiE9qCBaSX7i
DDd38og/iWhArkCC8CvGgvfF8blrXm9oUsPWuVwT/Cu8qEymrlSTC3lGWqLQKze+
jBZl3c8BiMinUqb6vX4Ff82R6EGQ6KYXsovpuSoler17K794xlpMIrV1Ey1uuHag
XXn15duFr0U98D6vb43GX2gMysxT8yn46mldtyHkFL1NLixpT2AVjwIDAQABo4ID
nDCCA5gwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQmFH7g3Nem9+LUBCffYfHC
7OcyyjByBggrBgEFBQcBAQRmMGQwQAYIKwYBBQUHMAKGNGh0dHA6Ly9jZXJ0LnNz
bC5jb20vU1NMY29tLVN1YkNBLVNTTC1SU0EtNDA5Ni1SMS5jZXIwIAYIKwYBBQUH
MAGGFGh0dHA6Ly9vY3Nwcy5zc2wuY29tMIGGBgNVHREEfzB9ghFhbmRyZXd3YXR0
ZXJzLmNvbYILKi5zbGFzaC5sYXeCEyouYW5kcmV3d2F0dGVycy5jb22CESoudGlu
YXdhdHRlcnMuY29tgg0qLnJhZWxsaWMuY29tghIqLmRldmFocXVhcnRldC5jb22C
ECouamVud2F0dGVycy5jb20wUQYDVR0gBEowSDAIBgZngQwBAgEwPAYMKwYBBAGC
qTABAwEBMCwwKgYIKwYBBQUHAgEWHmh0dHBzOi8vd3d3LnNzbC5jb20vcmVwb3Np
dG9yeTAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwRQYDVR0fBD4wPDA6
oDigNoY0aHR0cDovL2NybHMuc3NsLmNvbS9TU0xjb20tU3ViQ0EtU1NMLVJTQS00
MDk2LVIxLmNybDAdBgNVHQ4EFgQUYh9RmVfj1NzctA7U2odPSq6vtYwwDgYDVR0P
AQH/BAQDAgWgMIIBgAYKKwYBBAHWeQIEAgSCAXAEggFsAWoAdwA7U3d1Pi25gE6L
MFsG/kA7Z9hPw/THvQANLXJv4frUFwAAAYWCkFi9AAAEAwBIMEYCIQD61Zsa2b2k
SdWEn2JUyI18z6uH8knY+LkQhdqWX2Pk8wIhANgt1lcO+hUzn3EEnSEwfnG/P9Wt
h757AUijSDb9C1h0AHYA7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEftZsA
AAGFgpBYvgAABAMARzBFAiA+DXOlmYCPj47t/0Ou/1TBDt+1AuYH0K0ybJjszYnD
4QIhAP4SEhmJJ+Kaq6BnoW5RNdLVQylaPhsAC+mtot3lC6K/AHcAb1N2rDHwMRnY
mQCkURX/dxUcEdkCwQApBo2yCJo32RMAAAGFgpBY6wAABAMASDBGAiEA4oFBS51v
Z4YexduKDdOU3zPKvjTpdyfdsmOgzovKHK4CIQDtgI70lEuGAifbD9cy20jVZbsu
E6B9v8k7icBmE70f3TANBgkqhkiG9w0BAQsFAAOCAgEAaMRjorP8Y7NfSRhTqtC8
yXzVi1u7Z5yFzDesudvn+WFrkzNPrhkLh7voRUlt7WtfpQCQOxXxm9oLhcF2bOl5
YbhJ6XBsZw7qj6VyW8aNi756dfrzjaGdXk+LWcVW8be8zZdY0yE+CsMl+BubaN2H
YsQRdmi5SUhq7Q4AhQxoF0LbqgTU2OH8v3gDFdk8Z+akiV5rbCbR+6WiHsGDHpyY
EE7iftwb8/yc5mXxqckCq+4qA+x0r9mLWv+jbk41zLuPt9JaM3RzxLBi44rTUcKu
vr/H/CKIV0sb8gYF9dC8UcGKEcBV+v7qhSVr3yZuo7Y5L1Bs8bAr0gv6oQqrh8Ti
1VeMq9VnPSoeCbz7cFyrUSMmuYaZ5/7ZWpeRQk13VzqtEoalZWY/MLIfxJlfOoOF
VixpPTJZJm2gb99fbZwhfj8J4sgmeUC2YAbyCzy0m4yuPDzfg8+++kg8FhRV+Dle
3MlBdaB5PG8J9lRHl4WdKkZwG1jtimvKg9tkjK0NXKexcwer2A+Hjt+tShb5xNce
MWJBS4pROOJek271qm+wuyk+L00gCZ2rGIn33OuzewGHV0g9XQpSuXSwtUXtzCbK
dYUI6ngwK3xfn1UvV7EkOBQ71ouHGCZ8UEy7gQ0Opb0NoBr1JzOEeBcXfR+6z10X
n05QrlHjmLWordt1/2tgRwk=
-----END CERTIFICATE-----
subject=/CN=andrewwatters.com
issuer=/C=US/ST=Texas/L=Houston/O=SSL Corporation/CN=SSL.com RSA SSL subCA
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 6482 bytes and written 316 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 8F74B48575BB67C87EC5516294BDBBB55FD7464C6979AF44F07C09F973E47038
    Session-ID-ctx: 
    Master-Key: 079ADB39E794D816C55C8ED0D6AAC3C74A1936256E12FB19C533BF0E04279AFBA2EA5904F4E627463EB63835AFF72AEB
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - d0 b2 36 43 b2 9c 1c d9-e7 68 ff d6 2e 89 fd 75   ..6C.....h.....u
    0010 - f5 19 ba f5 24 d8 37 57-2a da 46 b4 e6 41 6d fe   ....$.7W*.F..Am.
    0020 - 14 1b 05 da 0e b9 31 8f-d7 44 46 86 82 1e 21 d8   ......1..DF...!.
    0030 - 37 3b 70 d8 0e 0e 1d 8b-1e 5b 5b bb 80 c5 51 49   7;p......[[...QI
    0040 - db bd 54 b9 76 55 7e 71-62 7c 3a af 5f 5a 12 99   ..T.vU~qb|:._Z..
    0050 - 70 ba b4 48 fe 2b f5 5c-ef ae ab 20 32 e3 84 f2   p..H.+.\... 2...
    0060 - b4 6c 63 de 05 ac 95 56-d4 ff 49 90 94 8e 47 27   .lc....V..I...G'
    0070 - 0c 11 2e 39 cf 94 b4 65-d4 ab 07 85 c0 94 94 4d   ...9...e.......M
    0080 - 55 f1 69 e7 a5 ed d0 2e-26 d7 97 50 93 e4 b0 ea   U.i.....&..P....
    0090 - 75 0c 6a 1d db 80 1a 64-22 1b 33 91 07 a4 18 06   u.j....d".3.....

    Start Time: 1678885386
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
250 SMTPUTF8
helo ps354511.dreamhostps.com    
250 mail.andrewwatters.com
mail from: test@ps354511.dreamhostps.com
250 2.1.0 Ok
rcpt to: andrew@andrewwatters.com
250 2.1.5 Ok
data
354 End data with .
From: test@ps354511.dreamhostps.com
To: andrew@andrewwatters.com
Subject: test

hello
.
250 2.0.0 Ok: queued as 1BCA1C042476
quit
221 2.0.0 Bye
closed

Interestingly, DreamHost does not have any SPF or DMARC records for this VPS, which means I can impersonate it from any IP address (sad). The chances of someone doing so are low, but it could be easily automated-- they should fix that.

Add to root's crontab:

0 1 * * * tar -cvf /mnt/raid5/storage/backups/mail.tar-`date +\%Y\%m\%d` /mnt/raid5/mail > /mnt/raid5/storage/backups/log 2>&1 ; echo "Email backup completed." | mail -s "Email backup results" andrew@andrewwatters.com
0 5 * * * find /mnt/raid5/storage/backups/mail.tar* -mtime +6 -type f -delete
0 5 * * * sshpass -p "[redacted]" scp /mnt/raid5/storage/backups/mail.tar-`date +\%Y\%m\%d`* [redacted]@[redacted]:~/backups/

tar cf - 'Outlook Data File' -P | pv -s $(du -sb 'Outlook Data File' | awk '{print $1}') | gzip > backup.tar.gz

In vim, it's the following commands:

:s/</\&lt;/gc
:s/>/\&gt;/gc

Hot! This may be my top tip of all time-- huge time saver! Extremely handy when you are just adding to an existing list and can't possibly keep track of all the items. I use this to generate the "live" table of contents at right as well as the <a name="n"> tags in the body. Check this out:

passthru('let count=0; cat index.php | grep -v "Table of Contents" | grep "<li>" | while read LINE; do let count=count+1; echo $LINE | sed "s/<li><p><strong><a name=\".*\">//" | sed "s/<\/strong><\/p>//" | sed "s/^/<li><a href=\"#$count\">/" | sed "s/$/<\/a><\/li>/"; done; let count=0; cat index.php | grep -v "Table of Contents" | grep "<li>" | while read LINE; do let count=count+1; TIP=`echo $LINE | sed "s/^<li><p><strong>//" | sed "s/<\/strong><\/p>//" | sed "s/<a name=\"[0-9]\+\">//"`; sed -i "s|<a name=\"[0-9]\+\">$TIP|<a name=\"$count\">$TIP|" index.php; done;');

What would be super cool is sorting the items by popularity instead of age. That's going to be a tough one because I can't track "#" clicks without a Javascript to count them. Challenge accepted.

By default, Postfix prefers IPv6. The problem is that if you have a smaller IPv6 prefix than a /64, such as a /112, Google apparently counts your network neighbors' email volume in the same /64 range against your /112 IPv6 address range. I see where they are coming from with this, in that there are about a billion IPv6 addresses in a /112 prefix and it would be impossible to keep track of individual IPv6 addresses. But the problem is that when being efficient with IPv6, you get blocked with zero justification. I switched to IPv4 preferred using my "seasoned" IPv4 address that has been in use for the last 18 months, and I seem to have stopped having problems delivering to Google users. Your mileage may vary. Add this to /etc/main.cf:

smtp_address_preference = ipv4

Add this line to the end of /etc/ssh/sshd_config:

AllowUsers username1 username2 username3

Add this to your /etc/postfix/access file (note: /etc/postfix/main.cf must be set to use PCRE on the access file):

# Block the entirety of Russia except for government domains 6 18 2022
/(?<!mil|ac|gov|edu)\.ru$/	REJECT We have made the difficult decision to reject all private sector .ru domains due to the amount of spam.  Government users or educational senders may call to confirm access, see https://www.andrewwatters.com/#contact

find . -exec grep -B 5 "To: tinachung.re" {} \;

I started using /missing.php to be more user-friendly when URLs are broken. But I wanted to include a sitemap for people to get back where they were going. Here you go:

wget --spider --force-html -r -l2 https://www.andrewwatters.com 2>&1 | grep "http" | cut -d " " -f 4 | grep -v '\(\.png\)\|\(\.jpg\)\|\(\.pdf\)\|\(\.svg\)\|\(\.js\)\|\(.jpeg\)\|\(.txt\)\|\(.css\)\|\(.JPG\)\|\(.mp4\|.MP4\|.MOV\|.mov\)\|\(.xlsx\)\|\(\/site.webmanifest\)' | sed "s/https\:\/\/www.andrewwatters.com/<a href=\"/g" | uniq | grep . | uniq | sort -f -k 1.2 | grep -v "http" | sed "s/\/$/\/\">/g" | awk -F "\"" '{print $1,"\"",$2,"\"",$3,$2,"</a><BR>"}' | sed 's/ //g' | sed 's/ahref/a href/g'

I could have been more efficient here, but it is what it is. This will print something like this, which is exactly what I needed:

I mass renamed my .phtml files to .php during an update. That left about 64 occurrences of a particular function in my sites, as well as hundreds of links to .phtml files. Check this out:

find ./* -name "*.php" -type f -exec sed -i "s/filemtime(\"index\.phtml\")/filemtime(\"index\.php\")/g" {} \;

New and improved:

sudo find /var/log/httpd/ -name "*andrew*" -exec grep "\/6x7\/ " {} \; | cut -d "-" -f 1,2,3 | sort -k 1 | sed "s/\[//" | sed "s/- - /- /g"

This one works on SELinux systems where grep may not be allowed to use wildcards in privileged directories, and also works with IPv6. This searches all of my web server logs and produces similar output to the below example, though I've chosen to sort by IP address instead of date:

69.160.160.60 - 16/Mar/2023:15:04:45 
71.36.217.2 - 17/Mar/2023:14:56:53 
71.36.217.2 - 17/Mar/2023:18:13:21 
73.31.252.175 - 23/Mar/2023:19:32:39 
74.51.155.55 - 27/Mar/2023:12:07:31 
74.51.155.55 - 27/Mar/2023:12:12:18 
75.221.96.253 - 06/Apr/2023:12:42:56 
76.202.243.201 - 02/Apr/2023:16:12:16 
85.188.1.90 - 16/Mar/2023:08:31:53 
85.76.163.193 - 16/Mar/2023:08:15:08 
85.76.163.193 - 16/Mar/2023:08:27:08 
88.99.250.124 - 25/Mar/2023:19:08:15 
88.99.250.124 - 25/Mar/2023:19:31:16 
91.240.118.252 - 18/Mar/2023:08:23:45 
91.240.118.252 - 18/Mar/2023:17:05:46 
94.130.219.240 - 09/Apr/2023:16:21:03 
96.71.249.81 - 21/Mar/2023:11:52:13 
97.113.131.70 - 29/Mar/2023:13:12:08 
99.50.128.161 - 28/Mar/2023:08:54:22 
99.89.37.180 - 15/Mar/2023:10:29:23

Prior version: I need to update this to be compatible with IPv6 now that I also serve web pages over IPv6. But for now, here it is:

grep "GET \/syn\/ " /var/log/httpd/andrew*access* | cut -d " " -f 1,4 | cut -d ":" -f 2,3,4,5 | sed "s/\[//g" | sort -k 1 > ~/test

Sample output:

95.216.96.170 10/Nov/2021:12:49:47
95.216.96.170 15/Aug/2021:00:55:18
95.216.96.170 25/May/2021:03:22:44
95.216.96.170 26/Feb/2021:07:31:56
95.216.96.242 09/Apr/2021:03:20:43
95.216.96.242 12/Oct/2021:14:26:58
95.216.96.242 25/Jun/2021:10:56:46
95.216.96.242 26/Feb/2021:12:42:44
95.216.96.244 09/May/2021:03:40:59
95.216.96.244 09/Nov/2021:09:52:37
95.216.96.244 13/Aug/2021:21:23:57
95.216.96.244 13/Sep/2021:10:53:24
95.216.96.244 24/Apr/2021:03:48:56
95.216.96.244 27/Mar/2021:10:12:56
95.216.96.244 28/Jan/2021:20:07:30
95.216.96.245 25/Jun/2021:07:15:02
95.216.96.254 08/Apr/2021:21:07:26
95.216.96.254 09/Jun/2021:15:18:03
95.216.96.254 12/Mar/2021:17:07:54
95.216.96.254 19/Jan/2021:03:40:15
95.216.96.254 20/Feb/2021:04:45:35
95.216.96.254 30/Jun/2021:01:31:44

[[redacted] ~]$ let count=0; while read line; do let count=(count + 1); echo "Line: $count IP: $line"; geoiplookup $line; done < access.txt | grep -B 1 Russia
Line: 237 IP: 195.201.166.168
GeoIP Country Edition: RU, Russian Federation
--
Line: 269 IP: 212.192.246.107
GeoIP Country Edition: RU, Russian Federation
--
Line: 566 IP: 77.88.5.40
GeoIP Country Edition: RU, Russian Federation
Line: 567 IP: 77.88.5.50
GeoIP Country Edition: RU, Russian Federation
--
Line: 612 IP: 93.158.161.33
GeoIP Country Edition: RU, Russian Federation
Line: 613 IP: 93.158.161.58
GeoIP Country Edition: RU, Russian Federation

This is a lifesaver if someone guesses your password or steals a SSH key. Original post is from Stack Exchange, I made a couple of small changes.

Add the following to your .bash_profile:

IP="$(echo $SSH_CONNECTION | cut -d " " -f 1)"
HOSTNAME=$(hostname)
NOW=$(date +"%e %b %Y, %a %r")
echo 'Someone from '$IP' logged into '$HOSTNAME' as you at '$NOW'.' | mail -s 'SSH Login Notification' user@example.com

<script>function slider() { document.getElementById('rangevalue').value = document.getElementById('slider').value; window.location.href="./?slider=" + document.getElementById('slider').value; }</script> $lines = stripslashes(strip_tags($_GET['slider'])); if ($lines == "") { $lines = 500; } $logfile = exec('find /var/log/httpd/andrew-ssl-access* -exec ls -1t "{}" + | head -n 1'); passthru('tail -n ' . $lines . ' ' . $logfile . ' | grep "andrew\|lindsey\|arielle\|lauren" | cut -d " " -f 4 | sort | head -n 1 | sed "s/\(\]\|\[\)//g"; tail -n ' . $lines . ' ' . $logfile . ' | grep "andrew\|lindsey\|arielle\|lauren" | cut -d " " -f 4 | sort | tail -n 1 | sed "s/\(\]\|\[\)/to /g"; echo "<BR>"; tail -n ' . $lines . ' ' . $logfile . ' | grep "andrew\|lindsey\|arielle\|lauren" | cut -d " " -f 3,1 | sort -k 1 | sort -u | grep -v "-" | perl -pe "s/\n/<BR>/g"'); echo "Analyze last <input type=\"text\" id=\"rangevalue\" size=\"4\" value=\"$lines\"> lines<BR><input id=\"slider\" type=\"range\" min=\"100\" max=\"1000\" step=\"100\" value=\"$lines\" onmouseup=slider()><BR>of $logfile";

Explanation: searches the most recent SSL access log for unique IP's/users during a variable window of time covered by the last X log lines, with the number of lines specified by a HTML5 slider. if you have a web portal secured by .htaccess files, you can easily get the logged-in user names in a PHP global variable: $_SERVER['REMOTE_USER']. This script doesn't require any of that; all you have to do is list your .htaccess users in the grep expression. It is extremely helpful to see the last couple of days of logins using an adjustable slider. It currently looks at the newest log file matching the specified pattern. Next step, I could just cat the .htpasswd file and run each of those names so I don't have to change the hard-code here each time.

Sample output

You can embed webserver statistics inline for any purpose:

As of this moment, the last 90 days of log files show <strong><span style="color:red;"><?php passthru("grep \" \/syn\/ \" /var/log/httpd/andrew*access* | grep -E -o \"([0-9]{1,3}[\.]){3}[0-9]{1,3}\" | wc -l"); ?></span></strong> visits, of which <?php passthru("grep \" \/syn\/ \" /var/log/httpd/andrew*access* | grep -E -o \"([0-9]{1,3}[\.]){3}[0-9]{1,3}\" | sort -u | wc -l"); ?> were from unique IP's.

This is useful if you want to see whether any visitors to your website also attempt to hack your server via SSH. I ran this expecting to find a bunch of matches, but there were literally a couple of matches and they have legitimate explanations.

Method one: grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" /var/log/secure | sort -u | while read -r line; do echo "Processing $line..."; grep $line access* | grep -v "50\.237\."; done

Explanation: searches the most recent SSH log for unique IP's and matches those against web server log files, excluding my own IP address. I ran this and thought that I found one actual hacker, who visited my website and then tried to log in via SSH...but this turned out to be me using SFTP from a client's office...lol.

Method two: grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" iptables-output | grep -v "0\.0\.0\.0" | sort -u | while read -r line; do echo "Processing $line..."; grep $line access*; done

cat andrew-*access* | grep "/linux/ " | cut -d " " -f 1,2,3,4,5,6,7,8 | sort -k 4 -k 2M -k1 -k4 | grep -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -k 7

Method one:

result=$(if ps ax | grep -q -v "74189"; then echo "Process done"; else echo "Process pending"; fi;); until (echo $result | grep -v "pending"); do sleep 2; echo $result; done; say "all done"

Explanation: replace "74189" with the process ID you want to monitor. This polls the system every two seconds and (on macOS) verbally says "all done" when the process is done. I'll probably ring the terminal bell on Linux. Next, I will do a one-line command that doesn't require finding the process ID first, and will provide the option of email notification if the process takes longer than the specified time.

Method two:

result=$(if ps ax | grep -q -v "300851"; then echo "Process done"; else echo "Process pending"; fi;); until (echo $result | grep -q -v "pending"); do sleep 2; echo $result; done; echo "Process done" | mail -s "Process notification" andrew@andrewwatters.com;

Explanation: same as method one, but emails you upon completion. Critically important to this one, the "until" clause must have the -q flag for grep, or else it won't work. I intend to do a one-liner in the near future where you don't have to find the process ID first. Should be straightforward using the cut command, since the process ID is given immediately whenever you use the & operator.

Method three: watch "ps ax | grep tar" and you'll at least have a simple indication that the process you're looking for is still running.

curl -v --silent https://ecf.cand.uscourts.gov/cgi-bin/rss_outside.pl 2>&1 | grep "Watters"

Here, I search the output of my local U.S. District Court's RSS feed to see the last 24 hours or so of filings and whether I am in them. The above command is manual, which is pointless due to the volume of filings and short window in the RSS feed. Instead, I recommend a twice-daily cron job that emails you if matches are found (because you would want to know as soon as possible):

if curl -v --silent https://ecf.cand.uscourts.gov/cgi-bin/rss_outside.pl 2>&1 | grep -q "Apple"; then echo "Apple found in CAND ECF" | mail -s "CAND RSS results" andrew@andrewwatters.com; else echo not found; fi

In this variant, I email myself if the search term is found so that I can go back and check the particular item manually. The -q flag in grep returns success if the term is found, and null if the term is not found-- useful if there are multiple matches. I've put this in a shell script with some other items and have added a cron job, with email notification suppressed so that I only get emails if the search term is found. I think this is the fastest way to find out whether you or someone whose litigation you want to follow have been sued in Federal court, and it will capture most civil and criminal filings (it won't capture sealed indictments, but if that happens then you have bigger problems). Next on the feature list, I will add context so that I get links to the relevant filings, case number, etc. and can click them directly from the email notice.

Adapted from a post at Stack Exchange. This is useful when you are exporting hundreds of EML files from Thunderbird, which puts the date at the end instead of the beginning of the file name. Renaming the files en masse lets you sort by date.

Preview: for file in *.eml; do no_extension=${file%.eml}; the_date=$(echo "{$no_extension}" | rev | cut -d ' ' -f 2 | rev); year=${the_date:0:4}; month=${the_date:5:2}; day=${the_date:8:2}; date_part=${year}-${month}-${day}; filename_part=$(echo "${no_extension}" | cut -d ' ' -f 1-); new_file="${date_part} ${filename_part}.eml"; echo "${file} -> ${new_file}"; done

Actual: for file in *.eml; do no_extension=${file%.eml}; the_date=$(echo "{$no_extension}" | rev | cut -d ' ' -f 2 | rev); year=${the_date:0:4}; month=${the_date:5:2}; day=${the_date:8:2}; date_part=${year}-${month}-${day}; filename_part=$(echo "${no_extension}" | cut -d ' ' -f 1-); new_file="${date_part} ${filename_part}.eml"; mv "${file}" "${new_file}"; done

Method one: cat logs/raellic.com/http/access.log | grep "Vision" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | nslookup | grep "name ="

Returns a list of all the hosts in my server access log that accessed a PDF with Vision in the title.

Method two: cat logs/raellic.com/http/access.log | grep "Vision" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" > blah.txt; while read IP; do echo "$IP"; whois "$IP"; sleep 2; done < blah.txt > bluh.txt &

Executes whois instead of nslookup. Note that this may result in rate limiting or even a ban if there are too many lines in the log file. I recommend including "sleep 2" in the command so there is a built-in rate limit, but you still need to be careful with the total number of queries.

Wrap the long lines of the Base64-encoded data to 78 characters and break with a newline character (\n):

<? wordwrap($attachment,78,"\n",true); ?>

Now your DKIM signature will be valid because the message body is intact when signed.

This is really annoying: libmilter is only available if you download the Sendmail source code and manually build it. Here you go:

wget https://www.andrewwatters.com/linux/sendmail-current.tar.gz

tar -xvzf sendmail-current.tar.gz

cd sendmail-8.15.2

cd libmilter

./Build

make

sudo make install

That's it! Now when you try to install opendkim, you won't get error messages about libmilter being missing.

ffmpeg -framerate 1 -loop 1 -i final-cover.png -i album.ogg -acodec copy -vcodec libvpx-vp9 -crf 20 -b:v 0 -vf subtitles=album.ass -shortest album1.webm

Seriously, this is one of the hardest things I have ever done. Finding the right tool was half the challenge.

localhost$ sudo systemctl enable multi-user.target

localhost$ reboot

[After rebooting]

localhost$ startx /usr/bin/startkde

You would not believe how long it took me to figure that out. If you just do startx, it will freeze your system if there are any problems with GNOME.