mkdocs-benoit.jp.net/docs/Howtos/HowtoSysadmin.md

Sysadmin tips

List of connected machines:

ss -tanpur | awk '{print $6}' | cut -d':' -f1 | sort | uniq

Archlinux merge new config files:

for i in $(find /etc -iname '*.pacnew'); do sudo meld $i ${i%%.pacnew} && rm -i $i; done
for i in $(find /etc -iname '*.pacsave'); do sudo meld $i ${i%%.pacsave} && rm -i $i; done

Get Github or Gitlab user key:

$ curl https://github.com/<username>.keys
$ curl https://gitlab.com/<username>.keys

Enter a namespace, for example LXD (which is in a NS by Snap).

$ nsenter -t $(cat /var/snap/lxd/common/lxd.pid) -m

SSH into a machine without checking host key. Useful when servers are in a rescue mode.

$ ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -o "GlobalKnownHostsFile=/dev/null"

Certbot manual example.

# certbot certonly --non-interactive --webroot --webroot-path /var/www/html/ -d foo.bar -d www.foo.bar

GPG-agent list SSH key and remove.

gpg-connect-agent
KEYINFO --ssh-list --ssh-fpr
DELETE_KEY $HASH

Show md5 fingerprint of SSH key.

$ ssh-keygen -l -E md5 -f .ssh/key.pub

Password recovery. At grub stage, press e to edit the kernel line and add init=/bin/bash. It will drop you in a shell before init system (systemd).

# mount -o remount,rw /
# passwd

Mount partitions on an image file using losetup.

# losetup -P -f --show my.img

List all software installed from particular component (non-free, contrib)

$ dpkg-query -W -f='${Section}\t${Package}\n' | grep ^non-free

Manually rotate a file without logrotate, with savelog(8).

$ savelog

What processes uses swap?

$ for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

MySQL "fast" shutdown.

> set global innodb_max_dirty_pages_pct = 0;
$ mysqladmin ext -i10 | grep dirty

mkfs.ext4 for old systems in rescue mode (Debian Wheezy, …).

# mkfs.ext4 -O ^64bit,^metadata_csum

Send a mail from queue.

$ postcat -q ID > mail
$ < mail sendmail -f FROM TO

Python Simple HTTP Server (useful for Munin for example).

$ cd /var/cache/munin/www
$ python -m SimpleHTTPServer 8080

Show custom certs (not a link) and expiration in /etc/ssl/certs.

# find /etc/ssl/certs/ -type f -print -exec openssl x509 -text -in {} \; | grep --color=auto -e etc -e CN= -e DNS: -e After;

Edit Bind DNS serial (needs modifications, not generic).

$ sed -ri 's/^\s*[0-9]+\s*; serial/\t\t\t 2017041010\t ; serial/' db.*

After Debian/Ubuntu upgrade, merge local config files according to config files shipped in packages.

# for file in $(find /etc -iname '*.dpkg-dist'); do vimdiff ${file%%.dpkg-dist} $file; rm $file; done
# for file in $(find /etc -iname '*.dpkg-old'); do vimdiff ${file%%.dpkg-old} $file; rm $file; done
# for file in $(find /etc -iname '*.dpkg-new'); do vimdiff ${file%%.dpkg-new} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-dist'); do vimdiff ${file%%.ucf-dist} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-old'); do vimdiff ${file%%.ucf-old} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-new'); do vimdiff ${file%%.ucf-new} $file; rm $file; done

Debug php with strace and php-cgi (especially useful for wp multisites).

$ HTTP_HOST=www.site.com SCRIPT_FILENAME=index.php REDIRECT_STATUS=CGI SERVER_NAME=www.site.com strace -s 65535 -o /tmp/strace php-cgi -f index.php

ps with long user fields (here 20).

$ ps axo user:20,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm

WTF is happening in apache (or other)? Let's strace all apache processes.

# strace -p $(ps auwwwx | grep apache | tr -s '\t' ' ' | cut -d' ' -f2 | tr '\n' ' ' | sed 's/ / -p /g') 9999

WTF is happening? Let's tail all logs.

# tail -f $(lsof | grep -F .log | tr -s '\t' ' ' | cut -d' ' -f10 | sort | uniq | tr -s '\n' ' ') 

Search for suspects POST in apache.log (often attacks).

# grep -Eo '"POST .*.php' access.log | grep -ve cron -e login -e admin -e xmlrpc -e trackback -e comment -e 404 | sort -u

Check for crashed MySQL table in syslog and launch a repair.

#!/bin/bash
tables=$(grep crashed /var/log/syslog | grep -Eo \'\./.*\' --color=auto | sed s#\'./## | sed s#\'## | uniq | tr -s '\n' ' ')
for tableC in $tables; do
    db=${tableC%/*}
    table=${tableC#*/}
    mysqlcheck --auto-repair --check $db $table
done

Get the groups of an user and add another user into these groups.

# for group in $(grep user1 /etc/group | cut -d':' -f1 | sed '/user1/d'); do adduser user2 $group; done

Get the last acceded URLs in Squid Access list.

# tail -n100 /var/log/squid3/access.log | grep -oE 'http.*' | cut -d ' ' -f1 | sort | uniq

Migrate MySQL users.

# #SRC Server
# mysql mysql -e "select * from user WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_user';"
# mysql mysql -e "select * from db WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_db';"

# #DST Server
# scp server:/tmp/mysql_{db,user} /tmp
# chmod 664 /tmp/mysql_{db,user}
# mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_user' INTO TABLE user;"
# mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_db' INTO TABLE db;"

Find userid of mails in mailq.

$ for i in $(mailq | grep -Eo [A-F0-9]{10} | tr -s '\n' ' '); do postcat -q $i | grep userid | grep -Eo "[0-9]{4,}" >> tmp/userid; done
$ sort -n /tmp/userid | uniq

Kill every MySQL SELECT older than X seconds – Original: https://anothersysadmin.wordpress.com/2008/10/29/kill-every-mysql-select-older-than-x-seconds/

#!/bin/bash
# From https://anothersysadmin.wordpress.com/2008/10/29/kill-every-mysql-select-older-than-x-seconds/
SEC=$1
IFS='|'
if [[ $SEC -lt 1 ]]; then
    echo "Usage: $0 SECONDS"
    exit 1
fi
mysqladmin proc -v|grep Query|grep -Evi "delete|update|insert|alter table" |while read dummy qid qusr qhost qdb qstat qsec qstat2 query; do
    if [ $qsec -gt $SEC ]; then
        echo "Killing query $qid..."
        mysqladmin kill $qid
    fi
done

List of contacts when sending a mail for technical purpose on a domain which doesn't announce their technical contacts in a whois.

abuse@<domain>, admin@<domain>, administrator@<domain>, contact@<domain>, info@<domain>, postmaster@<domain>, support@<domain>, webmaster@<domain>

itk change rights.

# find /tmp/ -user www-user.old -exec chown www-user:user {} \;
# find /tmp/ -user user.old -exec chown user:user {} \;

* Détecter les fichiers non lisibles par Apache (lecture sur le groupe) : find ./ -type f ! -perm /g=r -exec ls -l {} \;
* Détecter les répertoires non lisibles par Apache (lecture/exécution sur le groupe) : find ./ -type d \( ! -perm /g=r -o ! -perm /g=x \) -exec ls -ld {} \;
* Détecter les fichiers/répertoires accessibles en écriture par Apache (écriture sur le groupe) : find ./ -perm /g=w
* Détecter les fichiers/répertoires accessibles en écriture par tous : find ./ -perm -007 -o -type f -perm -006 

Get useradd command for migrating account.

# for i in user1 user2 user3...; do echo -n 'useradd -m -s /bin/bash -u '$(grep -E "^$i" /etc/passwd | cut -d':' -f3) && echo -en ' -p' \'$(grep -E "^$i" /etc/shadow | cut -d ':' -f2)\' $i '\n'; done

Output : 
useradd -m -s /bin/bash -u USERID -p 'USERPWD' username

Find files newert than (mtime) a precise date, and execute an action.

# find . ! -newermt '2012-09-19 11:40:00' -exec cp {} /tmp/mails \;