mkdocs-benoit.jp.net/docs/Howtos/HowtoSysadmin.md

# Sysadmin tips

Archlinux merge new config files:

```console
for i in $(find /etc -iname '*.pacnew'); do sudo meld $i ${i%%.pacnew} && rm -i $i; done
for i in $(find /etc -iname '*.pacsave'); do sudo meld $i ${i%%.pacsave} && rm -i $i; done
```

Get Github or Gitlab user key:

```console
$ curl https://github.com/<username>.keys
$ curl https://gitlab.com/<username>.keys
```

Enter a namespace, for example LXD (which is in a NS by Snap).

```console
$ nsenter -t $(cat /var/snap/lxd/common/lxd.pid) -m
```

SSH into a machine without checking host key. Useful when servers are in a rescue mode.

```console
$ ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -o "GlobalKnownHostsFile=/dev/null"
```

Certbot manual example.

```console
# certbot certonly --non-interactive --webroot --webroot-path /var/www/html/ -d foo.bar -d www.foo.bar
```

GPG-agent list SSH key and remove.

```
gpg-connect-agent
KEYINFO --ssh-list --ssh-fpr
DELETE_KEY $HASH
```

Show md5 fingerprint of SSH key.

```console
$ ssh-keygen -l -E md5 -f .ssh/key.pub
```

Password recovery. At grub stage, press `e` to edit the kernel line and add `init=/bin/bash`. It will drop you in a shell before init system (systemd).

```console
# mount -o remount,rw /
# passwd
```
Mount partitions on an image file using losetup.

```console
# losetup -P -f --show my.img
```

List all software installed from particular component (non-free, contrib)

```console
$ dpkg-query -W -f='${Section}\t${Package}\n' | grep ^non-free
```

Manually rotate a file without logrotate, with savelog(8).

```console
$ savelog
```

What processes uses swap?

```console
$ for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
```

MySQL "fast" shutdown.

```console
> set global innodb_max_dirty_pages_pct = 0;
$ mysqladmin ext -i10 | grep dirty
```

mkfs.ext4 for old systems in rescue mode (Debian Wheezy, …).

```console
# mkfs.ext4 -O ^64bit,^metadata_csum
```

Send a mail from queue.

```console
$ postcat -q ID > mail
$ < mail sendmail -f FROM TO
```

Python Simple HTTP Server (useful for Munin for example).

```console
$ cd /var/cache/munin/www
$ python -m SimpleHTTPServer 8080
```

Show custom certs (not a link) and expiration in `/etc/ssl/certs`.

```console
# find /etc/ssl/certs/ -type f -print -exec openssl x509 -text -in {} \; | grep --color=auto -e etc -e CN= -e DNS: -e After;
```

Edit Bind DNS serial (needs modifications, not generic).

```console
$ sed -ri 's/^\s*[0-9]+\s*; serial/\t\t\t 2017041010\t ; serial/' db.*
```

After Debian/Ubuntu upgrade, merge local config files according to config files shipped in packages.

```console
# for file in $(find /etc -iname '*.dpkg-dist'); do vimdiff ${file%%.dpkg-dist} $file; rm $file; done
# for file in $(find /etc -iname '*.dpkg-old'); do vimdiff ${file%%.dpkg-old} $file; rm $file; done
# for file in $(find /etc -iname '*.dpkg-new'); do vimdiff ${file%%.dpkg-new} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-dist'); do vimdiff ${file%%.ucf-dist} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-old'); do vimdiff ${file%%.ucf-old} $file; rm $file; done
# for file in $(find /etc -iname '*.ucf-new'); do vimdiff ${file%%.ucf-new} $file; rm $file; done
```

Debug php with strace and php-cgi (especially useful for wp multisites).

```console
$ HTTP_HOST=www.site.com SCRIPT_FILENAME=index.php REDIRECT_STATUS=CGI SERVER_NAME=www.site.com strace -s 65535 -o /tmp/strace php-cgi -f index.php
```

`ps` with long user fields (here 20).

```console
$ ps axo user:20,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm
```

WTF is happening in apache (or other)? Let's strace all apache processes.  
```console
# strace -p $(ps auwwwx | grep apache | tr -s '\t' ' ' | cut -d' ' -f2 | tr '\n' ' ' | sed 's/ / -p /g') 9999
```

WTF is happening? Let's tail all logs.

```console
# tail -f $(lsof | grep -F .log | tr -s '\t' ' ' | cut -d' ' -f10 | sort | uniq | tr -s '\n' ' ') 
```

Search for suspects POST in apache.log (often attacks).  
```console
# grep -Eo '"POST .*.php' access.log | grep -ve cron -e login -e admin -e xmlrpc -e trackback -e comment -e 404 | sort -u
```

Check for crashed MySQL table in syslog and launch a repair.  
```bash
#!/bin/bash
tables=$(grep crashed /var/log/syslog | grep -Eo \'\./.*\' --color=auto | sed s#\'./## | sed s#\'## | uniq | tr -s '\n' ' ')
for tableC in $tables; do
    db=${tableC%/*}
    table=${tableC#*/}
    mysqlcheck --auto-repair --check $db $table
done
```

Get the groups of an user and add another user into these groups.  
```console
# for group in $(grep user1 /etc/group | cut -d':' -f1 | sed '/user1/d'); do adduser user2 $group; done
```

Get the last acceded URLs in Squid Access list.  
```console
# tail -n100 /var/log/squid3/access.log | grep -oE 'http.*' | cut -d ' ' -f1 | sort | uniq
```

Migrate MySQL users.  
```console
# #SRC Server
# mysql mysql -e "select * from user WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_user';"
# mysql mysql -e "select * from db WHERE USER='user1' OR USER='user2' INTO OUTFILE '/tmp/mysql_db';"

# #DST Server
# scp server:/tmp/mysql_{db,user} /tmp
# chmod 664 /tmp/mysql_{db,user}
# mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_user' INTO TABLE user;"
# mysql mysql -e "LOAD DATA INFILE '/tmp/mysql_db' INTO TABLE db;"
```

Find userid of mails in mailq.  
```console
$ for i in $(mailq | grep -Eo [A-F0-9]{10} | tr -s '\n' ' '); do postcat -q $i | grep userid | grep -Eo "[0-9]{4,}" >> tmp/userid; done
$ sort -n /tmp/userid | uniq
```

Kill every MySQL SELECT older than X seconds – Original: https://anothersysadmin.wordpress.com/2008/10/29/kill-every-mysql-select-older-than-x-seconds/  
```bash
#!/bin/bash
# From https://anothersysadmin.wordpress.com/2008/10/29/kill-every-mysql-select-older-than-x-seconds/
SEC=$1
IFS='|'
if [[ $SEC -lt 1 ]]; then
    echo "Usage: $0 SECONDS"
    exit 1
fi
mysqladmin proc -v|grep Query|grep -Evi "delete|update|insert|alter table" |while read dummy qid qusr qhost qdb qstat qsec qstat2 query; do
    if [ $qsec -gt $SEC ]; then
        echo "Killing query $qid..."
        mysqladmin kill $qid
    fi
done
```

List of contacts when sending a mail for technical purpose on a domain which doesn't announce their technical contacts in a whois. 
```
abuse@<domain>, admin@<domain>, administrator@<domain>, contact@<domain>, info@<domain>, postmaster@<domain>, support@<domain>, webmaster@<domain>
```

itk change rights.  
```console

# find /tmp/ -user www-user.old -exec chown www-user:user {} \;
# find /tmp/ -user user.old -exec chown user:user {} \;

* Détecter les fichiers non lisibles par Apache (lecture sur le groupe) : find ./ -type f ! -perm /g=r -exec ls -l {} \;
* Détecter les répertoires non lisibles par Apache (lecture/exécution sur le groupe) : find ./ -type d \( ! -perm /g=r -o ! -perm /g=x \) -exec ls -ld {} \;
* Détecter les fichiers/répertoires accessibles en écriture par Apache (écriture sur le groupe) : find ./ -perm /g=w
* Détecter les fichiers/répertoires accessibles en écriture par tous : find ./ -perm -007 -o -type f -perm -006 
```

Get useradd command for migrating account.  
```console
# for i in user1 user2 user3...; do echo -n 'useradd -m -s /bin/bash -u '$(grep -E "^$i" /etc/passwd | cut -d':' -f3) && echo -en ' -p' \'$(grep -E "^$i" /etc/shadow | cut -d ':' -f2)\' $i '\n'; done

Output : 
useradd -m -s /bin/bash -u USERID -p 'USERPWD' username
```

Find files newert than (mtime) a precise date, and execute an action.  
```bash
# find . ! -newermt '2012-09-19 11:40:00' -exec cp {} /tmp/mails \;
```