How to disable the www-data user to send emails with postfix

Sometimes an insecure configuration allows spammer to use the www-data user to send emails with you postfix server.
Normally this is the case, when you get a bunch of error email from your mailserver, that some emails from www-data@hostname.tld could not be delivered.

To be sure, that this situation cannot exist, you can add the www-data user to a block list:
In /etc/postfix/main.cf just add

authorized_submit_users = !www-data, static:all

and restart you postfix server.
You will now get a lot of error in you log like this one:

postfix/sendmail[8563]: fatal: User www-data(33) is not allowed to submit mail

After some time, the spammers will hopefully loose interest in you server :-).

2017/12/122017/12/12

20min Handson ZFS

ZFS is often called the last word in file systems.
It is a new approach to deal with large pools of disks originally invented by Sun.
It was later then ported to FreeBSD, MacOS (only 10.5) and Linux.

This text should show some of the basic feature of ZFS and demonstrate them handson by example.

Prerequisites

-> FreeBSD
-> Solaris
-> MacOS (only Userland)

In our example we use

SunOS openindiana 5.11 oi_151a5 i86pc i386 i86pc Solaris.

as an environment.

But most commands also work on the other systems.

Since we do all the work within a VM, our commands have the pattern:

Input VM:

command

Output VM:

result

Pool Creation

The first information we need is the number of disk, present in our environment.
There are several ways to get a basic disk listing. Under (Open-)Solaris this can be done with:

Input VM:

format < /dev/null

Output VM:

AVAILABLE DISK SELECTIONS:
0. c4t0d0 
/pci@0,0/pci8086,2829@d/disk@0,0
1. c5t0d0 
/pci@0,0/pci1000,8000@16/sd@0,0
2. c5t1d0 
/pci@0,0/pci1000,8000@16/sd@1,0
3. c5t2d0 
/pci@0,0/pci1000,8000@16/sd@2,0
4. c5t3d0 
/pci@0,0/pci1000,8000@16/sd@3,0
5. c5t4d0 
/pci@0,0/pci1000,8000@16/sd@4,0
6. c5t5d0 
/pci@0,0/pci1000,8000@16/sd@5,0
7. c5t6d0 
/pci@0,0/pci1000,8000@16/sd@6,0
8. c5t7d0 
/pci@0,0/pci1000,8000@16/sd@7,0

about Pools

With ZFS it is possible to create different kinds of pools on a specific number of disk.
You can also create several pools within one system.

The following Pools are possible and most commonly used:

Type	Info	Performance	Capacity	Redundancy	Command
JBOD	Just a bunch of disks. In theory it is possible to create on pool for each disk in the system, although this is not quite commonly used.	of each disk	of each disk	–	zpool create disk1 pool1 zpool create disk1 pool2 …
Stripe	This is equivalent to RAID0, the data is distributed over all disks in the pool. If one disks fails, all the data is lost. But you can also stripe several Pools (e.g. two raidz pools) to have better redundancy.	very high	N Disks	no	zpool create disk1 disk2 pool1
Mirror	This is equivalent to RAID1, the data is written to both disks in the Pool. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk.	normal	N-1 Disks	+1	zpool create mirror disk1 disk2 pool1
Raidz	This is equivalent to RAID5. One disk contains the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disks.	high	N-1 Disks	+1	zpool create raidz disk1 disk2 disk3 pool1
Raidz2	This is equivalent to RAID6. Two disks containing the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk with parity data.	high	N-2 Disks	+2	zpool create raidz2 disk1 disk2 disk3 disk4 pool1
Raidz3	There is no real equivalent existing for that one. You have basically three disks with parity data.	high	N-3 Disks	+3	zpool create raidz3 disk1 disk2 disk3 disk4 disk5 pool1

You can also add hot-spares for a better fallback behaviour, SSDs for caching reads (cache) and writes (logs).
I also created a benchmark with various combinations.

create a basic Pool (raidz)

Input VM:

zpool create tank raidz c5t0d0 c5t1d0 c5t2d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0

errors: No known data errors

(Raid5)

You already can access the newly created pool:

Input VM:

ls -al /tank

Output VM:

 
...
total 4
drwxr-xr-x  2 root root  2 2012-10-23 22:02 .
drwxr-xr-x 25 root root 28 2012-10-23 22:02 ..

create a basic Pool (raidz) with one spare drive

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
        spares
          c5t3d0    AVAIL   

errors: No known data errors

List the availibe Layout

Input VM:

zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   185K  1,46G         -     0%  1.00x  ONLINE  -

*The 1,5G does not reflect the real availible space. If you copy a 1G File to the Pool it will use 1,5G (1G + 512M Parity).

create a stripped pool

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 raidz1 c5t4d0 c5t5d0 c5t6d0

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            c5t4d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0

errors: No known data errors

(Raid50 = Raid5 + Raid5)

deal with disk failures

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0

Failure Handling

Input Host:

echo /dev/random >> 1.vdi

Wait for it…
or Input VM:

  pool: tank
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 66K in 0h0m with 0 errors on Tue Oct 23 22:14:19 2012
config:

        NAME          STATE     READ WRITE CKSUM
        tank          DEGRADED     0     0     0
          raidz1-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c5t0d0  DEGRADED     0     0    64  too many errors
              c5t3d0  ONLINE       0     0     0
            c5t1d0    ONLINE       0     0     0
            c5t2d0    ONLINE       0     0     0
        spares
          c5t3d0      INUSE     currently in use

errors: No known data errors

Input VM:

zpool clear tank
...
zpool detach tank c5t0d0
zpool replace tank c5t0d0 c5t7d0

Create File systems

Input VM:

zfs create tank/home
zfs create tank/home/user1
...
chown -R user:staff /tank/home/user1
...
zfs get all tank/home/user1
...
zfs set sharesmb=on tank/home/user1
...
zfs set quota=500M tank/home/user1

Copy File from MacOS into SMB Share.

Snapshot

Input VM:

zfs snapshot tank/home/user1@basic
...
zfs list
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home/user1@basic                0      -  42,6K  -

Input VM:

zfs snapshot -r tank/home@backup
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home@backup                     0      -  41,3K  -
tank/home/user1@basic                0      -  42,6K  -
tank/home/user1@backup               0      -  42,6K  -

Input VM:

zfs clone tank/home/user1@basic tank/home/user2

Output VM:

tank/home/user2          1,33K   894M  70,3M  /tank/home/user2

Restoring Snapshots

Delete ZIP File in SMB-Share.

Input VM:

ls -al tank/home/user1
...
zfs rollback tank/home/user1@backup

Output VM:

ls -al tank/home/user1

Resizing a Pool

Input VM:

zpool list
...
zpool replace tank c5t0d0 c5t4d0
zpool replace tank c5t1d0 c5t5d0
zpool replace tank c5t2d0 c5t6d0
...
zpool scrub tank
...
zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   381K  1,46G     1,50G     0%  1.00x  ONLINE  -

Input VM:

zpool set autoexpand=on tank

Using ZFS for Backups

Bash-Script

rsync -avrz --progress --delete /Users/user root@nas.local::user-backup/
backupdate=$(date "+%Y-%m-%d")
ssh root@nas.local zfs snapshot tank/backup@$backupdate

2017/05/042017/05/04

solving Security Error while starting Java WebStart (e.g. IPMI Remote)

Most of the IPMI Systems out there still using good old Java based Remote Applications to connect to the remote console.
Sine Java 8 update 111, the MD5 singing algorithm was marked as insecure (aka disabled) by Oracale (see Relase Notes for that Release ” Restrict JARs signed with weak algorithms and keys”).
You will get an “Security Error while using MD5withRSA Signature”:

The only solution to fix this error is to have your Hardware Vendor to update the IPMI Firmware with JARs, signed with a more up to date singing algorithm. A work around is to re-enable MD5 for the time being. For that you need to get into your Browser Java Installation.
On my Mac this is in

/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home

You need to edit lib/security/java.security and remove MD5 from jdk.jar.disabledAlgorithms.

Change

jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

jdk.jar.disabledAlgorithms=MD2, RSA keySize < 1024

2016/02/142017/02/09

Disk failure and suprises

Once in a while – and especially if you have a System with an uptime > 300d – HW tends to fail.

Good thing, if you have a Cluster, where you can do the maintance on one Node, while the import stuff is still running on the other ones. Also good to always have a Backup of the whole content, if a disk fails.

One word before I continue: Regarding Software-RAIDs: I had a big problem once with a HW RAID Controller going bonkers and spent a week to find another matching controller to get the data back. At least for redundant Servers it is okay for me to go with SW RAID (e.g. mdraid). And if you can, you should go with ZFS in any case :-).

Anyhow, if you see graphs like this one:

You know that something goes terribly wrong.

Doing a quick check, states the obvious:

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda4[0](F) sdb4[1]
      1822442815 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sda3[0](F) sdb3[1]
      1073740664 blocks super 1.2 [2/1] [_U]

md1 : active raid1 sda2[0](F) sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active raid1 sda1[0](F) sdb1[1]
      33553336 blocks super 1.2 [2/1] [_U]

unused devices: <none>

So /dev/sda seems to be gone from the RAID. Let’s do the checks.

Hardware check

hdparm:

# hdparm -I /dev/sda

/dev/sda:
HDIO_DRIVE_CMD(identify) failed: Input/output error

smartctl:

# smartctl -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.32-34-pve] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /0:0:0:0
Product:
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

/dev/sda is dead, Jim

Next thing is to schedule a Disk Replacement and moving all services to another host/prepare the host to shutdown for maintenance.

Preparation for disk replacement

Stop all running Containers:

# for VE in $(vzlist -Ha -o veid); do vzctl stop $VE; done

I also disabled the “start at boot” option to have a quick startup of the Proxmox Node.

Next: Remove the faulty disk from the md-RAID:

# mdadm /dev/md0 -r /dev/sda1
# mdadm /dev/md1 -r /dev/sda2
# mdadm /dev/md2 -r /dev/sda3
# mdadm /dev/md3 -r /dev/sda4

Shutting down the System.

… some guy in the DC moved to the server at the expected time and replaces the faulty disk …

After that, the system is online again.

copy partition table from /dev/sdb to /dev/sda
```
# sgdisk -R /dev/sda /dev/sdb
```
recreatea another GUID for /dev/sda
```
# sgdisk -G /dev/sda
```

Then add /dev/sda to the RAID again.

# mdadm /dev/md0 -a /dev/sda1
# mdadm /dev/md1 -a /dev/sda2
# mdadm /dev/md2 -a /dev/sda3
# mdadm /dev/md3 -a /dev/sda4



# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda4[2] sdb4[1]
      1822442815 blocks super 1.2 [2/1] [_U]
      [===>.................]  recovery = 16.5% (301676352/1822442815) finish=329.3min speed=76955K/sec

md2 : active raid1 sda3[2] sdb3[1]
      1073740664 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[2] sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]
        resync=DELAYED

md0 : active raid1 sda1[2] sdb1[1]
      33553336 blocks super 1.2 [2/2] [UU]

unused devices: <none>

After nearly 12h, the resync was completed:

and then this happened:

# vzctl start 300
# vzctl enter 300
enter into CT 300 failed
Unable to open pty: No such file or directory

There are plenty of comments if you search for Unable to open pty: No such file or directory

But

# svzctl exec 300 /sbin/MAKEDEV tty
# vzctl exec 300 /sbin/MAKEDEV pty
# vzctl exec 300 mknod --mode=666 /dev/ptmx c 5 2

did not help:

# vzctl enter 300
enter into CT 300 failed
Unable to open pty: No such file or directory

And

# strace -ff vzctl enter 300

produces a lot of garbage – meaning stacktraces that did not help to solve the problem.
Then we were finally able to enter the container:

# vzctl exec 300 mount -t devpts none /dev/pts

But having a look into the process list was quite devastating:

# vzctl exec 300 ps -A
  PID TTY          TIME CMD
    1 ?        00:00:00 init
    2 ?        00:00:00 kthreadd/300
    3 ?        00:00:00 khelper/300
  632 ?        00:00:00 ps

That is not really what you expect when you have a look into the process list of a Mail-/Web-Server, isn’t it?
After looking araound into the system and searching through some configuration files, it became obvious, that there was a system update in the past, but someone forgot to install upstart. So that was easy, right?

# vzctl exec 300 apt-get install upstart
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  sysvinit
The following NEW packages will be installed:
  upstart
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 486 kB of archives.
After this operation, 851 kB of additional disk space will be used.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?] Yes, do as I say!

BUT:

Err http://ftp.debian.org/debian/ wheezy/main upstart amd64 1.6.1-1
  Could not resolve 'ftp.debian.org'
Failed to fetch http://ftp.debian.org/debian/pool/main/u/upstart/upstart_1.6.1-1_amd64.deb  Could not resolve     'ftp.debian.org'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

No Network – doooh.

So… that was that. Another plan to: chroot. Let’s start:
First we need to shutdown the container – or at least what is running of it:

# vzctl stop 300
Stopping container ...
Container is unmounted

Second, we have to mount a bunch of devices to the FS:

# mount -o bind /dev /var/lib/vz/private/300/dev
# mount -o bind /dev/shm /var/lib/vz/private/300/dev/shm
# mount -o bind /proc /var/lib/vz/private/300/proc
# mount -o bind /sys /var/lib/vz/private/300/sys

Then perform the chroot and the installtion itself:

# chroot /var/lib/vz/private/300 /bin/bash -i

# apt-get install upstart
# exit

At last, umount all the things:

# umount -l /var/lib/vz/private/300/sys
# umount -l /var/lib/vz/private/300/proc
# umount -l /var/lib/vz/private/300/dev/shm
# umount -l /var/lib/vz/private/300/dev

If you have trouble, because some of the devices are busy, kill the processes you find out with:

# lsof /var/lib/vz/private/300/dev

Or just clean the whole thing

# lsof 2> /dev/null | egrep '/var/lib/vz/private/300'

Try to umount again :-).

Now restarting the container again.

# Starting container ...
# Container is mounted
# Adding IP address(es): 10.10.10.130
# Setting CPU units: 1000
# Setting CPUs: 2
# Container start in progress...

And finally:

 # vzctl enter 300
 root@300 #  
 root@300 # ps -A | wc -l
 142

And this looks a lot better \o/.

2015/09/292017/02/09

Install CoreOS on Proxmox

Some words before we start…

Hello Blog, it’s been a while. I still have to deliver the last part of the Munin Plugin Development Series (Part 1, 2, 3).

Today I would like to write something about the Setup of a CoreOS Environment on Proxmox. Proxmox is a Debian based Distribution that bundles a Web UI for OpenVZ+KVM and some great Tools for Clustering and Multi-Tenancy Installations. I am using Proxmox as a Hosting Platform for some years now and I am still amazed about the stability and the way things work out so far. I plan to create another Series about things around Proxmox (e.g. Cluster Setup using Tinc/Live Migration of VMs and the overall Network Setup).

But now, let’s dive into the Topic…

VM Setup

My Proxmox Hosts uses private Networks, both for OpenVZ Containers as well as for KVM VMs.
Both private Networks have Internet Access via the Standard Linux IP Forwarding Functions.
Configuration is done via iptables, e.g. for our private KVM Network 10.10.0.0:

iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o eth0 -j SNAT --to ${EXT_IP}

Now, create a (KVM) VM in Proxmox. I picked 2 Cores and 2Gigs of RAM. Choose VirtIO for the Disk as well as the Network. This will provide much better Performance and works out of the Box, since CoreOS has build-in support for VirtIO.

The basic steps for the Setup are:

Now start you VM and open the Console:

Preparations

Downlaod the CoreOS ISO

[user@proxmox]# pwd
/var/lib/vz/template/iso
[user@proxmox]# wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_iso_image.iso

Note your public SSH Key

[user@proxmox]# cat ~/.ssh/id_rsa.pub

becoming root

coreos ~ # sudo su - root

update the root password

coreos ~ # passwd

Setup the basic Network.

coreos ~ # ifconfig eth0 10.10.0.111 netmask 255.255.255.0 up

SSH into your system

[root@cleopatra iso]# ssh root@10.10.0.111
The authenticity of host '10.10.0.111 (10.10.0.111)' can't be established.
RSA key fingerprint is XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX.
Are you sure you want to continue connecting (yes/no)? yes    
root@10.10.0.111's password:
CoreOS stable (766.3.0)
Update Strategy: No Reboots

Finish Network Configuration

coreos ~ # route add default gw 10.10.0.1
coreos ~ # echo "nameserver 8.8.8.8" > /etc/resolv.conf

Installation

see https://coreos.com/os/docs/latest/installing-to-disk.html

Download Config Template

coreos ~ # wget https://gist.githubusercontent.com/phaus/e52241b66576d4484f6f/raw/9032faaa69bc05ebc8b08efb518f2a90bfef4dca/coreos1-config-coreos.yml

Adjust the Configuration as required

coreos ~ # cat coreos1-config-coreos.yml
#cloud-config
hostname: "coreos1"

# include one or more SSH public keys
ssh_authorized_keys:
  - ssh-rsa XXX

coreos:

  units:
    - name: systemd-networkd
      command: stop
    - name: 00-static.network
      runtime: true
      content:  |
        [Match]
        Name=eth*
        [Network]
        Gateway=10.10.0.1
        Address=10.10.0.111/24    
        DNS=8.8.8.8    
    - name: systemd-networkd
      command: start
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start

Replace XXX with your public SSH Key.

Install CoreOS to /dev/vda (it is vda since VirtIO Device are mapped to vdX)

coreos ~ # coreos-install -d /dev/vda -C stable -c ~/coreos1-config-coreos.yml
Checking availability of "local-file"
Fetching user-data from datasource of type "local-file"
Downloading the signature for http://stable.release.core-os.net/amd64-usr/766.3.0/coreos_production_image.bin.bz2...
2015-09-28 20:59:39 URL:http://stable.release.core-os.net/amd64-usr/766.3.0/coreos_production_image.bin.bz2.sig [543/543] -> "/tmp/coreos-install.2oAX9KwZlj/coreos_production_image.bin.bz2.sig" [1]
Downloading, writing and verifying coreos_production_image.bin.bz2...
2015-09-28 21:00:09 URL:http://stable.release.core-os.net/amd64-usr/766.3.0/coreos_production_image.bin.bz2 [195132425/195132425] -> "-" [1]
gpg: Signature made Wed Sep  2 04:32:09 2015 UTC using RSA key ID E5676EFC
gpg: key 93D2DCB4 marked as ultimately trusted
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "CoreOS Buildbot (Offical Builds) <buildbot@coreos.com>" [ultimate]
gpg: Note: This key has expired!
Primary key fingerprint: 0412 7D0B FABE C887 1FFB  2CCE 50E0 8855 93D2 DCB4
     Subkey fingerprint: EEFA 7555 E481 D026 CC40  D8E6 A5A9 6635 E567 6EFC
Installing cloud-config...
Success! CoreOS stable 766.3.0 is installed on /dev/vda

Check your Installation

coreos ~ # mount /dev/vda9 /mnt
coreos ~ # cd /mnt/

Please keep in mind, that most of the Configuration will take place during the first boot of your new Instance.

Time for a Shutdown

coreos ~ # shutdown -h now
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
Connection to 10.10.0.111 closed by remote host.
Connection to 10.10.0.111 closed.

First Boot

Start the VM again (this time it should boot from the internal disk – you can also remove the ISO File, just to be sure). Also the Node should apply the correct Network Configuration.

You should see something like this:

SSH into your new node

[root@cleopatra iso]# ssh core@10.10.0.105

You might get this Warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /root/.ssh/known_hosts:13
RSA host key for 10.10.0.111 has changed and you have requested strict checking.
Host key verification failed

That is fine, since the CoreOS Host just changed it’s SSH Host Key. Just remove the problematic line (in this case line 13) from you /root/.ssh/known_hosts.

After that you should be fine:

[user@proxmox]# ssh core@10.10.0.111
Last login: Tue Sep 29 08:50:48 2015 from 10.10.0.1
CoreOS stable (766.3.0)
Failed Units: 1
  user-cloudinit@var-lib-coreos\x2dinstall-user_data.service
core@coreos1 ~ $ sudo -s
coreos1 core #

Now we need to fix the Configuration. Before that, we should create two more CoreOS Hosts to have a Cluster ready.

2014/12/052018/03/06

Writing Munin Plugins pt3: some Stats about VMWare Fusion

In a project where we had the need for VMs being capable of doing CI for Java and also doing CI for iOS Application (using XCode Build Bots), we decided to go with a Mac OS Server as the Host Platform and using VMWare Fusion as the base Virtualisation System. We had several VMs there (Windows, Solaris, Linux and Mac OS). Doing a proper Monitoring for theses VMs was not that easy. We already had a working Munin Infrastructure, but no Plugin for displaying VMWare Fusion Stats existed.

The first approach was to use the included VMTools for gathering the information, since we already used them to start/stop/restart VMs via CLI/SSH:

#!/bin/bash

echo "starting VMS..."
VM_PATH=/Users/Shared/VMs
TOOL_PATH=/Applications/VMTools
$TOOL_PATH/vmrun -T fusion start $VM_PATH/Mac_OS_X_10.9.vmwarevm/Mac_OS_X_10.9.vmx nogui

#!/bin/bash

echo "starting VMS..."
VM_PATH=/Users/Shared/VMs
TOOL_PATH=/Applications/VMTools
$TOOL_PATH/vmrun -T fusion stop $VM_PATH/Mac_OS_X_10.9.vmwarevm/Mac_OS_X_10.9.vmx

But it was very hard to receive the interesting Data from the Log Files (statistica data is only really supported in VMWare ESXi). So we choose the direct way, to receive the live data, using ps. So this approach is also applicable for other Applications as well.

Our goal was to get at lease three Graphs (% of used CPU, % of used Memory and physically used Memory) sorted by VM Name.

ps -A | grep vmware-vmx

provides us with a list of all running vmware processes. Since we only need specific Data, we add some more filters:

ps -A -c -o pcpu,pmem,rss=,args,comm -r | grep vmware-vmx

29,4 14,0 2341436   J2EE.vmx                                                 vmware-vmx
1,7 12,9 2164200    macos109.vmx                                             vmware-vmx
1,4 17,0 2844044    windows.vmx                                              vmware-vmx
0,7  6,0 1002784    Jenkins.vmx                                              vmware-vmx
0,0  0,0    624     grep vmware-vmx

where this is the description (man ps) of the used columns:

%cpu percentage CPU usage (alias pcpu)
%mem percentage memory usage (alias pmem)
rss the real memory (resident set) size of the process (in 1024 byte units).

You might see several things: First we have our data and the Name of each VM. Second, we have to get rid of the last line, since that is our grep process. Third, we might need to do some String Operations/Number Calculation to get some valid Data at the end.

Since Perl is a good choice if you need to do some String Operations, the Plugins is written in Perl :-).

Let’s have a look.
The Config Element is quite compact (e.g. for the physical mem):

my $cmd = "ps -A -c -o pcpu,pmem,rss=,args,comm -r | grep vmware-vmx";
my $output = `$cmd`;
my @lines=split(/\n/,$output);
...
if( $type eq "mem" ) {
    print $base_config;
    print "graph_args --base 1024 -r --lower-limit 0\n";    
    print "graph_title absolute Memory usage per VM\n";
    print "graph_vlabel Memory usage\n";
    print "graph_info The Graph shows the absolute Memory usage per VM\n";  
    foreach my $line(@lines) {
        if( $line  =~ /(?<!grep)$/ ) {  
            my @vm = ();
            my $count = 0;
            my @array=split(/ /,$line); 
            foreach my $entry(@array) {
                if( length($entry) > 2 ){
                    $vm[$count]=$entry;
                    $count++;
                }
            }
            $vm[3] = clean_vmname($vm[3]);  
            if( $vm[3] =~ /(?<!comm)$/) {           
                if( $lcount > 0 ){
                    print "$vm[3]_mem.draw STACK\n";
                } else {
                    print "$vm[3]_mem.draw AREA\n";
                }
                print "$vm[3]_mem.label $vm[3]\n";
                print "$vm[3]_mem.type GAUGE\n";            
                $lcount++;      
            }           
        }
    }                       
}

After the basic Setup (Category, Graph Type, Labels, etc. ) we go through each line of the output from the ps command, filtering the line containing grep.
We use the stacked Graph Method, so the first entry has to be the base Graph, the following ones will just be layer on top of the first. To get clean VM Names, we have a quite simple function clean_vmname:

sub clean_vmname {
    my $vm_name = $_[0];
    $vm_name =~ s/\.vmx//;
    $vm_name =~ s/\./\_/g;
    return $vm_name;
}

The Code, that delivers the Data looks similar. We just piping the values from the ps command to the output:

foreach my $line(@lines) {
    if( $line  =~ /(?<!grep)$/ ) {
        my @vm = ();
        my $count = 0;
        my @array=split(/ /,$line); 
        foreach my $entry(@array) {
            if( length($entry) > 2 ){
                $vm[$count]=$entry;
                $count++;
            }
        }
        $vm[3] = clean_vmname($vm[3]);
        if( $vm[3] =~ /(?<!comm)$/) {   
            if( $type eq "pcpu" ) {
                print "$vm[3]_pcpu.value $vm[0]\n";
            }
            if( $type eq "pmem" ) {
                print "$vm[3]_pmem.value $vm[1]\n";
            }
            if( $type eq "mem" ) {
                my $value =  ($vm[2]*1024);
                print "$vm[3]_mem.value $value\n";
            }
        }
    }
}

You can find the whole plugin here on GitHub.

Here are some example Graphs you will get as a result:

2014/08/182018/03/06

Adding Background Image to SVG Circles

If you want to create nifty Graphics and Animation in the web, you cannot avoid d3.js. D3.js uses SVG as the basic Displaying Technologie. And sometimes you know, why SVG had such a hard time persuading developers.

It is a simple task:
Creating a Circle with an image as a background. The everyday Web-Developer would just create a Circle Element and would try to add the Background Image via CSS.
But that did not work at all.
After asking friend Google for a while, i found the answer to this question over at stackoverflow.
So you need SVG patterns (never heard about them before). And reference the image-pattern via the ID.

So here is a brief example:


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" width="400" height="400">
    <defs>
        <pattern id="image" x="-32" y="-32" patternUnits="userSpaceOnUse" height="64" width="64">
            <image x="0" y="0" height="64" width="64" xlink:href="http://0.gravatar.com/avatar/902a4faaa4de6f6aebd6fd7a9fbab46a?s=64"/>
        </pattern>
    </defs>
    <line class="link" style="stroke: #9ecae1; stroke-width: 1;" x1="200" y1="0" x2="200" y2="400"/>
    <text dy=".35em" transform="translate(0 , 190)">0,200</text>
    <line class="link" style="stroke: #9ecae1; stroke-width: 1;" y1="200" x1="0" y2="200" x2="400"/>
    <text dy=".35em" transform="translate(200 , 10)">200,0</text>
    <circle id="top" transform="translate(200,200)" r="32" fill="url(#image)"/>
</svg>

We used that in our small project GitHubble (you may find the sources here).

2014/08/032014/08/03

Disable GitHub Image Cache for CI Build Badges

Since some time, GitHub caches Images, that are linked in Wiki-Pages or Readme files.
That’s not optimal, when you want to display current states (e.g. the Build Status of your CI Job).

To disable the Caching for a specific Image, you need to configure a proper Caching Header.
So before the Changes, you have:

$ curl -L -I https://consolving.de/jenkins/buildStatus/icon?job=de.consolving.chatlogconverter
    HTTP/1.1 200 OK
    Date: Sun, 03 Aug 2014 10:08:44 GMT
    Server: Jetty(8.y.z-SNAPSHOT)
    ETag: /static/e5920a00/success.png
    Expires: Sun, 01 Jan 1984 00:00:00 GMT
    Content-Type: image/png
    Content-Length: 2339
    Via: 1.1 consolving.de

That means GitHub will cache the Image (for some time), although the Expires Header is set to the past.
In our case, the Jenkins CI is behind an Apache2 Server. So you need to update your Configuration.

  ...
  # Disable Cache for /jenkins/buildStatus
  <LocationMatch "/jenkins/buildStatus/icon*">
    Header set Cache-Control "no-cache"
    Header set Pragma "no-cache"
  </LocationMatch>
  ...

After that, you have now some more Caching Headers set and GitHub won’t Cache your image anymore.

$ curl -L -I https://consolving.de/jenkins/buildStatus/icon?job=de.consolving.chatlogconverter
HTTP/1.1 200 OK
Date: Sun, 03 Aug 2014 10:13:16 GMT
Server: Jetty(8.y.z-SNAPSHOT)
ETag: /static/e5920a00/success.png
Expires: Sun, 01 Jan 1984 00:00:00 GMT
Content-Type: image/png
Content-Length: 2339
Via: 1.1 consolving.de
Cache-Control: no-cache
Pragma: no-cache

2014/06/222018/03/06

Writing Munin Plugins pt2: counting VPNd Connections

Preamble

Every Munin Plugin should have a preamble by default:

#!/usr/bin/env perl
# -*- perl -*-

=head1 NAME

dar_vpnd a Plugin for displaying VPN Stats for the Darwin (MacOS) vpnd Service.

=head1 INTERPRETATION

The Plugin displays the number of active VPN connections.

=head1 CONFIGURATION

No Configuration necessary!

=head1 AUTHOR

Philipp Haussleiter <philipp@haussleiter.de> (email)

=head1 LICENSE

GPLv2

=cut

# MAIN
use warnings;
use strict;

As you can see, this Plugin will use Perl as the Plugin language.

After that you have some information about the Plugin Usage:

Name of the Plugin + some description
Interpretation of the delivered Data
Information about the Plugins Configuration (not necessary here, we will see that in the other Plugins)
Author Name + Contact Email
License

# MAIN marks the beginngin of the (main) code.

Next you see some Perl Setup, using strict Statements and also show warnings.

Gathering Data

First you should always have a basic idea how you want collect your Data (e.g. which user will use what command to get what kind of data).

For Example we can get all VPN Connections in Mac OS (Server) searching the process List for pppd processes.

ps -ef | grep ppp

    0   144     1   0  5Mär14 ??        10:35.34 vpnd -x -i com.apple.ppp.l2tp
    0 29881   144   0  4:12pm ??         0:00.04 pppd serverid com.apple.ppp.l2tp nodetach proxyarp plugin L2TP.ppp ms-dns 10.XXX.YYY.1 debug logfile /var/log/ppp/vpnd.log idle 7200 noidlesend lcp-echo-interval 60 lcp-echo-failure 5 mru 1500 mtu 1280 receive-all ip-src-address-filter 1 novj noccp intercept-dhcp require-mschap-v2 plugin DSAuth.ppp plugin2 DSACL.ppp l2tpmode answer :10.XXX.YYY.233
    0 22567   144   0  4:12pm ??         0:00.04 pppd serverid com.apple.ppp.l2tp nodetach proxyarp plugin L2TP.ppp ms-dns 10.XXX.YYY.1 debug logfile /var/log/ppp/vpnd.log idle 7200 noidlesend lcp-echo-interval 60 lcp-echo-failure 5 mru 1500 mtu 1080 receive-all ip-src-address-filter 1 novj noccp intercept-dhcp require-mschap-v2 plugin DSAuth.ppp plugin2 DSACL.ppp l2tpmode answer :10.XXX.YYY.234

Collecting only the IP you need some more RegExp using awk:

ps -ef | awk '/[p]ppd/ {print substr($NF,2);}'

10.XXX.YYY.233
10.XXX.YYY.234

We are only interested in the total Connection Count. So we use wc for counting all IPs:

ps -ef | awk '/[p]ppd/ {print substr($NF,2);}' | wc -l

So we now have a basic command that gives us the Count of currentyl connected users.

Configuration

The next thing is how your Data should be handled by the Munin System.
Your Plugin needs to provide Information about the Field Setup.

The most basic (Perl) Code looks like this:

if ( exists $ARGV[0] and $ARGV[0] eq "config" ) {
    # Config Output
    print "...";    
} else {
    # Data Output
    print "...";
}

For a more Information about fieldnames, please follow the above Link.

Our Plugin Source looks like this:

# MAIN
use warnings;
use strict;


my $cmd = "ps -ef | awk '/[p]ppd/ {print substr(\$NF,2);}' | wc -l";

if ( exists $ARGV[0] and $ARGV[0] eq "config" ) {
    print "graph_category VPN\n";
    print "graph_args --base 1024 -r --lower-limit 0\n";    
    print "graph_title Number of VPN Connections\n";
    print "graph_vlabel VPN Connections\n";
    print "graph_info The Graph shows the Number of VPN Connections\n"; 
    print "connections.label Number of VPN Connections\n";
    print "connections.type GAUGE\n";   
} else {
    my $output = `$cmd`;
    print "connections.value $output";
}

Implementation

To test the Plugin you can use munin-run:

> /opt/local/sbin/munin-run dar_vpnd config

graph_category VPN
graph_args --base 1024 -r --lower-limit 0
graph_title Number of VPN Connections
graph_vlabel VPN Connections
graph_info The Graph shows the Number of VPN Connections
connections.label Number of VPN Connections
connections.type GAUGE

> /opt/local/sbin/munin-run dar_vpnd

connections.value        1

Example Graphs

Some basic (long time) Graphs look like this:

2014/06/222018/03/06

Writing Munin Plugins pt1: Overview

Writing your own Munin Plugins

Around February this year, we at innoQ had the need for setting up a Mac OS based CI for a Project. Besides building of integrating some standard Java Software, we also had to setup an Test Environment with Solaris/Weblogic, Mac OS for doing a CI for an iOS Application and a Linux System that contains the Jenkins CI itself.
Additionally the whole Setup should be reachable via VPN (also the iOS Application itself should be able to reach the Ressources via VPN).

To have the least possible obsticles in Setting up the iOS CI and the iOS (iPad) VPN Access, we decide to use Mac OS Server as the Basic Host OS. As the Need for Resources are somehow limited for the other Systems (Solaris/Weblogic, Linux/Jenkins), we also decide to do a basic VM Setup with VMWare Fusion.

Since we have a decent Munin Monitoring Setup in our Company for all our Systems, we need some Monitoring for all Services used in our Setup:

Beside the Standard Plugins (like Network/CPU/RAM/Disk) that was basically

Jenkins CI
VMware Fusion
VPN

After searching through the Munin Plugin Repository we couldn’t find any plugins providing the necessary monitoring. So the only choice was to write your own set of plugins. Since all three Plugins use different Approaches for collecting the Data, i plan two writer three different posts here. One for each Plugin. The Sources are availible online here and might be added to the main Munin Repo as soon as the Pull Requests are accepted.

How Munin works

But first a brief overview of Munin. Munin is a TCP based Service that has normally one Master and one Node for each System that needs to be monitored. The Master Node ask all Nodes periodicly for Monitoring Updates.
The Node Service, delivering the Updated Data runs on Port 4949 per default. To add some level of security, you normal add a IP to a whitelist, that is allowed to query the Nodes Data.

You can use normal telnet for accessing the Nodes Data:

telnet localhost 4949

Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at amun

Every Node delivers Information about specific Services provided by Plugins. To get an overview about the configured plugins you do a:

# munin node at amun
list

df df_inode fusion_mem fusion_pcpu fusion_pmem if_en0 if_err_en0 load lpstat netstat ntp_offset processes users

A Plugin always provides a Configuration Output and a Data Output. By Default if you query a Plugin, you will always get the Data Output:

# munin node at amun
df

_dev_disk1s2.value 34
_dev_disk0s1.value 48
_dev_disk3s2.value 62
_dev_disk2s1.value 6
_dev_disk2s2.value 32

To trigger the Config Output you need to add a config to your command:

# munin node at amun
df config

graph_title Filesystem usage (in %)
graph_args --upper-limit 100 -l 0
graph_vlabel %
graph_scale no
_dev_disk0s1.label /Volumes/Untitled
_dev_disk1s2.label /
_dev_disk2s1.label /Volumes/System-reserviert
_dev_disk2s2.label /Volumes/Windows 7
_dev_disk3s2.label /Volumes/Data

You can also use the tool munin-run for doing a basic test (it will be installed when installing your munin-node Binary)

 munin-run df

_dev_disk1s2.value 34
_dev_disk0s1.value 48
_dev_disk3s2.value 62
_dev_disk2s1.value 6
_dev_disk2s2.value 32

munin-run df config

graph_title Filesystem usage (in %)
graph_args --upper-limit 100 -l 0
graph_vlabel %
graph_scale no
_dev_disk0s1.label /Volumes/Untitled
_dev_disk1s2.label /
_dev_disk2s1.label /Volumes/System-reserviert
_dev_disk2s2.label /Volumes/Windows 7
_dev_disk3s2.label /Volumes/Data

Summary

So a Plugin needs to provide an Output both modes:

Configuration Output when run with the config argument
The (normal) Data Output when called withouth additional arguments

Plugins are Shell Scripts that can be written in every Programming language that is supported by the Nodes Shell (e.g. Bash, Perl, Python, etc.)

Since it is one of the easier Plugins we will have a look at the Plugin, monitoring the VPN Connections at our Mac OS Server in the next Post.