Fixing nix Setup on MacOS Catalina

With MacOS Catalina (10.15), Apple decided to decrease the possibilities of system users to install software applications within the system. That means, that it is not possible anymore to install software at specific location in your system, since most system folder ware mounted read-only at boot to improve the overall system security.

That leads to an installation error if you want to install Nix in MacOS Catalina /nix will not be writable on macOS Catalina #2925 .

In another Issue above there are three Options mentioned by user garyverhaegen-da to circumvent this problem, all with some more or less hefty drawbacks.

The good thing is, that with the use of the APFS System, it is very easy possible to setup a specific FS for nix without the use of an DMG Image and mount it under the correct /nix path.

These are the steps you can do to fix the problem:

!! This approach does only work if you have a Disk that is formated with APFS
(that should always be the case, if your OS is running of a SSD).

You can check this with:

% diskutil list | grep APFS

…
0:      APFS Container Scheme -                      +250.8 GB   disk1
…

Of you have an output here, everything is fine. Otherwise you might have a look into How to Convert to APFS .

1 Create another Volume on your Disk for NIX

This can be done via the Disk Utility Program,

or via the CLI:

% sudo diskutil apfs addVolume disk1 ‘APFS’ nix

Will export new APFS Volume "nix" from APFS Container Reference disk1
Started APFS operation on disk1
Preparing to add APFS Volume to APFS Container disk1
Creating APFS Volume
Created new APFS Volume disk1s6
Mounting APFS Volume
Setting volume permissions
Disk from APFS operation: disk1s6
Finished APFS operation on disk1

After that, you should see a volume nix if you are running diskutil list.
The great thing is, that this volume does not need to have a fixed size. APFS shares the free disk spaces over all configured volumes!

1b (optional) encrypt disk

If you want to, you can also activate Disk Encryption for te new Disk. First you need to find out the name of your new Disk (if you did not note it from the previous command ;-)).

% diskutil list | grep nix

4:                APFS Volume nix                      7.7 GB    disk1s6

% sudo diskutil apfs encryptvolume disk1s6 -user disk

You need to enter a passphrase for the encryption. You have to remember that passphrase once – you can add it to your key chain later on.

Passphrase for the new "Disk" user (672C4CFF-34C6-4407-83ED-294C1C42E161):
Repeat passphrase:

After that the disk encryption will start in the background:

Starting background encryption with the new "Disk" crypto user on disk1s6
The new "Disk" user will be the only one who has initial access to disk1s6
The new APFS crypto user UUID will be 672C4CFF-34C6-4407-83ED-294C1C42E161
Background encryption is ongoing; see "diskutil apfs list" to see progress

2 setup Mount Point

MacOS Catalina does not allow to create folders directly under your Root Path /. But we can use another method to have MacOS create that folder for us. To do this, we have to add an entry into the file /etc/synthetic.conf:

% sudo bash -c “echo nix >> /etc/synthetic.conf”

Now, the next time, the system starts, a mount point /nix will be created. The next task is to have our Volume mounted at Boot.

3 setup Mount

For the Mount Configuration, we need to the UUID of the Volume. We can find this via the diskutil tool:

% diskutil info /dev/disk1s6 | grep UUID

Volume UUID:               1D9389C1-5676-4077-88F5-8D5304A0B1A6
Disk / Partition UUID:     1D9389C1-5676-4077-88F5-8D5304A0B1A6

In previoues version, you would just edit /etc/fstab and adding an entry there. But here is a clear info in that file to only edit it via vifs. The Tool vifs works like vi so an edit is quite easy.

% sudo vifs

We have to enter instert mode whith prssing the Key I.

After that, we can just paste this line into the editor window:

UUID=1D9389C1-5676-4077-88F5-8D5304A0B1A6 /nix apfs  rw

!! This UUID is only valid for my volume – your mileage may vary

We now have to write our changes: press ESC to leave the insert mode, type :w + ENTER to save the changes and :q + ENTER to exit the editor.

4 Reboot

Before we can start our nix Setup, we have to reboot the system.

If you have enabled the disk encryption, you will be asked for the passphrase after the system has restarted. You need to enter it once and mark “add to key chain”. The next time, the disk should be mounted automatically.

5 Setup Nix

You can now proceed with the installation of Nix. Everything should work as expected. Since it is a realy Mount Point and no Softlink, there should be no errors expected.

% sh <(curl https://nixos.org/nix/install)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2399  100  2399    0     0   9913      0 --:--:-- --:--:-- --:--:--  9995
downloading Nix 2.3.4 binary tarball for x86_64-darwin from 'https://nixos.org/releases/nix/nix-2.3.4/nix-2.3.4-x86_64-darwin.tar.xz' to '/var/folders/y3/29k7dx8s50l4y_l2tr78b_vh0000gn/T/nix-binary-tarball-unpack.XXXXXXXXXX.Fe7OlFzr'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    87  100    87    0     0    878      0 --:--:-- --:--:-- --:--:--   878
100 26.6M  100 26.6M    0     0  11.3M      0  0:00:02  0:00:02 --:--:-- 12.0M
Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation
performing a single-user installation of Nix...
copying Nix to /nix/store.............................................
installing 'nix-2.3.4'
building '/nix/store/96xp8q08cm412ibj8bhrgivx3hgc2gfx-user-environment.drv'...
created 7 symlinks in user environment
installing 'nss-cacert-3.49.2'
building '/nix/store/qjabndqd4kdjwlpiphwxrk84acy7x60k-user-environment.drv'...
created 9 symlinks in user environment
unpacking channels...
created 1 symlinks in user environment
modifying /Users/philipp/.bash_profile...

Installation finished!  To ensure that the necessary environment
variables are set, either log in again, or type

  . $HOME/.nix-profile/etc/profile.d/nix.sh

in your shell.

5b Hint

To have the nix commands in your path, you have to add the following to your .zshrc file, since Catalina uses ZSH as the default shell:

  . $HOME/.nix-profile/etc/profile.d/nix.sh

How to disable the www-data user to send emails with postfix

Sometimes an insecure configuration allows spammer to use the www-data user to send emails with you postfix server.
Normally this is the case, when you get a bunch of error email from your mailserver, that some emails from www-data@hostname.tld could not be delivered.

To be sure, that this situation cannot exist, you can add the www-data user to a block list:
In /etc/postfix/main.cf just add

authorized_submit_users = !www-data, static:all

and restart you postfix server.
You will now get a lot of error in you log like this one:

postfix/sendmail[8563]: fatal: User www-data(33) is not allowed to submit mail

After some time, the spammers will hopefully loose interest in you server :-).

20min Handson ZFS

ZFS is often called the last word in file systems.
It is a new approach to deal with large pools of disks originally invented by Sun.
It was later then ported to FreeBSD, MacOS (only 10.5) and Linux.

This text should show some of the basic feature of ZFS and demonstrate them handson by example.

Prerequisites

-> FreeBSD
-> Solaris
-> MacOS (only Userland)

In our example we use

SunOS openindiana 5.11 oi_151a5 i86pc i386 i86pc Solaris.

as an environment.

But most commands also work on the other systems.

Since we do all the work within a VM, our commands have the pattern:

Input VM:

command

Output VM:

result

Pool Creation

The first information we need is the number of disk, present in our environment.
There are several ways to get a basic disk listing. Under (Open-)Solaris this can be done with:

Input VM:

format < /dev/null

Output VM:

AVAILABLE DISK SELECTIONS:
0. c4t0d0 
/pci@0,0/pci8086,2829@d/disk@0,0
1. c5t0d0 
/pci@0,0/pci1000,8000@16/sd@0,0
2. c5t1d0 
/pci@0,0/pci1000,8000@16/sd@1,0
3. c5t2d0 
/pci@0,0/pci1000,8000@16/sd@2,0
4. c5t3d0 
/pci@0,0/pci1000,8000@16/sd@3,0
5. c5t4d0 
/pci@0,0/pci1000,8000@16/sd@4,0
6. c5t5d0 
/pci@0,0/pci1000,8000@16/sd@5,0
7. c5t6d0 
/pci@0,0/pci1000,8000@16/sd@6,0
8. c5t7d0 
/pci@0,0/pci1000,8000@16/sd@7,0

about Pools

With ZFS it is possible to create different kinds of pools on a specific number of disk.
You can also create several pools within one system.

The following Pools are possible and most commonly used:

Type	Info	Performance	Capacity	Redundancy	Command
JBOD	Just a bunch of disks. In theory it is possible to create on pool for each disk in the system, although this is not quite commonly used.	of each disk	of each disk	–	zpool create disk1 pool1 zpool create disk1 pool2 …
Stripe	This is equivalent to RAID0, the data is distributed over all disks in the pool. If one disks fails, all the data is lost. But you can also stripe several Pools (e.g. two raidz pools) to have better redundancy.	very high	N Disks	no	zpool create disk1 disk2 pool1
Mirror	This is equivalent to RAID1, the data is written to both disks in the Pool. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk.	normal	N-1 Disks	+1	zpool create mirror disk1 disk2 pool1
Raidz	This is equivalent to RAID5. One disk contains the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disks.	high	N-1 Disks	+1	zpool create raidz disk1 disk2 disk3 pool1
Raidz2	This is equivalent to RAID6. Two disks containing the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk with parity data.	high	N-2 Disks	+2	zpool create raidz2 disk1 disk2 disk3 disk4 pool1
Raidz3	There is no real equivalent existing for that one. You have basically three disks with parity data.	high	N-3 Disks	+3	zpool create raidz3 disk1 disk2 disk3 disk4 disk5 pool1

You can also add hot-spares for a better fallback behaviour, SSDs for caching reads (cache) and writes (logs).
I also created a benchmark with various combinations.

create a basic Pool (raidz)

Input VM:

zpool create tank raidz c5t0d0 c5t1d0 c5t2d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0

errors: No known data errors

(Raid5)

You already can access the newly created pool:

Input VM:

ls -al /tank

Output VM:

 
...
total 4
drwxr-xr-x  2 root root  2 2012-10-23 22:02 .
drwxr-xr-x 25 root root 28 2012-10-23 22:02 ..

create a basic Pool (raidz) with one spare drive

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
        spares
          c5t3d0    AVAIL   

errors: No known data errors

List the availibe Layout

Input VM:

zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   185K  1,46G         -     0%  1.00x  ONLINE  -

*The 1,5G does not reflect the real availible space. If you copy a 1G File to the Pool it will use 1,5G (1G + 512M Parity).

create a stripped pool

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 raidz1 c5t4d0 c5t5d0 c5t6d0

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            c5t4d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0

errors: No known data errors

(Raid50 = Raid5 + Raid5)

deal with disk failures

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0

Failure Handling

Input Host:

echo /dev/random >> 1.vdi

Wait for it…
or Input VM:

  pool: tank
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 66K in 0h0m with 0 errors on Tue Oct 23 22:14:19 2012
config:

        NAME          STATE     READ WRITE CKSUM
        tank          DEGRADED     0     0     0
          raidz1-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c5t0d0  DEGRADED     0     0    64  too many errors
              c5t3d0  ONLINE       0     0     0
            c5t1d0    ONLINE       0     0     0
            c5t2d0    ONLINE       0     0     0
        spares
          c5t3d0      INUSE     currently in use

errors: No known data errors

Input VM:

zpool clear tank
...
zpool detach tank c5t0d0
zpool replace tank c5t0d0 c5t7d0

Create File systems

Input VM:

zfs create tank/home
zfs create tank/home/user1
...
chown -R user:staff /tank/home/user1
...
zfs get all tank/home/user1
...
zfs set sharesmb=on tank/home/user1
...
zfs set quota=500M tank/home/user1

Copy File from MacOS into SMB Share.

Snapshot

Input VM:

zfs snapshot tank/home/user1@basic
...
zfs list
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home/user1@basic                0      -  42,6K  -

Input VM:

zfs snapshot -r tank/home@backup
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home@backup                     0      -  41,3K  -
tank/home/user1@basic                0      -  42,6K  -
tank/home/user1@backup               0      -  42,6K  -

Input VM:

zfs clone tank/home/user1@basic tank/home/user2

Output VM:

tank/home/user2          1,33K   894M  70,3M  /tank/home/user2

Restoring Snapshots

Delete ZIP File in SMB-Share.

Input VM:

ls -al tank/home/user1
...
zfs rollback tank/home/user1@backup

Output VM:

ls -al tank/home/user1

Resizing a Pool

Input VM:

zpool list
...
zpool replace tank c5t0d0 c5t4d0
zpool replace tank c5t1d0 c5t5d0
zpool replace tank c5t2d0 c5t6d0
...
zpool scrub tank
...
zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   381K  1,46G     1,50G     0%  1.00x  ONLINE  -

Input VM:

zpool set autoexpand=on tank

Using ZFS for Backups

Bash-Script

rsync -avrz --progress --delete /Users/user root@nas.local::user-backup/
backupdate=$(date "+%Y-%m-%d")
ssh root@nas.local zfs snapshot tank/backup@$backupdate

solving Security Error while starting Java WebStart (e.g. IPMI Remote)

Most of the IPMI Systems out there still using good old Java based Remote Applications to connect to the remote console.
Sine Java 8 update 111, the MD5 singing algorithm was marked as insecure (aka disabled) by Oracale (see Relase Notes for that Release ” Restrict JARs signed with weak algorithms and keys”).
You will get an “Security Error while using MD5withRSA Signature”:

The only solution to fix this error is to have your Hardware Vendor to update the IPMI Firmware with JARs, signed with a more up to date singing algorithm. A work around is to re-enable MD5 for the time being. For that you need to get into your Browser Java Installation.
On my Mac this is in

/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home

You need to edit lib/security/java.security and remove MD5 from jdk.jar.disabledAlgorithms.

Change

jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

jdk.jar.disabledAlgorithms=MD2, RSA keySize < 1024

Find a Class in a bunch of JAR Files

for j in *.jar; do echo $j:; jar -tvf "$j" | grep -Hsi ClassName; done

Iterate over every jar in your folder.
Output every JAR Name (to match found classes to JARs lateron)
List Content of every JAR ( jar -tvf “$j” )
Grep for that specific ClassName grep -Hsi ClassName

Original Post: https://coderwall.com/p/7d-mta/find-a-class-in-a-bunch-of-jar-files

SSH Tunnel RDP Session over another Host

Let’s say you want to tunnel an RDP connection via SSH.
User: phaus
SSH-Server: ssh.example.com
RDP-Server: win2k8.example.com

#!/bin/bash
ssh -L13389:win2k8.example.com:3389 phaus@ssh.example.com

You need to run that script and keep your session open.
You can then access the remote System with a RDP Client over localhost 13389.

Original Post: https://coderwall.com/p/wzvxhg/ssh-tunnel-rdp-session-over-another-host

Fixing Error installing cocoapods on MacOS

If you installing cocoapods on a vanilla MacOS System you might get some error like this:

# sudo gem install cocoapods
Building native extensions.  This could take a while...
ERROR:  Error installing cocoapods:
    ERROR: Failed to build gem native extension.

    /usr/local/bin/ruby extconf.rb
creating Makefile

make
gcc -I. -I/usr/local/include/ruby-1.9.1/x86_64-darwin10.6.0 -I/usr/local/include/ruby-1.9.1/ruby/backward -I/usr/local/include/ruby-1.9.1 -I. -DJSON_GENERATOR -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE   -fno-common -D_XOPEN_SOURCE=1 -fno-common -pipe -O3 -Wall -O0 -ggdb  -o generator.o -c generator.c
gcc -dynamic -bundle -o generator.bundle generator.o -L. -L/usr/local/lib -L. -L/usr/local/lib -Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,suppress -Wl,-flat_namespace  -lruby.1.9.1  -lpthread -ldl -lobjc

make  install
/opt/local/bin/gmkdir -p /usr/local/lib/ruby/gems/1.9.1/gems/json-1.8.0/ext/json/ext/generator/.gem.20130925-59217-5zt5ke/json/ext
make: /opt/local/bin/gmkdir: No such file or directory
make: *** [/usr/local/lib/ruby/gems/1.9.1/gems/json-1.8.0/ext/json/ext/generator/.gem.20130925-59217-5zt5ke/json/ext] Error 1

The important error is make: /opt/local/bin/gmkdir: No such file or directory

You need to install coreutils for fixing this.

With MacPorts:

sudo port install coreutils

For Homebrew:

sudo brew install coreutils

Original Post: https://coderwall.com/p/ehijxg/fixing-error-installing-cocoapods-on-macos

use 3rd party Dependencies in your Maven Project

Deploy your JAR to your local (file) Repository. You shold also commit this to your CVM System. You can also add JARs as Javadoc or Source.

    mvn deploy:deploy-file \
        -DgroupId=<group-id> \
        -DartifactId=<artifact-id> \
        -Dversion=<version> \
        -Dfile=<path-to-file> \
        -Dpackaging=jar \
        -Durl=file:///<project-dir>/repo 
        ( -Dclassifier=javadoc | sources  )

You can now use these JARs in your pom File if they were in a normal Repository:

    <project>
        ...
        <repositories>
        ...
            <repository>
                <id>project.local</id>
                <name>project</name>
                <url>file:${project.basedir}/repo</url>
            </repository>     
        ...
        </repositories>
        ...
    </project>

Original Post: https://coderwall.com/p/r28cew/use-3rd-party-dependencies-in-maven-project

Map Char to Alpha Position with Java

Sometimes, you would like to have Mapping Char to Int ( A = 0, B = 1, etc.)
One easy way to do this is using indexOf

"ABCDEFGHIJKLMNOPQRSTUVWXYZ".indexOf('F')

// returns 5

Original Post: https://coderwall.com/p/ql-wng/map-char-to-alpha-position-with-java

using an ISP SMTP for sendmail

Just a Link: http://cri.ch/linux/docs/sk0009.html