Philipps Blog


2014/03/28  Creating encrypted Volumes on ZFS Pools

Filed under:Hacking,ZFS — Philipp @ 8:21 pm

One of the most anticipated Features of ZFS is transparent Encryption. But since Oracle decided to do not make updates from Solaris 11 availible as Open Source, the Feature of on-Disk Encryption is not availible on Illumos (e.g. Open-Source) based Distributions. But there are some ways to create transparent encrypted ZPools with current avaiblibe ZFS Version using pktool, lofiadm, zfs and zpool.

lofiadm- administer files available as block devices through lofi

http://www.unix.com/man-page/opensolaris/1m/lofiadm

That means, you can use normal Files as Block Devices while adding some Features to them (e.g. compression and also encryption). The Goal of this Post is to create a transparent encrypted Volume, that uses a Key-File for deryption (that might be stored on an usb stick or will be uploaded via a Browser once to mount the device). For an easy Start, i created a Vagrant File based on OmniOS here.

If you do not know Vagrant, here is an easy Start for you:

  1. Get yourself a VirtualBox Version matching your Platform: https://www.virtualbox.org/wiki/Downloads
  2. Get yourself a Vagrant Version matching your Platform: http://www.vagrantup.com/downloads.html
  3. Move to the Folder where you have saved your Vagrantfile
  4. Start your Box (will need some time, since the OmniOS Box will needs to be downloaded first)
    vagrant up
  5. After your box is finished, you can ssh into it with
    vagrant ssh
  6. Have a look around:
    zpool status

    You will find exactly one (Root-) Pool configured in that system:

      pool: rpool
     state: ONLINE
      scan: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            rpool       ONLINE       0     0     0
              c1d0s0    ONLINE       0     0     0

Next we want to create our encrypted Device, for that we need some “files” for using them with lofiadm. One very handy feature of ZFS is the possibility to also create Volumes (ZVols) in your ZPool.
First we need to finde out how big our Pool is:

zpool list

will give us an overview of the configured Volumes and File Systems:

NAME           SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
rpool         39,8G  2,28G  37,5G         -     5%  1.00x  ONLINE  -
vagrant-priv      -      -      -         -      -      -  FAULTED  -

So we have roughly around 37G free space. For this Test we would like to create an encrypted Volume with 2G of Space.
Creating a ZVol is as easy as creating a normal ZFS Folder:

sudo zfs create -V 2G rpool/export/home/vagrant-priv

You can now see the new ZVol with the reserved size of 2G:

zfs list
NAME                             USED  AVAIL  REFER  MOUNTPOINT
rpool                           5,34G  33,8G  35,5K  /rpool
rpool/ROOT                      1,74G  33,8G    31K  legacy
rpool/ROOT/omnios               1,74G  33,8G  1,46G  /
rpool/ROOT/omniosvar              31K  33,8G    31K  legacy
rpool/dump                       512M  33,8G   512M  -
rpool/export                    2,06G  33,8G    32K  /export
rpool/export/home               2,06G  33,8G    41K  /export/home
rpool/export/home/vagrant-priv  2,06G  35,9G    16K  -
rpool/swap                      1,03G  34,8G  34,4M  -

Next we need a Key for en-/de-crypting the Device. That can be done with the pktool:

> pktool genkey keystore=file outkey=lofi.key keytype=aes keylen=256 print=y
< Key Value ="93af08fcfa9fc89724b5ee33dc244f219ac6ce75d73df2cb1442dc4cd12ad1c4"

We can now use this key with lofiadm to create an encrypted Device:

> sudo lofiadm -a /dev/zvol/rdsk/rpool/export/home/vagrant-priv -c aes-256-cbc -k lofi.key
< /dev/lofi/1

lofi.key is the File that contains the Key for the Encryption. You can keep it in that folder or move it to another device. If you want to reactivate the device (we will see later how to do this), you will need that key file again.
/dev/lofi/1 is our encrypted Device. We can use that for creating a new (encrypted) ZPool:

sudo zpool create vagrant-priv /dev/lofi/1

You know can use that Pool as a normal ZPool (including Quotas/Compression, etc.)

> zpool status

< pool: vagrant-priv
 state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        vagrant-priv   ONLINE       0     0     0
          /dev/lofi/1  ONLINE       0     0     0

errors: No known data errors

You should change the Folder permissions of that mount-point:

sudo chown -R vagrant:other vagrant-priv

Creating some Test-Files:

cd /vagrant-priv/
mkfile 100m file2
> du -sh *
< 100M   file2

So what happens if we want to deactivate that Pool?

  1. Leave the Mount-Point:
    cd /
  2. Deactivate the Pool:
    sudo zpool export vagrant-priv
  3. Deactivate the Lofi Device:
    sudo lofiadm -d /dev/lofi/1

That’s all. Now let’s reboot the system and let us see how we can re-attach that Pool again.

Leave the Vagrant Box:

> exit
< logout
< Connection to 127.0.0.1 closed.

Restart the Box:

> vagrant halt
< [default] Attempting graceful shutdown of VM...
> vagrant up
...
< Waiting for machine to boot. This may take a few minutes...
< [default] VM already provisioned. Run `vagrant provision` or use `--provision` to force it

Re-Enter the Box:

vagrant ssh

So where is our Pool?

zpool status

Only gives us the default root-Pool.
First we need to re-create our Lofi-Device:

> sudo lofiadm -a /dev/zvol/rdsk/rpool/export/home/vagrant-priv -c aes-256-cbc -k lofi.key
< /dev/lofi/1

Instead of creating a new ZPool (that would delete our previous created Data), we need to import that ZPool. That’s can be done in two steps, using zpool. First we need to find our Pool:

sudo zpool import -d /dev/lofi/

That lists all ZPools, that are on Devices in that Directory. We need to find the id of “our” Pool (that needs to be done once, since that id stays the same, as long as the Pool exitsts).

...   
   pool: vagrant-priv
     id: 1140053612317909839
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        vagrant-priv   ONLINE
          /dev/lofi/1  ONLINE
...

We can now import that ZPool using the id 1140053612317909839:

sudo zpool import -d  /dev/lofi/ 1140053612317909839

After that we can again access our Pool as usual:

> cd /vagrant-priv/
> du -sh *
< 100M   file2

2014/02/09  Managing Mac OS Software with Munki and Subversion

Filed under:Mac,Tech — Philipp @ 9:08 pm

At the Lisa ’13, some folks from Google did a talk how they managing all their Desktop (and Server?) Macs at Google. Besides obvious things (like using Puppet), they mentioned another Tool, Munki, for rolling out Software and Software Updates to different Clients. Since i am using several Mac Machines (Laptop, Workstation and some VMs) that used to have a quite similar Software Stack, i decided to give Munki a try. Instead of using a dedicated Webserver, i decided to go with a Subversion Repository, for having User Authentication and Versioning at the Backend.

Munki uses some concepts for organising its stuff:

  • catalogs: these are basically the Listings of Applications. Each Catalog contains some Applications to be installed.
  • manifests: these are configurations for the specific client setups – e.g. Java-Dev. You can combine several Manifest Files while including one in another. You can also define mandatory and optional Packages here.
  • pkgs: here are the basic DMG/PGK Packages stored. All Filenames are unique so you can have several Versions of one Program in one Repository.
  • pkgsinfo: Here the Basic Application Info is stored. You can have Dependencies between Packages, as well as Requirements for installing Packages.

There is an excellent Starting Guide here and a description for a Demo-Setup, how to setup a basic Munki Installation. So i won’t repeat it here.

pkgs and pkgsinfo can be strcutured into sub-folders.
My actuals setup looks like this:

> tree -L 2
.
├── catalogs
│   ├── all
│   └── testing
├── manifests
│   ├── developer_munki_client
│   └── test_munki_client
├── pkgs
│   ├── dev
│   ├── media
│   ├── utils
│   └── work
└── pkgsinfo
    ├── dev
    ├── media
    ├── utils
    └── work

So you basically configure your munki-client towards

bash-3.2$ /usr/local/munki/munkiimport --configure
Path to munki repo (example: /Users/philipp/munki  
Repo fileshare URL (example:  afp://munki.example.com/repo): https://example.com/svn/munki
pkginfo extension (Example: .plist): 
pkginfo editor (examples: /usr/bin/vi or TextMate.app): TextMate.app 
Default catalog to use (example: testing): testing

After that you can use munkiimport ##path-to-dmg## for importing Applications to Munki. After you did the final Import, you can use a Tool like MunkiAdmin to configure your Client-Setup and Application Dependencies.

The next step is to commit your changes to a Repository (that is reachable under https://example.com/svn/munki). You need to update every change to the Munki Repository to keep all Clients actual. The last Step is to implement the HTTP Basic Auth Access to the Subversion Repository. There is a good Description for that as well. You need to update your /Library/Preferences/ManagedInstalls.plist Files – that should actually be moved to /private/var/root/Library/Preferences/ManagedInstalls.plist, since it now contains some User Credentials. To add this Credentials you should use this Command, where You need to have username:password as a Base64 encoded String.

defaults write /Library/Preferences/ManagedInstalls AdditionalHttpHeaders -array "Authorization: Basic V...Q="

2013/09/09  Plotting UNIX Processes with D3

Filed under:Graphics,Javascript — Philipp @ 7:47 pm

I created another Plot, using D3js:

PS Graph with D3

You can find all Scripts here.

I am still have to figure out how to do graph updates and add label to the nodes.

2013/09/03  Plotting UNIX Processes with DOT

Filed under:Graphics,Javascript,Tooling — Philipp @ 8:41 pm

Inspired by this Post, i started playing around with ps, nodejs and GraphViz.

After reading some ps man Pages, i found the necessary ps parameters.
For MacOS i used

ps -A -c -o pid,ppid,pcpu,comm,uid -r

For Linux i used

ps -A -o pid,ppid,pcpu,comm,uid

You then get some Output like:

    PID    PPID %CPU COMMAND           UID
      1       0  0.0 init                0
      2       1  0.0 kthreadd            0
      3       2  0.0 migration/0         0
      4       2  0.0 ksoftirqd/0         0
      5       2  0.0 migration/0         0
      6       2  0.0 watchdog/0          0

So you are getting the ProcessID, the Parent ProcessID, CPU Usage (i am not using for plotting atm), the Command and the UserID.
I created a simple Node Script, that you can run either directly under MacOS (for all other Unices you need to update the ps command).
Or you can give the script a previous generated ps output for parsing:

plotPS.sh /tmp/host.log > /tmp/host.dot

The resulting DOT Code is then Piped into a DOT File.

Here are some examples:

My MacOS Laptop:

MacBook Pro

bigger

A Sinlge Linux Host with Dovecot and Apache2/Passenger:

Apache2 / Mail Server

bigger

A Linux Host with OpenVZ and KVM Instances:

OpenVZ / KVM Host

bigger

In the original Post, there were also Dependencies between CPU Usage and Size of the Graphical Nodes, also it would be more useful to only plotting the processes of one VM from its inside.
But i guess for one evening the result is okay :-).

2013/08/04  Build and Test Project TOX under MacOS

Filed under:Build,Hacking,Network,Tooling — Philipp @ 3:43 pm

Some Steps to do

  1. You need to have XCode with installed CLI Tools (see here)
  2. If you are using MacPorts (you really should), you need to install all necessary Dependencies:
    port install libtool automake autoconf libconfig-hr libsodium cmake
  3. Checkout the Project TOX Core Repository:
    git clone --recursive https://github.com/irungentoo/ProjectTox-Core.git
  4. cd ProjectTox-Core
    cmake .
    make all
  5. You need two tools:
    DHT_bootstrap in /other
    and nTox in /testing
  6. Bootstrap Tox (aka get your Public Key):
    ./DHT_bootstrap
    Keys saved successfully
    Public key: EA7D7BD2566A208F83F81F8876DE6C1BDC1F8CA1788300296E5D4F4CB142CD77
    Port: 33445

    The key is also in PUBLIC_ID.txt in the same Directory.

  7. Run nTox like so:
    ./ntox 198.46.136.167 33445 728925473812C7AAC482BE7250BCCAD0B8CB9F737BF3D42ABD34459C1768F854

    Where:

    Some Tox Node
    198.46.136.167
    Port of that TOX Node
    33445
    Public Key of that TOX Node
    728925473812C7AAC482BE7250BCCAD0B8CB9F737BF3D42ABD34459C1768F854
  8. Et voilà:
    /h for list of commands
    [i] ID: C759C4FC6511CEED3EC846C0921229CA909F37CAA2DCB1D8B31479C5838DF94C
    >>

    You can add a friend:

    /f ##PUBLIC_ID##

    List your friends:

    /l

    Message a friend:

    /m ##friend_list_index##  ##message##

2013/08/01  Downgrading Subversion from 1.8 to 1.7 in MacPorts

Filed under:Snippets,Tooling — Philipp @ 11:19 pm
bash-3.2# cd /tmp
bash-3.2# svn co http://svn.macports.org/repository/macports/trunk/dports/devel/subversion --revision 108493
A    subversion/files
A    subversion/files/patch-Makefile.in.diff
A    subversion/files/patch-osx_unicode_precomp.diff
A    subversion/files/config_impl.h.patch
A    subversion/files/servers.default
A    subversion/Portfile
Ausgecheckt, Revision 108493.
bash-3.2# cd subversion/
bash-3.2# port install
--->  Computing dependencies for subversion
--->  Fetching archive for subversion
--->  Attempting to fetch subversion-1.7.10_1.darwin_12.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/subversion
--->  Attempting to fetch subversion-1.7.10_1.darwin_12.x86_64.tbz2.rmd160 from http://mse.uk.packages.macports.org/sites/packages.macports.org/subversion
...
--->  Scanning binaries for linking errors: 100.0%
--->  No broken files found.
bash-3.2# port installed subversion
The following ports are currently installed:
  subversion @1.7.10_1
  subversion @1.8.1_1 (active)
bash-3.2# port activate subversion @1.7.10_1
--->  Computing dependencies for subversion
--->  Deactivating subversion @1.8.1_1
--->  Cleaning subversion
--->  Activating subversion @1.7.10_1
--->  Cleaning subversion

2013/07/31  Creating a SmartOS Dataset for Windows Server 2012 r2

Filed under:SmartOS,Uncategorized,Virtualisierung — Tags: — Philipp @ 12:21 am

I postet about SmartOS almost 1 1/2 Year ago. Since i am still using SmartOS here and then for primary testing purpose, i decided to create a new Post about how to make your own Datasets, to make several VMs based on the same Image possible. As Microsoft also introduces their latest Server OS (Windows Server 2012 r2) as a Preview Version, i choosed this OS for the Test, since i guess besides SmartOS itself and Linux, which has already a lot of Dataset present, Windows might be the next common OS to use in a VM.

Prerequisites

You should already have these Software Packages:

SmartOS Download
http://wiki.smartos.org/display/DOC/Download+SmartOS
Windows Server 2012 R2 Preview
http://technet.microsoft.com/de-DE/evalcenter/dn205287.aspx you should rename the ISO e.g. to win2k12r2_de.iso
TightVNC Java Viewer JAR
http://www.tightvnc.com/download.php

You should have SmartOS installed on a proper Hardware (at least > 50GB Disk and proper KVM support – this means a recent Intel CPU and working VT-x/EPT). I am using VMWare Fusion 5 for this, since there is a specific Setting for supporting VM internal Virtualization.

The Setup of SmartOS is pretty straigh forward. You can have a short description here [1].

In this tutorial i will choose [client] when you have to run the command on your Workstation and [smartos] when you should run it on your SmartOS Host (e.g. via SSH).

SmartOS Configuration

You should check the current Host Configuration of you SmartOS System first. The CLI Tools for this are sometimes different than e.g. in Linux [2].

First you need to find your Host Network Gateway (here 172.16.108.2)

[smartos] /usr/bin/netstat -r

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              172.16.108.2         UG        3        443 e1000g0
localhost            localhost            UH        2        310 lo0
172.16.108.0         00-50-56-28-a8-14    U         5    1694314 e1000g0

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
localhost                   localhost                   UH      2      12 lo0

[smartos] /usr/bin/sysinfo -p

Live_Image='20130629T040542Z'
System_Type='SunOS'
Boot_Time='1375215743'
Manufacturer='VMware, Inc.'
Product='VMware Virtual Platform'
Serial_Number='VMware-56 4d 56 5e e8 ff 08 7b-38 81 e6 57 ac d8 b9 7d'
VM_Capable='true'
CPU_Type='Unknown'
CPU_Virtualization='vmx'
CPU_Physical_Cores=4
Nic_Tags=admin
Setup=''
UUID='564d565e-e8ff-087b-3881-e657acd8b97d'
Hostname='00-50-56-28-a8-14'
CPU_Total_Cores=4
MiB_of_Memory=8191
Disk_c1t0d0_size_in_GB=268
Disk_c1t1d0_size_in_GB=268
NIC_admin='e1000g0'
Network_Interface_e1000g0_MAC_Address='00:50:56:28:a8:14'
Network_Interface_e1000g0_IPv4_Address='172.16.108.135'
Network_Interface_e1000g0_NIC_Names='admin'
Network_Interface_e1000g0_Link_Status='up'
Network_Interface_e1000g1_MAC_Address='00:50:56:21:c3:ec'
Network_Interface_e1000g1_IPv4_Address=''
Network_Interface_e1000g1_NIC_Names=''
Network_Interface_e1000g1_Link_Status='up'
Bootparam_console='text'
Bootparam_root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA'
Bootparam_smartos='true'

[smartos] /usr/sbin/dladm show-link

LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       vmwarebr   --
e1000g1     phys      1500   up       --         --
vmwarebr0   bridge    1500   up       --         e1000g0

VM Setup

VM Configuration

[smartos] /usr/bin/cat /opt/win2k12r2_vm.json

{
  "brand": "kvm",
  "alias": "win2k12r2",
  "vcpus": 2,
  "autoboot": false,
  "ram": 2048,
  "resolvers": ["8.8.8.8"],
  "disks": [
    {
       "boot": true,
       "model": "ide", 
      "size": 40960
    }
  ],
  "nics": [
    {
      "nic_tag": "admin",
      "model": "e1000",
      "ip": "172.16.108.201",
      "netmask": "255.255.255.0",
      "gateway": "172.16.108.2",
      "primary": 1
    }
  ]
}

Your main Tool for VM Administration ist vmadm [3].

[smartos] /usr/sbin/vmadm create -f /opt/win2k12r2_vm.json

Successfully created VM 37d3cef6-01a6-4b25-a927-928cc2681744

[smartos] /usr/sbin/vmadm list

UUID                                  TYPE  RAM      STATE             ALIAS
37d3cef6-01a6-4b25-a927-928cc2681744  KVM   2048     stopped           win2k12r2

[smartos] /usr/sbin/zfs list

NAME                                               USED  AVAIL  REFER  MOUNTPOINT
zones                                             52,3G   192G   652K  /zones
zones/37d3cef6-01a6-4b25-a927-928cc2681744        39,5K  10,0G  39,5K  /zones/37d3cef6-01a6-4b25-a927-928cc2681744
zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0    40G   232G    16K  -
zones/config                                        41K   192G    41K  legacy
zones/cores                                         62K  10,0G    31K  /zones/global/cores
zones/cores/37d3cef6-01a6-4b25-a927-928cc2681744    31K  10,0G    31K  /zones/37d3cef6-01a6-4b25-a927-928cc2681744/cores
zones/dump                                        4,00G   192G  4,00G  -
zones/opt                                         31,5K   192G  31,5K  legacy
zones/swap                                        8,25G   200G    16K  -
zones/usbkey                                       127K   192G   127K  legacy
zones/var                                         2,08M   192G  2,08M  legacy

You need to copy the installation ISO Image into your VM Zone:

[client] scp win2k12r2_de.iso root@172.16.108.135:/zones/37d3cef6-01a6-4b25-a927-928cc2681744/root/

After that you can boot your VM with the ISO Image attached:

[smartos] /usr/sbin/vmadm boot 37d3cef6-01a6-4b25-a927-928cc2681744 order=cd,once=d cdrom=/win2k12r2_de.iso,ide

Successfully started VM 37d3cef6-01a6-4b25-a927-928cc2681744

You can access your VM via a VNC Viewer (we are using TightVNC). You can get the necessary connection setting via:

[smartos] /usr/sbin/vmadm info 37d3cef6-01a6-4b25-a927-928cc2681744 vnc

{
  "vnc": {
    "host": "172.16.108.135",
    "port": 34783,
    "display": 28883
  }
}

You can now extract the TightVNC Zip you downloaded before. Withing the folder classes you type:

[client] java VncViewer HOST 172.16.108.135 PORT 34783

The Port and the Host URL may differ. The Port will change everytime your VM reboots.

Setup

VNCSetup1
VNCSetup2

Running System

After a Reboot, the Login Screen greets us:

VNCRunning1

The Network also seems to work :-).

VNCRunning2

Dataset Creation

We can now start to create our VM Package. For that we need to shutdown our VM:

VNCShutdown1

VNCShutdown2

[smartos] /usr/sbin/vmadm list

UUID                                  TYPE  RAM      STATE             ALIAS
37d3cef6-01a6-4b25-a927-928cc2681744  KVM   2048     stopped           win2k12r2

The first step is to create a ZFS Snapshot of that specific VM Disk:

So let’s have a look:
[smartos] /usr/sbin/zfs list

NAME                                               USED  AVAIL  REFER  MOUNTPOINT
zones                                             52,3G   192G   652K  /zones
zones/37d3cef6-01a6-4b25-a927-928cc2681744        39,5K  10,0G  39,5K  /zones/37d3cef6-01a6-4b25-a927-928cc2681744
zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0    40G   232G    16K  -
zones/config                                        41K   192G    41K  legacy
zones/cores                                         62K  10,0G    31K  /zones/global/cores
zones/cores/37d3cef6-01a6-4b25-a927-928cc2681744    31K  10,0G    31K  /zones/37d3cef6-01a6-4b25-a927-928cc2681744/cores
zones/dump                                        4,00G   192G  4,00G  -
zones/opt                                         31,5K   192G  31,5K  legacy
zones/swap                                        8,25G   200G    16K  -
zones/usbkey                                       127K   192G   127K  legacy
zones/var                                         2,08M   192G  2,08M  legacy

It seems, that our disk is zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0 a zvol.

[smartos] /usr/sbin/zfs snapshot zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0@dataset

Let’s look for the Snapshot:

[smartos] /usr/sbin/zfs list -t snapshot

NAME                                                       USED  AVAIL  REFER  MOUNTPOINT
zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0@dataset      0      -  7,36G  -

There it is. Now we need to compress it, since this will be our vanilla Image.

[smartos] /usr/sbin/zfs send zones/37d3cef6-01a6-4b25-a927-928cc2681744-disk0@dataset | /usr/bin/gzip > /tmp/win2k12r2_de.zvol.gz

Wait…

Now we only need size and the SHA1 Checksum of that Image for the Manifest file.

[smartos] /usr/bin/digest -a sha1 win2k12r2_de.zvol.gz

Wait…

0d955b91973bdaccc30ffa75b856709ce9a1953a

[smartos] /usr/bin/ls /tmp

...
-rw-r--r--   1 root     root     3933399939 Jul 30 22:20 win2k12r2_de.zvol.gz
...

So our matching Dataset Manifest File might look like this:

[smartos] /usr/bin/cat /tmp/win2k12r2_vm.json

{
    "name": "win2k12r2de",
    "version": "1.0",
    "type": "zvol",
    "cpu_type": "host",
    "description": "Windows Server 2012 R2 Preview DE",
    "created_at": "2013-07-31T11:40:00.000Z",
    "updated_at": "2013-07-31T11:40:00.000Z",
    "published_at": "2013-07-31T11:40:00.000Z",
    "os": "windows",
    "image_size": "40960",
    "files": [
        {
            "path": "win2k12r2_de.zvol.gz",
            "sha1": "0d955b91973bdaccc30ffa75b856709ce9a1953a",
            "size": 3933399939
        }
    ],
    "requirements": {
        "networks": [
            {
                "name": "net0",
                "description": "public"
            }
        ]
    },
    "disk_driver": "ide",
    "nic_driver": "e1000",
    "uuid": "857ea9a0-f965-11e2-b778-0800200c9a66",
    "creator_uuid": "0e70a5a1-0115-4a2b-b260-94723fd31bf1",
    "vendor_uuid": "0e70a5a1-0115-4a2b-b260-94723fd31bf1",
    "owner_uuid": "0e70a5a1-0115-4a2b-b260-94723fd31bf1",
    "creator_name": "Philipp Haussleiter",
    "platform_type": "smartos",
    "urn": "smartos:phaus:win2k12r2de:1.0"
}

You can now install the new Dataset to your local Image Store:

[smartos] /usr/sbin/imgadm install -m /tmp/win2k12r2_de.json -f /tmp/win2k12r2_de.zvol.gz

Installing image 857ea9a0-f965-11e2-b778-0800200c9a66 (win2k12r2de 1.0)
857ea9a0-f965-11e2-b778-0800200c9a66      [==============================>                 ]  19% 727.97MB  11.39MB/s  4m25s
…
Installed image 857ea9a0-f965-11e2-b778-0800200c9a66 to "zones/857ea9a0-f965-11e2-b778-0800200c9a66".

Wait…

[smartos] /usr/sbin/imgadm list

UUID                                  NAME         VERSION  OS       PUBLISHED
…
857ea9a0-f965-11e2-b778-0800200c9a66  win2k12r2de  1.0      windows  2013-07-31T11:40:00Z
…

You can now using this image as a normal SmartOS Dataset with the UUID 857ea9a0-f965-11e2-b778-0800200c9a66.

Feel free to comment if you have questions or problems.

Links

[1] SmartOS – Basic Setup
http://blog.smartcore.net.au/smartos-the-basics/
[2] The Linux-to-SmartOS Cheat Sheet
http://wiki.smartos.org/display/DOC/The+Linux-to-SmartOS+Cheat+Sheet
[3] Using vmadm to manage virtual machines
http://wiki.smartos.org/display/DOC/Using+vmadm+to+manage+virtual+machines
[4] Some extra Configuration Options for SmartOS
http://wiki.smartos.org/display/DOC/extra+configuration+options
[5] How to create a SmartOS Dataset
http://www.nickebo.net/creating-smartos-datasets

2013/07/28  Adding LDAP Authentication to a Play! 2 Application

Filed under:Hacking,Java,Play! Framework,Tooling — Philipp @ 2:35 pm

As of Play! 1 is not really supported anymore, i will describe the steps for accessing Data from your LDAP Directory with your Play! 2 with this Post.

Prerequisites

As also mentioned in my last Post, for this example we are using the Vagrant vagrant-rundeck-ldap VM, I already mentioned here.

Setup

After you setup a basic Play! 2 Application with

play new ldap-test

We need to update our Applications Dependencies. To achieve this, we need to change project/Build.scala:


import sbt._
import Keys._
import play.Project._

object ApplicationBuild extends Build {

  val appName         = "play2-ldap-example"
  val appVersion      = "1.0-SNAPSHOT"

  val appDependencies = Seq(
    // Add your project dependencies here,
    javaCore,
    javaJdbc,
    javaEbean,
    "com.innoq.liqid" % "ldap-connector" % "1.3"
  )

  val main = play.Project(appName, appVersion, appDependencies).settings(
    // Add your own project settings here      
  )
}

The Command:


play compile

Should download all necessary Project Depedencies and will do an initial Compilation of all your Project Files (currently not really many).

You also need to add the LDAP Settings to your Applications configuration. For this example we will use an external Properties File conf/ldap.properties:


# Settings for LiQID
# ~~~~~
ldap.user.objectClasses=person
ldap.group.objectClasses=groupOfUniqueNames

default.ldap=ldap1
# LDAP Listsing, divided by ","
ldap.listing=ldap1

ldap1.ou_people=ou=users
ldap1.ou_group=ou=roles
ldap1.url=ldap://localhost:3890
ldap1.principal=dc=Manager,dc=example,dc=com
ldap1.credentials=password

ldap1.base_dn=dc=example,dc=com
ldap1.admin.group.id=admin

ldap1.user.id.attribute=cn
ldap1.user.object.class=person

ldap1.group.id.attribute=cn
ldap1.group.object.class=groupOfUniqueNames
ldap1.group.member.attribute=uniqueMember

Implementation

Since Play1 2 does not really support the use of Before Filters, we will use Custom Actions for making our Login Authentication work.
We will create a new Package app/actions with two new files: an annotation interface BasicAuth and the implementation itself BasicAuthAction.
Annotations will be used to Set a specific Controller to use Basic Auth.
So lets start with BasicAuth:


package actions;

import play.mvc.With;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@With(BasicAuthAction.class)
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD, ElementType.TYPE })
@Inherited
@Documented
public @interface BasicAuth {
}

After that you can annotate Controllers with @BasicAuth (but this won’t work, since the Implementation is still missing).
Here then the BasicAuthAction:


package actions;

import com.ning.http.util.Base64;

import models.User;
import play.mvc.Action;
import play.mvc.Http.Context;
import play.mvc.Result;

public class BasicAuthAction extends Action

As you can see, there are no LDAP specific Dependencies at all, since all necessary Logic is the User Model, in User.authenticate(username, password).
So let’s have a look into that Model:


package models;

import play.Play;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.ldap.connector.LdapUser;
import com.innoq.liqid.utils.Configuration;

public class User {

	private static LdapHelper HELPER = getHelper();

	public String sn;
	public String cn;
	public String dn;

	public User(String cn) {
		this.cn = cn;
	}

	public String toString() {
		StringBuilder sb = new StringBuilder();
		sb.append("cn: ").append(cn).append("\n");
		sb.append("sn: ").append(sn).append("\n");
		sb.append("dn: ").append(dn).append("\n");
		return sb.toString();
	}

	public static User authenticate(String username, String password) {
		if (HELPER.checkCredentials(username, password)) {
			return new User(username);
		}
		return null;
	}

	public static User getUser(String username) {
		LdapUser ldapUser = (LdapUser) LdapHelper.getInstance().getUser(
				username);
		User user = new User(username);
		user.cn = ldapUser.get("cn");
		user.sn = ldapUser.get("sn");
		user.dn = ldapUser.getDn();
		return user;
	}

	private static LdapHelper getHelper() {
		Configuration.setPropertiesLocation(Play.application().path()
				.getAbsolutePath()
				+ "/conf/ldap.properties");
		return LdapHelper.getInstance();
	}
}

You also have a static Instace of that LDAP Helper, but authenticate User Credentials and Login are in two different Methods.
Last thing is to Load a User from the LDAP Directory:

Here the Admin Controller:


package controllers;

import models.User;
import actions.BasicAuth;
import play.mvc.Controller;
import play.mvc.Result;
import views.html.Admin.index;

@BasicAuth
public class Admin extends Controller {
    public static Result index() {
    	User u = User.getUser(request().username());
        return ok(index.render("Hello Admin!", u));
    }
}

And here the used Template File:


@(message: String, user: User)

@main("Admin Index") {
@{message} 
<p>   
<pre>
@{user}
</pre>
</p>
}

Links

You can find the Sources of that Library here: https://github.com/innoq/LiQID
You can find an example Project here: https://github.com/phaus/play-ldap/tree/master/play2-ldap-example

  Adding LDAP Authentication to a Play! 1 Application

Filed under:Hacking,Play! Framework,Tooling — Philipp @ 1:40 pm

You will often find yourself in a situation where you need a public and a private (normally some Administration) Area in your application.

This Post is about how easy you can access User Data stored in you companies LDAP Directory with a public availible Library in a Play! 1 Application. I will also give you an example for Play! 2 in one of the upcoming Posts here.

Prerequisites

For this example we are using the Vagrant vagrant-rundeck-ldap VM, I already mentioned here.

Setup

After you setup a basic Play! 1 Application with

play new ldap-test

We need to add the necessary Dependencies to our Application. Since the Library is availible over the public Maven Repository the Changes in your conf/dependecies.yml are quite simple:


# Application dependencies
require:
    - play
    - com.innoq.liqid -> ldap-connector 1.3

You then need to update your Applications Dependencies with:

play deps --sync

We also need to add some Configration Settings to the conf/application.conf File:
(there are some more, since the Example Directory did not match to the Libraries default LDAP Layout)


...
# Settings for LiQID
# ~~~~~
# Basic LDAP objectClasses for Users and Groups
ldap.user.objectClasses=person
ldap.group.objectClasses=groupOfUniqueNames

# You can add several LDAP for Writing Changes, but you need one default LDAP for Access without a specifiy Instance String (e.g. ldap1, ldap2, etc).
default.ldap=ldap1
ldap.listing=ldap1

# Instance specific settings (e.g. credentials, layout, etc. )

ldap1.ou_people=ou=users
ldap1.ou_group=ou=roles
ldap1.url=ldap://localhost:3890
ldap1.principal=dc=Manager,dc=example,dc=com
ldap1.credentials=password

ldap1.base_dn=dc=example,dc=com
ldap1.admin.group.id=admin

ldap1.user.id.attribute=cn
ldap1.user.object.class=person

ldap1.group.id.attribute=cn
ldap1.group.object.class=groupOfUniqueNames
ldap1.group.member.attribute=uniqueMember

Implementation

We will use two different techniques:

  • Controller inheritance
  • Before Filters

The goal is, that for a specific Count of Controllers, the Request should be checked for a valid Autorization Header.
So all Controllers that need to be “protected” will inherit from a Controller named SecureApplication.
This SecureApplication Controller will implement a Before Filter and some Utility Methods to perform all necessary Login Verification.

This is the basic Implementation of that Controller:


package controllers;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.liqid.utils.Configuration;

import play.Logger;
import play.Play;
import play.mvc.Before;
import play.mvc.Controller;

public class SecureApplication extends Controller {
	private final static LdapHelper HELPER = getLdapHelper();
	@Before
	public static void checkLogin() {
		if (request.user == null || request.password == null
				|| !HELPER.checkCredentials(request.user, request.password)) {
			unauthorized("You need to Login first!");
		}
	}
	private static LdapHelper getLdapHelper() {
		Configuration.setPropertiesLocation(Play.applicationPath
				+ "/conf/application.conf");
		return LdapHelper.getInstance();
	}	
}

In the getLdapHelper() Methods you also see, that the location of the Configuration is changed to ../conf/application.conf.
Otherwise the Default Location would be in ~/.liqid/ldap.properties.

The Before Action checkLogin is triggered on every Request to a Controller that inherits from SecureApplication.
All other Controllers (inheriting the normal Controller Class), won’t trigger that Application.

You also can access all Data stored in the LDAP Directory. So if you want to access Attributes like cn, dn and sn, you need to add some more Code:
First we will create a basic POJO that contains all User data:


package models;

public class User {
	public String cn;
	public String sn;
	public String dn;

	public User(String cn, String sn, String dn) {
		this.cn = cn;
		this.sn = sn;
		this.dn = dn;
	}

	public String toString() {
		StringBuilder sb = new StringBuilder();
		sb.append("cn: ").append(cn).append("\n");
		sb.append("sn: ").append(sn).append("\n");
		sb.append("dn: ").append(dn).append("\n");
		return sb.toString();
	}
}

We then will update our SecureApplication Class:


package controllers;

import models.User;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.ldap.connector.LdapNode;
import com.innoq.liqid.model.Node;
import com.innoq.liqid.utils.Configuration;

import play.Logger;
import play.Play;
import play.mvc.Before;
import play.mvc.Controller;

public class SecureApplication extends Controller {
	private final static LdapHelper HELPER = getLdapHelper();
	protected static User ActiveUser;
	@Before
	public static void checkLogin() {
		if (request.user == null || request.password == null
				|| !HELPER.checkCredentials(request.user, request.password)) {
			unauthorized("You need to Login first!");
		} else {
			ActiveUser = getUser(request.user);
		}
	}

	private static LdapHelper getLdapHelper() {
		Configuration.setPropertiesLocation(Play.applicationPath
				+ "/conf/application.conf");
		return LdapHelper.getInstance();
	}

	private static User getUser(String cn){
		LdapNode ldapNode = (LdapNode) HELPER.getUser(cn);
		return new User(cn, ldapNode.get("sn"), ldapNode.getDn());
	}
}

As you can see, there is a specific NodeType LdapNode that has some Convenience Methods for accessing the LDAP Data.
As Example you can invoke the toString() Method of an User Object in your template.

So first we will have to create our Admin Controller:


package controllers;

import models.User;

public class Admin extends SecureApplication {
	public static void index(){
		User u = ActiveUser;
		render(u);
	}
}

You have to use another Object u for that User, since Play! 1 has a specific Behavior of how to map Variables in Templates.
Our Template app/views/Admin/index.html might look like this:


#{extends 'main.html' /}
#{set title:'Admin Area' /}
Hello Admin!
<pre>
${u}
</pre>

If you login then with the Credendials build/build your output might look then like this:

 Index | Administration
Hello Admin!

cn: build
sn: The account to use to demonstrate managing builds only
dn: cn=build,ou=users,dc=example,dc=com

Links

You can find the Sources of that Library here: https://github.com/innoq/LiQID
You can find an example Project here: https://github.com/phaus/play-ldap/tree/master/play1-ldap-example

Feel free to add issues about bugs or missing features to this :-).

2013/07/27  IMHO: Differences between a mobile Web-Page and a mobile Web-Application

Filed under:Uncategorized — Philipp @ 2:46 pm

About

Since i did some Web-Development for Mobile Device for my last projects, i would like to cover some of my thoughts and expierences i had during these months.

To come straight to the point, it is not really easy to develop stuff for mobile Devices these days, exspecially if you want to support older ones like Android <=2.3, since there are a lot of vendor specific hoops you have to jump through (and althought a lot of testing).

But i would like to start from the other site: the users site (the so called user experience).

 

mobile Web-Page

For a User, a mobile Web-Page is just a different view to a common know Homepage.
He can use saved or received links to access a specific resource.
He can rapidly change the Page Location while changing the URL in the address bar.
The Page opens in the Mobile Web-Browser (or an Browser like environment like WebView, hybrid App, etc.)
The Page need online Access to work.
The Page needs some time to load, although the content has been adjusted to mobile use (responsiveness, minifying, image optimazation, aggrssive Caching, CND, etc.)
Every Action will use a Request and will go through the Web-Page and will receive a Response for Displaying.
The User can add a Bookmark to Homescreen/Browser Favorites to have fast and easy access to that URL.

To summarize all this, a mobile Web-Page is the result of a web-optimized Web-Development Project, so e.g. you start with your common (Desktop-) Web-Flows using mobile agnostic CSS (if you must bootstrap or foundation) and modify Navigation, Javascript and Content-size to work with the mobile devices display.

In most cases you will find something like this on your screen:

Ohne Titel.002

You have a fitting View, on top your URL (that might be scrolled up via JS) and on the bottom your Browsers controls (that might be hidden by the Browsers default – e.g. Landscape view on iOS).

 

mobile Web-Application

A mobile Web-Application is specially developed for a mobile Device. You won’t start with already existing views or other assets. Do a App-Like Design (e.g. Views instead of Pages).
You should think about different Behaviour for this one (like more Icons instead of Full-Tested Links), hidden Menus that appear via slide. To be clear: use gestures and icons, your device has a touch sensor, so use it.
Use the availble Devices Settings to adjust your Applications appearance.
The User has to add the Application to her/his Homescreen to enter “App-Mode” (although the first Start uses the mobile Browser).
The App starts alway Full-Screen, so no URL is present at all, so the URL and Links are less relevant, although the Application my use a specific Model Routing like Backbone.js.
The App uses local storage (up to 10 megs) – if it has to store Data – first, then uses the Servers persistence logic.
The App should use the availible Device specific JavaScript Methods
An Action does not trigger a request automatically back to the Server (Backend).
A common use-case might be: Load Data from Backend, do something with the Data, save it locally, user other local Data, save the Data, close the App, …, restart the App, continue working, …, send the Data back to the Backend.
The App should work Offline (you have to implement some kind for update Logic – e.g. using a manifest file).
You should use a less Request/Response Cycles as possible to save Battery Time and Bandwidth.
You should use optimzied Resources (e.g. pack the structe of a survey in on JSON Feed instead of using Links to appending resources – with usefull exeption like sounds or images).
Consider using alternative Communications ways (like Websockets).

Ohne Titel.003

You will find a common Menu with Icons at the Bottom of the Page. Some Apps also using hidden Menus (that appear via swipe or a small Button).
You also will need some Kind of Activity Indicator (like a progress bar), if you need to fetch Data from or send Data to the Backend (or doing other operations that might use some time).

 

Disclaimer :-)

That’s for the moment. It really needs some more clearifications and examples. Of course you can argue a lot about some statements here. This i just my personal opinion, first of all to differentiate between the two things.

Most of the Links go to iOS Resources, since i had to do Web Devlopment mostly for iOS and since iOS was first planed as a Web-Application only Platform, it has a pretty good documenation of all availible features. But i am sure, there are similar Documentations for (at least the common) Android and Windows Phone Devices.

Older Posts »