Plotting UNIX Processes with DOT

Inspired by this Post, i started playing around with ps, nodejs and GraphViz.

After reading some ps man Pages, i found the necessary ps parameters.
For MacOS i used

ps -A -c -o pid,ppid,pcpu,comm,uid -r

For Linux i used

ps -A -o pid,ppid,pcpu,comm,uid

You then get some Output like:

    PID    PPID %CPU COMMAND           UID
      1       0  0.0 init                0
      2       1  0.0 kthreadd            0
      3       2  0.0 migration/0         0
      4       2  0.0 ksoftirqd/0         0
      5       2  0.0 migration/0         0
      6       2  0.0 watchdog/0          0

So you are getting the ProcessID, the Parent ProcessID, CPU Usage (i am not using for plotting atm), the Command and the UserID.
I created a simple Node Script, that you can run either directly under MacOS (for all other Unices you need to update the ps command).
Or you can give the script a previous generated ps output for parsing:

plotPS.sh /tmp/host.log > /tmp/host.dot

The resulting DOT Code is then Piped into a DOT File.

Here are some examples:

My MacOS Laptop:

MacBook Pro
bigger

A Sinlge Linux Host with Dovecot and Apache2/Passenger:

Apache2 / Mail Server
bigger

A Linux Host with OpenVZ and KVM Instances:

OpenVZ / KVM Host
bigger

In the original Post, there were also Dependencies between CPU Usage and Size of the Graphical Nodes, also it would be more useful to only plotting the processes of one VM from its inside.
But i guess for one evening the result is okay :-).

Build and Test Project TOX under MacOS

Some Steps to do

  1. You need to have XCode with installed CLI Tools (see here)
  2. If you are using MacPorts (you really should), you need to install all necessary Dependencies:
    port install libtool automake autoconf libconfig-hr libsodium cmake
  3. Checkout the Project TOX Core Repository:
    git clone --recursive https://github.com/irungentoo/ProjectTox-Core.git
  4. cd ProjectTox-Core
    cmake .
    make all
  5. You need two tools:
    DHT_bootstrap in /other
    and nTox in /testing
  6. Bootstrap Tox (aka get your Public Key):
    ./DHT_bootstrap
    Keys saved successfully
    Public key: EA7D7BD2566A208F83F81F8876DE6C1BDC1F8CA1788300296E5D4F4CB142CD77
    Port: 33445

    The key is also in PUBLIC_ID.txt in the same Directory.

  7. Run nTox like so:
    ./ntox 198.46.136.167 33445 728925473812C7AAC482BE7250BCCAD0B8CB9F737BF3D42ABD34459C1768F854

    Where:

    Some Tox Node
    198.46.136.167
    Port of that TOX Node
    33445
    Public Key of that TOX Node
    728925473812C7AAC482BE7250BCCAD0B8CB9F737BF3D42ABD34459C1768F854
  8. Et voilà:
    /h for list of commands
    [i] ID: C759C4FC6511CEED3EC846C0921229CA909F37CAA2DCB1D8B31479C5838DF94C
    >>

    You can add a friend:

    /f ##PUBLIC_ID##

    List your friends:

    /l

    Message a friend:

    /m ##friend_list_index##  ##message##

Downgrading Subversion from 1.8 to 1.7 in MacPorts

bash-3.2# cd /tmp
bash-3.2# svn co http://svn.macports.org/repository/macports/trunk/dports/devel/subversion --revision 108493
A    subversion/files
A    subversion/files/patch-Makefile.in.diff
A    subversion/files/patch-osx_unicode_precomp.diff
A    subversion/files/config_impl.h.patch
A    subversion/files/servers.default
A    subversion/Portfile
Ausgecheckt, Revision 108493.
bash-3.2# cd subversion/
bash-3.2# port install
--->  Computing dependencies for subversion
--->  Fetching archive for subversion
--->  Attempting to fetch subversion-1.7.10_1.darwin_12.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/subversion
--->  Attempting to fetch subversion-1.7.10_1.darwin_12.x86_64.tbz2.rmd160 from http://mse.uk.packages.macports.org/sites/packages.macports.org/subversion
...
--->  Scanning binaries for linking errors: 100.0%
--->  No broken files found.
bash-3.2# port installed subversion
The following ports are currently installed:
  subversion @1.7.10_1
  subversion @1.8.1_1 (active)
bash-3.2# port activate subversion @1.7.10_1
--->  Computing dependencies for subversion
--->  Deactivating subversion @1.8.1_1
--->  Cleaning subversion
--->  Activating subversion @1.7.10_1
--->  Cleaning subversion

Adding LDAP Authentication to a Play! 2 Application

As of Play! 1 is not really supported anymore, i will describe the steps for accessing Data from your LDAP Directory with your Play! 2 with this Post.

Prerequisites

As also mentioned in my last Post, for this example we are using the Vagrant vagrant-rundeck-ldap VM, I already mentioned here.

Setup

After you setup a basic Play! 2 Application with

play new ldap-test

We need to update our Applications Dependencies. To achieve this, we need to change project/Build.scala:


import sbt._
import Keys._
import play.Project._

object ApplicationBuild extends Build {

  val appName         = "play2-ldap-example"
  val appVersion      = "1.0-SNAPSHOT"

  val appDependencies = Seq(
    // Add your project dependencies here,
    javaCore,
    javaJdbc,
    javaEbean,
    "com.innoq.liqid" % "ldap-connector" % "1.3"
  )

  val main = play.Project(appName, appVersion, appDependencies).settings(
    // Add your own project settings here      
  )
}

The Command:


play compile

Should download all necessary Project Depedencies and will do an initial Compilation of all your Project Files (currently not really many).

You also need to add the LDAP Settings to your Applications configuration. For this example we will use an external Properties File conf/ldap.properties:


# Settings for LiQID
# ~~~~~
ldap.user.objectClasses=person
ldap.group.objectClasses=groupOfUniqueNames

default.ldap=ldap1
# LDAP Listsing, divided by ","
ldap.listing=ldap1

ldap1.ou_people=ou=users
ldap1.ou_group=ou=roles
ldap1.url=ldap://localhost:3890
ldap1.principal=dc=Manager,dc=example,dc=com
ldap1.credentials=password

ldap1.base_dn=dc=example,dc=com
ldap1.admin.group.id=admin

ldap1.user.id.attribute=cn
ldap1.user.object.class=person

ldap1.group.id.attribute=cn
ldap1.group.object.class=groupOfUniqueNames
ldap1.group.member.attribute=uniqueMember

Implementation

Since Play1 2 does not really support the use of Before Filters, we will use Custom Actions for making our Login Authentication work.
We will create a new Package app/actions with two new files: an annotation interface BasicAuth and the implementation itself BasicAuthAction.
Annotations will be used to Set a specific Controller to use Basic Auth.
So lets start with BasicAuth:


package actions;

import play.mvc.With;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@With(BasicAuthAction.class)
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD, ElementType.TYPE })
@Inherited
@Documented
public @interface BasicAuth {
}

After that you can annotate Controllers with @BasicAuth (but this won’t work, since the Implementation is still missing).
Here then the BasicAuthAction:



package actions;

import com.ning.http.util.Base64;

import models.User;
import play.mvc.Action;
import play.mvc.Http.Context;
import play.mvc.Result;

public class BasicAuthAction extends Action {

	private static final String AUTHORIZATION = "authorization";
	private static final String WWW_AUTHENTICATE = "WWW-Authenticate";
	private static final String REALM = "Basic realm=\"play2-ldap-example\"";

	@Override
	public Result call(Context context) throws Throwable {

		String authHeader = context.request().getHeader(AUTHORIZATION);
		if (authHeader == null) {
			return sendAuthRequest(context);
		}

		String auth = authHeader.substring(6);

		byte[] decodedAuth = Base64.decode(auth);
		String[] credString = new String(decodedAuth, "UTF-8").split(":");

		if (credString == null || credString.length != 2) {
			return sendAuthRequest(context);
		}

		String username = credString[0];
		String password = credString[1];
		User authUser = User.authenticate(username, password);
		if (authUser == null) {
			return sendAuthRequest(context);
		}
		context.request().setUsername(username);
		return delegate.call(context);
	}

	private Result sendAuthRequest(Context context) {
		context.response().setHeader(WWW_AUTHENTICATE, REALM);
		return unauthorized();
	}
}

As you can see, there are no LDAP specific Dependencies at all, since all necessary Logic is the User Model, in User.authenticate(username, password).So let’s have a look into that Model:


package models;

import play.Play;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.ldap.connector.LdapUser;
import com.innoq.liqid.utils.Configuration;

public class User {

	private static LdapHelper HELPER = getHelper();

	public String sn;
	public String cn;
	public String dn;

	public User(String cn) {
		this.cn = cn;
	}

	public String toString() {
		StringBuilder sb = new StringBuilder();
		sb.append("cn: ").append(cn).append("\n");
		sb.append("sn: ").append(sn).append("\n");
		sb.append("dn: ").append(dn).append("\n");
		return sb.toString();
	}

	public static User authenticate(String username, String password) {
		if (HELPER.checkCredentials(username, password)) {
			return new User(username);
		}
		return null;
	}

	public static User getUser(String username) {
		LdapUser ldapUser = (LdapUser) LdapHelper.getInstance().getUser(
				username);
		User user = new User(username);
		user.cn = ldapUser.get("cn");
		user.sn = ldapUser.get("sn");
		user.dn = ldapUser.getDn();
		return user;
	}

	private static LdapHelper getHelper() {
		Configuration.setPropertiesLocation(Play.application().path()
				.getAbsolutePath()
				+ "/conf/ldap.properties");
		return LdapHelper.getInstance();
	}
}

You also have a static Instace of that LDAP Helper, but authenticate User Credentials and Login are in two different Methods.Last thing is to Load a User from the LDAP Directory:Here the Admin Controller:


package controllers;

import models.User;
import actions.BasicAuth;
import play.mvc.Controller;
import play.mvc.Result;
import views.html.Admin.index;

@BasicAuth
public class Admin extends Controller {
    public static Result index() {
    	User u = User.getUser(request().username());
        return ok(index.render("Hello Admin!", u));
    }
}

And here the used Template File:


@(message: String, user: User)

@main("Admin Index") {
@{message} 
<p>   
<pre>
@{user}
</pre>
</p>
}

Links

You can find the Sources of that Library here: https://github.com/innoq/LiQIDYou can find an example Project here: https://github.com/phaus/play-ldap/tree/master/play2-ldap-example

Adding LDAP Authentication to a Play! 1 Application

You will often find yourself in a situation where you need a public and a private (normally some Administration) Area in your application.

This Post is about how easy you can access User Data stored in you companies LDAP Directory with a public availible Library in a Play! 1 Application. I will also give you an example for Play! 2 in one of the upcoming Posts here.

Prerequisites

For this example we are using the Vagrant vagrant-rundeck-ldap VM, I already mentioned here.

Setup

After you setup a basic Play! 1 Application with

play new ldap-test

We need to add the necessary Dependencies to our Application. Since the Library is availible over the public Maven Repository the Changes in your conf/dependecies.yml are quite simple:


# Application dependencies
require:
    - play
    - com.innoq.liqid -> ldap-connector 1.3

You then need to update your Applications Dependencies with:

play deps --sync

We also need to add some Configration Settings to the conf/application.conf File:
(there are some more, since the Example Directory did not match to the Libraries default LDAP Layout)


...
# Settings for LiQID
# ~~~~~
# Basic LDAP objectClasses for Users and Groups
ldap.user.objectClasses=person
ldap.group.objectClasses=groupOfUniqueNames

# You can add several LDAP for Writing Changes, but you need one default LDAP for Access without a specifiy Instance String (e.g. ldap1, ldap2, etc).
default.ldap=ldap1
ldap.listing=ldap1

# Instance specific settings (e.g. credentials, layout, etc. )

ldap1.ou_people=ou=users
ldap1.ou_group=ou=roles
ldap1.url=ldap://localhost:3890
ldap1.principal=dc=Manager,dc=example,dc=com
ldap1.credentials=password

ldap1.base_dn=dc=example,dc=com
ldap1.admin.group.id=admin

ldap1.user.id.attribute=cn
ldap1.user.object.class=person

ldap1.group.id.attribute=cn
ldap1.group.object.class=groupOfUniqueNames
ldap1.group.member.attribute=uniqueMember

Implementation

We will use two different techniques:

  • Controller inheritance
  • Before Filters

The goal is, that for a specific Count of Controllers, the Request should be checked for a valid Autorization Header.
So all Controllers that need to be “protected” will inherit from a Controller named SecureApplication.
This SecureApplication Controller will implement a Before Filter and some Utility Methods to perform all necessary Login Verification.

This is the basic Implementation of that Controller:


package controllers;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.liqid.utils.Configuration;

import play.Logger;
import play.Play;
import play.mvc.Before;
import play.mvc.Controller;

public class SecureApplication extends Controller {
	private final static LdapHelper HELPER = getLdapHelper();
	@Before
	public static void checkLogin() {
		if (request.user == null || request.password == null
				|| !HELPER.checkCredentials(request.user, request.password)) {
			unauthorized("You need to Login first!");
		}
	}
	private static LdapHelper getLdapHelper() {
		Configuration.setPropertiesLocation(Play.applicationPath
				+ "/conf/application.conf");
		return LdapHelper.getInstance();
	}	
}

In the getLdapHelper() Methods you also see, that the location of the Configuration is changed to ../conf/application.conf.
Otherwise the Default Location would be in ~/.liqid/ldap.properties.

The Before Action checkLogin is triggered on every Request to a Controller that inherits from SecureApplication.
All other Controllers (inheriting the normal Controller Class), won’t trigger that Application.

You also can access all Data stored in the LDAP Directory. So if you want to access Attributes like cn, dn and sn, you need to add some more Code:
First we will create a basic POJO that contains all User data:


package models;

public class User {
	public String cn;
	public String sn;
	public String dn;

	public User(String cn, String sn, String dn) {
		this.cn = cn;
		this.sn = sn;
		this.dn = dn;
	}

	public String toString() {
		StringBuilder sb = new StringBuilder();
		sb.append("cn: ").append(cn).append("\n");
		sb.append("sn: ").append(sn).append("\n");
		sb.append("dn: ").append(dn).append("\n");
		return sb.toString();
	}
}

We then will update our SecureApplication Class:


package controllers;

import models.User;

import com.innoq.ldap.connector.LdapHelper;
import com.innoq.ldap.connector.LdapNode;
import com.innoq.liqid.model.Node;
import com.innoq.liqid.utils.Configuration;

import play.Logger;
import play.Play;
import play.mvc.Before;
import play.mvc.Controller;

public class SecureApplication extends Controller {
	private final static LdapHelper HELPER = getLdapHelper();
	protected static User ActiveUser;
	@Before
	public static void checkLogin() {
		if (request.user == null || request.password == null
				|| !HELPER.checkCredentials(request.user, request.password)) {
			unauthorized("You need to Login first!");
		} else {
			ActiveUser = getUser(request.user);
		}
	}

	private static LdapHelper getLdapHelper() {
		Configuration.setPropertiesLocation(Play.applicationPath
				+ "/conf/application.conf");
		return LdapHelper.getInstance();
	}

	private static User getUser(String cn){
		LdapNode ldapNode = (LdapNode) HELPER.getUser(cn);
		return new User(cn, ldapNode.get("sn"), ldapNode.getDn());
	}
}

As you can see, there is a specific NodeType LdapNode that has some Convenience Methods for accessing the LDAP Data.
As Example you can invoke the toString() Method of an User Object in your template.

So first we will have to create our Admin Controller:


package controllers;

import models.User;

public class Admin extends SecureApplication {
	public static void index(){
		User u = ActiveUser;
		render(u);
	}
}

You have to use another Object u for that User, since Play! 1 has a specific Behavior of how to map Variables in Templates.
Our Template app/views/Admin/index.html might look like this:


#{extends 'main.html' /}
#{set title:'Admin Area' /}
Hello Admin!
<pre>
${u}
</pre>

If you login then with the Credendials build/build your output might look then like this:

 Index | Administration
Hello Admin!

cn: build
sn: The account to use to demonstrate managing builds only
dn: cn=build,ou=users,dc=example,dc=com

Links

You can find the Sources of that Library here: https://github.com/innoq/LiQID
You can find an example Project here: https://github.com/phaus/play-ldap/tree/master/play1-ldap-example

Feel free to add issues about bugs or missing features to this :-).

Nice Vagrant Script for setting up a LDAP VM for Testing

I found a nice Setup Script at https://github.com/gschueler/vagrant-rundeck-ldap for creating a working LDAP VM with a basic

vagrant up

You just need to do some more steps before:

  • make sure you have installed Version 4.2.12 of Virtualbox! (the latest version – 4.2.14 – breaks vagrant)
  • get your vagrant setup from here
  • download the necessary ubuntu vagrant box:
    vagrant box add precise32 http://files.vagrantup.com/precise32.box

The LDAP port will be mapped to port 3890 in your local machine (the host where virtualbox is installed)
You can login (e.g. with Apache Directory Studio) with the following settings:

Encryption: none
Host: localhost
Port: 3890
User: dc=Manager,dc=example,dc=com
Pass: password

play-i18ned

This Module provides support for converting Play! i18n Files into an Excel Sheet and from an Excel Sheet to i18n Files.

Usage

You need to create the message files first (e.g. conf/messages, conf/messages.de, conf/messages.en)
You may enter some key/value entries to the Files.

The prefered format is:

# Description key=value 

Add the Module to your dependencies:


require: 
... - local -> i18ned 0.1 
... repositories: 
    - local: 
          type: local
          artifact: ${application.path}/../modules/[module]/dist/[module]-[revision].zip 
     contains: - local -> * 


Then install the Module

play deps --sync 

Export (creating Excel File)

play i18ned:export 

Your will find the Excel File in APP_DIR/tmp/i18n.xls

Import (Moving Changes from the Excel File into the Messages Files)

play i18ned:import 

TODO

  • Adding SQL-Export with SQL-Templates
  • Adding Support for selective Actions (Im/Export only specific files)
  • Import from other i18n Formats

Link

https://github.com/phaus/play-i18ned

Starting work on a new ZFS Web Admin

Hello there.

After the Stats showed me, that the ancient Post about a Web based ZFS Interfaces produces a lot attention, i decided to restart with that project from scratch.

You can find the current sources here. I am currently working with the FreeBSD ZFS Version (so basically FreeNAS), but there are already some Test-Cases for Nexenta.
The basic idea of the system is, to use remote SSH Sessions for accessing the different systems and parse the receiving data. There is already a quite stable, JNA based Java Binding for ZFS, but it seems to be broken for the latest Versions of ZFS. And to have a simple solution to manage ZFS on different system, seems to me a great idea.

I have some ideas how to do some things, but it still seems a lot of work :-).