Techmania

February 12, 2014

Operational Business Intelligence for the Reactive Enterprise

We are in a day and age where infrastructure, and to some extent, businesses, are moving towards reactive IT instead of traditional proactive IT. Why reactive IT? Think auto-scaling vs traditional capacity planning for instance. Or the move towards schema-less NoSQL instead of traditional RDBMS. All in all, reactive IT enables an enterprise and its infrastructure to react to its internal and external environment. In order to achieve this, a real-time, or near real time, event driven model is a must –  agents to pull/push streams of events, systems to process these events quickly and efficiently and the ability to use these processed events to react.

Operational Business Intelligence (knowns as OBI or operational BI) or Operational Intelligence is defined as the analysis of operational data and information in an enterprise – this deals with the real time or low latency analysis of streaming events or batched enterprise data providing feedback in terms of input to the enterprise. OI provides organisations with real time insights into enterprise operations, providing organisations the ability to act upon, or in other words react, to events – enabling organisations to ‘listen’ to and process events as they come in, detect anomalies and patterns, and take reactive measures.

Of course, with large amounts of data come big challenges, and with big challenges come big data. With the possibility of dealing with massive data sets within OI, big data concepts have started becoming a mainstay in organisational OI. Modern OI solutions have started focusing on large NoSQL data stores possibly processed in batch mode and streaming events.

OI would provide a unified, correlated view of streaming big data, processed big data, complex events and processes, with the ability to analyse, mine and process data and information – a prerequisite to building a reactive enterprise. Enterprise users and devops have come to expect the following kind of information from a unified OI view

  • Analysis of information, and mining for patterns
  • Adhoc, real time dashboards
  • Adhoc search of patterns across the enterprise
  • Alerting based on event occurences
  • Monitoring of system health, load, KPIs

Image

OI has many technology components, often with shared feature sets. Some of the notable solutions are

  • Business Activity Monitoring (BAM) – Monitoring of activities and events, usually batch processed and provided via dashboards. Used to track  KPIs related to activities and performance.
  • Complex Event Processing (CEP) – Processing of a continuous stream of events, usually in-memory. High performance with the ability to detect certain patterns and anomalies in the incoming streams.
  • Business Process Management (BPM) – Model driven execution of processes and policies, including business workflows with human intervention.

Independently the above components handle a very specific area of OI, and together they provide the building blocks for a comprehensive and unified OI.

In a later post, we will explore how the WSO2 stack, a comprehensive Open Source middleware stack, can be utilised for Operational Intelligence in an enterprise.

May 28, 2013

SSH must-haves for Productivity

Filed under: Development,FOSS,Linux,Uncategorized — mifan @ 9:21 am

Devops work with many different remote systems at once. For the lazy bunch ( a.k.a the truly productive bunch who look for ways to make their work easy – a.k.a the really bright ones ) there are ways to make life easier.

So instead of typing

ssh whitewizard@163.12.24.21

(or wait, was it 163.12.24.23??), and then entering the password that no one can remember given to us by the sysadmin, I’d now say

ssh devserver

and voila.

The steps to get there are:

SSH Passwordless Entry

Step one is to get rid of those pesky passwords required each time you SSH into a host. This can be achieved by setting up an RSA (public/private key). And besides, who’d say no to added security?

To generate a SSH key if you don’t have one already, run

mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa

You will be prompted for a location to save the keys, and a passphrase for the keys. This passphrase will protect your private key while it’s stored on the system.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/b/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/b/.ssh/id_rsa.
Your public key has been saved in /home/b/.ssh/id_rsa.pub.

Your public key is now available as .ssh/id_rsa.pub in your home folder.

Next, the little bit of magic where you’d copy your key over to the remote/host machine. If you can SSH into the machine, then this should be no problem. Run

ssh-copy-id whitewizard@163.12.24.21

Done. You can now login by saying

ssh whitewizard@163.12.24.21

And the system shouldn't ask you for a password.

SSH Hostname Shortcuts

But wait – we want more (or less, actually). The next step is to shorten this further. For this create a file .ssh/config with the following entries:

Host [shortcut]     
 Hostname [full-hostname]    
 User [username]

I’d use

Host devserver
     Hostname 163.12.24.21
     User whitewizard

And voila, now running the following will log you into the system

SSH devserver

Of course, replace the above hostnames and usernames with you own – else you’d be login into Saruman’s own server!

May 2, 2013

Real time Complex Analytics for Football

Filed under: Development,WSO2 — mifan @ 6:54 pm
Tags: , ,

Srinath recently wrote about the use of WSO2 CEP for the ACM DEBS 2013 challenge on Real time Complex Analytics which was a real eye opener. The challenge, enabled by sensor networks on player’s shoes, the goalkeeper and the football, was to conduct real time analytics on a football game, and provide useful analytics and real-time reports to the managers.

The post describes the usage of WSO2’s Complex Event Processor (CEP) which was used to implement the use cases of the challenge – namely Running analysis of the players, Ball possession analysis, a heatmap of player locations at various times and shots on goal analysis. CEP is a high performance and scalable event processor that can read streams of ‘data’, extract meaningful events and process them real time in memory – this means the ability to process large amounts of events, fast! In this case, continuous steams of data and events would be fed into the system via the various sensors, at a rate of 15,000 position events per second, whilst the player sensors and the ball sensors output events at a rate of 200MHz and 2000MHz respectively. According the blog, WSO2 CEP processed 50,000 events per second, which is quite impressive.

Just imagine the possibilities such an implementation can provide to the game – I’m awaiting the day when the TV alerts me, possibly 2 seconds before the event, that Robin Van Persie’s shot would have a 99% chance of finding the back of the net beating Petr Cech – based on analytics of the kick (the curvature , wind speed, rotation) and analytics of the defense (the distance of the goalkeeper and probability of him reaching the ball based on historical data, the distance between defenders). Or based on Theo Walcott’s speedy run and Carzola’s immaculate pass, as well as the position of the defense and the keeper, Walcott would end up with the ball beyond the defense in an on-side goal scoring position in the next 3 seconds – the mini siren on the TV goes off, telling me to watch the screen for the next 5 seconds (of course, that is assuming my reaction times are good, let alone the goal keeper’s). And I can hear us hard-core fans of football saying – “who would take their eyes off a football game anyways?”, but what if!

Minority Report for football, anyone?

XKCD Future Comic

Future (Source: XKCD)

February 7, 2012

Creating Patches with SVN Diff

Filed under: Development,FOSS,Linux — mifan @ 12:33 am
Tags: , ,

I recently came across a situation where I had to generate a patch from SVN, where I was in a multi-developer environment, my code was committed in 2 batches, and there were changes from other developers in between. Thanks to the power of SVN, this is a breeze (of course, I’m not even going to start the version control debate here!!).
The format for SVN diff is as follows:

svn diff [-r N[:M]] [--old OLD-TGT] [--new NEW-TGT] [PATH...]

In this case, to generate a patch between revisions 1000 to 1020, only for files file1.php and file2.php would be as follows

svn diff -r 1000:1020 dir1/file1.php dir2/file2.php > mypatch.patch

For conventional patches, you would checkout the code, make your changes, and from the root locations run the svn diff, as

svn diff > mypatch.patch


October 5, 2011

Download Directories with wget

Filed under: Development,FOSS,Linux — mifan @ 6:53 am

wget is a non-interactive download utility available in the *nix OS, which can be used to remotely download files, amongst other things.

I’ve looked for the correct wget command to download a directory remotely, but couldn’t find the right one. This one, however, worked for me:

wget -r --level=3 -np -nH "<URL of Directory>"

where

r – recursive download (download recursively looping through sub directories

np – no parent directories (without this command, you get some weird results from the recursion working for the parent directory as well)

nH – no host (get rid of the host name – without this, your folder will be created with some weird name that includes the host/url

level=3 -upto what level/depth you want to recurse or go into. Change to an appropriate number

Check the man page of wget for more information

September 16, 2011

Multiple Terminals in the same Session with Screen

Filed under: Development,FOSS,Linux — mifan @ 8:30 am

As an administrator or unix user who has just SSH’d into a remote server, have you had the requirement to open multiple remote terminals without having to resort to re-logging in multiple times? I’ve had this one too many times, where I needed to run multiple apps, view logs etc. etc. in separate terminals, which can be made easy using GNU Screen.
GNU Screen is a screen multiplexer, which can be installed via a simple apt-get install screen on your linux server. This should be available by default as well. Once that is done, the following steps should suffice

  • Login to your remote terminal (e.g: via SSH)
  • Open Screen
 user@remote> screen
  • Use the following commands to create, navigate etc. CTRL+A (CTRL A) is the control key in my Ubuntu server by default, and this might be the case for you as well. Thus when I refer to CTRL A p, I mean holding CTRL + A down, press the ‘p’ key
  • Create a new terminal:
user@remote> CTRL A c
  • Move to the next terminal:
user@remote> CTRL A n
  • Move to the previous terminal:
user@remote> CTRL A p

Etc. Some more commands are as follows

  • Move to a specific terminal: CTRL A “
  • Move to the last used terminal: CTRL A A
  • Move to terminal number 0 (0-9) : CTRL A 0

These commands were enough to get me going. For even further commands and to explore the power of screen in depth, here are some useful resources:

Resource 1, Resource 2

July 2, 2008

Debugging PHP in Eclipse using XDebug

Filed under: Development,FOSS,Linux — mifan @ 12:44 pm

My previous post touched upon the basics of getting Eclipse PDT (PHP Development Tools) up and running for PHP development. In this post, we’ll be looking at getting PHP debugging capabilities into our PHP dev environment. Again, I’ll be focusing on Ubuntu Linux – 8.04 LTS to be specific: however, with minor changes, this can be used on any platform.

Of course, there are simpler methods of debugging PHP scripts: a var_dump is quite handy in terms of simple debugging – however, having the right tools at hand for development means that we can spend more time in actual development than writing debug statements every once in a while – anyways, that’s my take.

XDebug is a remote debugger. Here we’ll take a look at getting XDebug working with Eclipse PDT providing a full debugging client. The first step would of course be to get Eclipse PDT up and running (refer previous post if needed). Once done, the next step would be to get Xdebug for our PHP environment set up – in my case, PHP5. On Ubuntu, this can be achieved by:
apt-get install php5-xdebug
This should install Xdebug for PHP5. Next, we’d configure Xdebug by editing the PHP configuration file. Most systems require the configuration items be added to the main php.ini, or the appropriate conf file. In Ubuntu 8.04 LTS, a separate configuration file,xdebug.ini, is created for Xdebug. Open this file using your favorite text editor (vim in my case):
sudo vim /etc/php5/conf.d/xdebug.ini
Add the following lines after the zend_extension= or zend_extension_ts= conf value.

xdebug.remote_enable=On
xdebug.remote_host="localhost"
xdebug.remote_port=9000
xdebug.remote_handler="dbgp"

And restart your web-server. In my case, Apache:
/etc/init.d/apache2 restart
To try out the debugger, open a PHP project in Eclipse (if you are importing an existing project via CVS, remember to add it as a PHP project using the New Project Wizard). In Window->Preferences->General->Web Browser, select external web browser, and set it to point to your default web browser. Next go into the debug section, by Window->Preferences->PHP->Debug, and set the PHP Debugger option to XDebug (by default this is set to the Zend Debugger). Note the chekbox “Break at first line” in this page as well – you can change this later on, as required. After you’ve applied your changes, go into the Debug menu, by Run->Open Debug Dialog… Here, make sure the Server Debugger is set to XDebug, and that the PHP server points to http://localhost or an appropriate server location. The file section should point to the file to be debugged, relative to your workspace (Note: if your project is missing from the list, check whether you’re project is configured as a PHP project – I’ve noticed that non-PHP projects fail to appear on this list). Thus if your file is index.php, the relative path maybe something/www/index.php etc.
Next, check the URL section to see whether it points to the correct location – if not, make sure you uncheck the “Auto-Generate” check box, and enter in the correct location. Apply the changes and close the dialog box.
A view of the Debug dialog is here:
EclipsePDT Debug Dialog
You should now be ready to debug your application. Click on Run->Debug or F11 to enable debugging – Eclipse should switch to the Debug View, and the page should start loading in your external browser. If “Break at First Line” checkbox was checked previously, then the program would have “stopped” at the first line – you can step into or step over subsequent lines using the controls on your top left corner. Alternatively, you can place breakpoints at various locations and stop code execution there as well. The variables tab should give you the variables and their values at various times. I good guide from which I borrowed from is here. Get creative – start debugging.
A view of the debugger in operation is here:
EclipsePDT XDebug for PHP

PHP development with Eclipse PDT

Filed under: Development,Linux — mifan @ 9:48 am

In this post, I’ll be outlining the steps to get a fully functional PHP IDE setup. I’ll be using the Ubuntu Linux way of installing stuff, but you can replace it with any repository or online location as needed. First things first, the ingredients: Sun Java 6, Eclipse Europa 3.3, Eclipse PDT. I’m using the latest version of Ubuntu at the time of writing: Ubuntu 8.04 LTS Hardy Heron.

Get the JDK: in Ubuntu, the latest version can be obtained by
apt-get install sun-java6-jdk
which installs the Sun version from the Sun Java website. If you already have GCJ (the GNU JDK), better replace it with this. Installing Java from the Ubuntu repository instead of downloading it manually from Sun means that you don’t have to muck about creating symlinks to Java and the likes – everything’s done for you. Next, install Eclipse. Eclipse is available as a Ubuntu package, but since Hardy Heron contains version 3.2 (not 3.3, which is required for the latest version of PDT) of Eclipse, lets resort to a manual install. What’s more, Eclipse Europa 3.3 bundled with the PHP Development Tools (PDT) can be obtained from the PDT site, so this is what we’ll do. Visit the Eclipse PDT download page. Select the download version you need (1.0.3 in my case), which will take you to the next page. Select the PDT All-in-One package for your platform: Linux, Windows or Mac OS. Download the relevant package and unzip/untar it somewhere. In Linux, I’d do:
tar -xvzf pdt-all-in-one-R20080603-linux-gtk.tar.gz
Which would create the Eclipse directory in the specified location. Go into the Eclipse folder and assign the correct permissions to the executable eclipse file under the Eclipse folder:
chmod 777 eclipse
And then run Eclipse by:
./eclipse
Alternatively, you can add a launcher/shortcut to the panel pointing to this so that Eclipse can be executed with a click of the mouse. Eclipse should be up and running now, with full PDT support. PDT provides error-detection, project builds, and debugging support amongst others.

Eclipse PDT contains many features useful for PHP development, including the much needed PHP debugging support. I’ll touch upon this in my next post. Enjoy Eclipse 🙂

June 26, 2008

Enabling Apache User Directories

Filed under: Development,Linux — mifan @ 10:25 am

In Apache, you have the choice of allowing users to have their own web URIs from within their home folders. Say for instance that my username is nazgul, on a domain http://somewhereonmiddleearth.com, then I can have a web-enabled folder within my home directory, which is accessible by http://somewhereonmiddleearth/~nazgul. To do this on Linux, do the following:
Create a folder named public_html
mkdir public_html
Go to your Apache folder (/etc/apache2/ on my system), and copy over the userdir modules from the mods-available folder to the mods-enabled folder:
cd /etc/apache2/
sudo cp -r mods-available/userdir.* mods-enabled/

Restart Apache2
sudo /etc/init.d/apache2 restart
And hey presto, your files under public_html would be accessible via a browser: thus if you are on your local machine, then you can access them via http://localhost/~nazgul . This is quite useful if you create web applications using Eclipse – where having your workspace inside public_html means they areautomatically web-accessible.

May 26, 2008

Python for Series 60 Upgrade

Filed under: Development,FOSS,Mobile — mifan @ 10:41 am

The newer version of Python for Series 60, 1.4.3, is out, and I’ve upgraded my Nokia 7610 with it. This is an Open Source project to port the beautiful Python programming language to Symbian Series 60 phones, which goes under the name of pys60. The advantage is that you’ve now got a scripting solution to access the Symbian Series 60 C++ APIs. Python provides the power of accessing underlying layers compared to J2ME, whilst remaining easier in terms of learning curve than Symbian C++. The python interpreter cum shell allows code to be executed directly on the phone – and the shell can be accessed via bluetooth from a PC, which means code written on the PC can be executed on the Phone real-time ! – More on that later though…
Installing Python on Series 60 devices is as simple as sending the relevant .SIS file of the interpreter to the phone. The SIS files are available on the project site: be sure to check the category of your Phone OS, and download the specific version. For Nokia phones, the version can be obtained at Forum Nokia. For instance, my Nokia 7610 is a Series 60 Feature Pack2, so I’ll be downloading Python for Series 60 FP2.
Now to check whether my previous code snippets are backward-compatible…. And yes, they are 🙂
Python Series 60

Next Page »

Blog at WordPress.com.