Connecting the dots... » Web Development

Notes on Git Part I / Quickstart Guide to Git

rp — Tue, 27 Dec 2011 12:25:46 +0000

Git stores all the information locally, commit logs, versions etc so there is very little you can’t do if you are offline
everything is check summed before it enters the repo, it uses SHA-1 hash which is a 40 character string composed of hex and calculated based on a file or directory structure in Git.
files have three states
- committed: data is safely stored in the local database
- modified: the file is modified but not committed
- staged: marked a file in its current version to go into your next commit snapshot.
Three main sections of a git project
- working directory: single check-out of one version of the project
- staging area: is a simple file that stores information of what will go in the next commit. its referred to an index
- git directory (repo) where all the metadata and the object database for your project is stored.

define your git identity:

$ git config --global user.name 'Rajat Pandit'
$ git config --global user.email 'rajat@virajsolutions.com'

define the editor

$ git config --global core.editor vim
$ git config --global merge.tool vimdiff

checking the settings $ git config --list
Initialize a git repo.$ git init
add files to the repo$ git add *.
commit message:$ git commit -m '';
cloning a repo: this is not the same as svn’s checkout. every version of every file is brought down. infect if the server crashes, any of the cloned copies can be used to bring the server back to the state in was in when it was cloned.
$ git clone git://bla.com/morebla.git mybladir
git has a number of transfer protocols
- git://
- http(s)://
- user@server:/path.git (ssh protocol)
files are either
- tracked: (modified, unmodified or staged)
- untracked: (any files in the working directory which are not present in the last snapshot and not in your staging area)
checking the status of files$ git status
revert a file change$ git reset HEAD # to upstage the command
if you do a "git add" and the modify the file and commit it the changes at the point of last 'git add' will get committed. if the new changes need to be added as well the file has to be 'git add''ed again’
defining ignore files by adding them to the .gitignore files. The following rules for the pattern work
- blank lines or lines starting with # are ignored
- standard glob patters work
- patterns can be ended with a / to specify a directory
- you can negate a pattern by starting it with a !

to see what will go in the next commit:

$ git diff --staged
$ git diff (will only show untagged changes)
$ git diff --staged (will show the diff for the staged files)

to remove a file$ git rm It removes it from the staging area and also physically moves to file to ensure that it doesn’t appear as a tracked file.
To remove a file from staging area but not form the file system.$ git rm --cached readme.txt
This is useful particularly in case where you possibly forgot to add it in the .gitignore file and it got pushed to the staging area and then had to be taken out but not removed from the filesystem.
To use the glob patterns, stuff needs to be escaped.
```
$ git rm log/\*.log
$ git rm \*~
```
moving files around:$ git mv file_from file_to
git status will show this as a file renamed. its the equivalent of renaming the file first, git rm the first file and then git add the new one
git has no meta data that tells you later that you had renamed the file.

viewing history using git log

$ git log -p -2
-p shows the diff
-2 limits the entries to 2 by default shows the most recent commit first.
$ git log --stat
--stat shows the abbreviated stats for all the changes that happened
$ git log --stat --pretty=
$ git log --stat --pretty=format:"%h - %an, %ar : %s"
(ref: http://progit.org/book/ch2-4.html)

$ git log --pretty=format:"%h %s" --graph

(adds a nice little ASCII graph showing your branch and merge history)
$ git log --since=2.weeks
$ git log --author= --grep="search in commit messages" to make both of them count you also need to pass --all-match option

git understands the difference between author and patcher.
- author: who created originally wrote the work
- committer: who applied the last of the work.
Amend a commit, in case you forgot to put a file in stage then you can put it as a second step:
```
$ git commit -m 'intial commit'
$ git add forgotten_file
$ git commit --amend
```

in case you have staged two files but want them to be committed in separate commits, how do you ‘upstage’ your files.

	$ git add * # staged all the changes
	$ git reset HEAD benchmarks.rb # keeps the file modified but takes it out of the staging area

reverting local modification $ git checkout -- benchmarks.rb
get the info about the remote git repo. if you have more than one remote repo then the command lists all of them
```
$ git remote -v
```

adding a remote repo

$ git remote add [shortname] [url]
$ git remote add pb git://github.com/paulboone/ticgit.git

now you can fetch all the changes in pb that are not in origin
$ git fetch pb
this will check out the changes in pb/master you can then merge it in to one of your branches or checkout a local branch at this point if you want to inspect it.
update your current code from the remote repo $ git pull # automatically pulls from the origin)
pushing changes to the report server
```
$ git push [remote-name] [branch-name]
$ git push origin master
```
note: push will only work only if no one else has pushed their changes, if that’s the case, your push will be rejected and you will have to pull to bring your code to the most recent state.
inspecting a remote $ git remote show [remote-name]
shows the branches on the remote and also the branch the pull is associated with.
git tagging $ git tag # list the number of tags n your code
git uses two main types of tags: lightweight and annotated. A lightweight tag is very much like a branch that doesn’t change – its just a pointer to a specific commit. Annotated tags, however are stored as full objects in the git database.The annotated tags are checksummed, contain a tagger name, email and date; have a tagging message and can be signed and verified with GNU privacy Guard. its generally recommended that you create annotated tags so that you can have all this information; but if you want a temporary tag or for some reason don’t want to keep the other information then keep the other information, lightweight tags are also available.

create and show new annotated tag

$ git tag -a v1.0 -m 'bla bla bla bla'
$ git show v1.0
$ git tag # lists all the tags on the code also create a signed tag
$ git tag -s v1.5 -m 'my super signed tag'

creating a light weight:

$ git tag v1.4
$ git tag
lists all the tag along with annotated, signed or lightweight tags
$ git show v1.4
* verifying a signed tag:
$ git tag -v v1..4

you require to have the public key installed to verify that the tag is from the right author

you can also tag later by specifying commit checksum or part of it.$ git tag -a v1.2 9fceb02
sharing tags, by default the tags are not transferred to the remote server when using git push,they have to be explicity pushed though
$ git push origin [tagname]
if you have lots of tags and want to push all of them at the same time.
$ git push origin --tags

Notes on VirtualBox / Quickstart guide to VirtualBox

rp — Fri, 23 Dec 2011 14:32:04 +0000

General Notes

“hosted” hypervisor or type 2 hypervisor where as a “bare-metal” or “type 1″ hypervisor would run directly on the hardware
open virtualization format (OVF)
VirtualBox Remote Desktop Extension (VRDE) – also support for full client USB Support
Extensible RDP authentication
USB Over RDP
Extension packs available
- the virtual USB 2.0 (EHCI) device
- virtual boc remote desktop protocol
- Intel PXE boot RPM with support for E1000 network card
fixed sized hard disk image is faster and has less overheads compared to an expanding overhead
different kinds of disks:
- VDI
- VMDK
- VHD
- HDD
Two importable formats:
- VMDK with a file in an XML Dialect with an .ovf extension. These files must reside in the same directory for the VirtualBox to be able to import them.
- Everything in a single archive called the .ova extension. its an archive file with a variant of the TAR archive format)
Commmands in the virtualbox package
- VirtualBox – virtualbox manager
- VBoxManage is a command line manager fro very detailed control
- VBoxSDL for simple graphical frontend
- VBoxHeadLess is yet another frontend to produce no visible output on the host at all.
Dynamic Kernal Module Support requires to be installed, this framework helps with building and upgrading kernel modules.
if the kernel of your linux host was updated and DKMS is not installed, in that case the kernel module will need to be reinstalled by executing (as root)
```
$ /etc/init.d/vboxdrv setup
```
virtualbox now not only supports PIIX3 but also the ICH9 chipset for latest operating systems like macosx
Auto mounting: linux guests automounted shared folders are mounted into /media directory align with the prefix sf_. for example, the shared folder mayflies would be mounted to /media/sf_myfiles on linux. The sf_ is controlled by the property /VirtualBox/GuestAdd/SharedFolders/MountPrefix
access to the automounted folders is only granted to members of the user vboxsf

listing the virtualbox guest property

$ VBoxManage guestproperty enumerate "Windows Vista"

or listing a specific property

$VBoxManage guestproperty get "windows vita" "/VirtualBox/GuestInfo/OS/Product"

to change values use $ VBoxControl guestproperty enumerate
VBoxManage guest-control to run commands inside the guest from the host machine
Memory ballooning is used to share memory associated with a vm with another vm. This is used for memory over commitment.
VBoxManage controlvm “VM Name? guestmemoryballoon where n is the memory the memory to allocate the guest vm. the guest additions software must be installed for this to run.
Virtual box dedups the page duplication by using PageFusion you can only turn it on when a vm is turned off
VBoxManage modifyvm "VM Name" --pagefusion on
you can observe page fusion operation using metrics RAM/VMM/Shared shows the total amount of fused pages, whereas the per-vm metric guest/RAM/Usage/Shared will return the amount of fused memory for a given VM.
Virtualbox allows you to manage the size of the disk even after the disk creation, even after data has been written to it.
Virtualbox disk format:
- normally by default for a new image, the format it uses is called Virtual Disk Image (VDI).
- VirtualBox also fully supports the popular and open VMDK contrainer format that is used by many popular virtualization softwares like VMWare.
- Alsofully supports the VHD format from microsoft.
- Image files from Parallers version 2 (HDD format) also supported. newer formats 3, and 4 are not supported however you can convert them into older format (version 2) by using tools provided by parallels.
Do not simply make copies of virtual disk images. if you import such a second copy into a virtual machine, virtualbox will complain with an error, since VirtualBox assigns a unique identifier (UUID) to each disk image to make sure it is only used once.
Disk modes:
- normal mode, allows you to take snapshots and restore from it.
- write-through hard disks: are completely unaffected by snapshots. their states are not taken and not restores when snapshots are restored.
- shareable hard disks: works the same as write-though hard disks however they can be used to be shared across multiple VMS writing to it directly. standard filesystem is not setup to be written by multiple file systems at the same time.only fixed size images can be used in this way, and dynamically growing images are rejected.
- immutable images only remember write access temporarily while virtual machines are running. so the approach is create a new ‘normal’ image and once you are happy with the setup and deem its contents useful then mark it ‘immutable’;
- if you take a snapshot of a machine with immutable images, then on every machine power-up those images are reset to the state of the last (current) snapshot (instead of the state of the original immutable image)
- As a result the same immutable image can be used by several virtual machines without restrictions. (Also looking at the ‘enabling the differencing disks’ that with auto-reset turned to off so that the vm’s sharing the immutable disk don’t forget the settings.
An image in ‘multi-attach mode’ an be attached to more than one virtual machine at the same time, even if these machines are running simultaneously. For each virtual m aching to which an image is attached, a differencing image is c reared. As a result data that is written to such a virtual disk by one machine is not seen by the other machines to which the image is attached. each machine creates its own history of the multi-attach image.
The standard approach would be ‘one immutable disk for the operating system’ and one ‘write-though’ for your data files.
Networking: virtualbox providers up to eight virtual PCI ethernet cards for each virtual machine. for each such card, you can individually select:
- The hardware that will be virtualised as well as
- The virtualization mode that the virtual card will be operating in with respect to your physical networking hardware on the host.
Networking modes:
- Not attached: in this mode the virtualbox reports to the guest that a network card is present but there is no connection, as if no ethernet cable was plugged into the card.
- NAT: default mode, only required to browse the web, download files and view email inside the guests. certain limitations when windows file sharing.
- Bridged network: more advanced needs such as network simulations and running servers in a guest. when enabled virtual box connects to on our installed network cards and exchanges network packets directly, circumventing your host operating system’s network stack.
- Internal networking: visible to only certain software based networking (selected virtual machine) but not to applications running on the host or the outside world. Internal networking is the same as bridged networking as that it lets the vm speak to the outside world except that the outside world is the other vm’s on the same host.
- Host-only networking: create a networking containing the host and a set of virtual machines without the need of the host’s physical network interface. instead a virtual network interface (similar to loopback interface) is created on the host, providing connectivity among virtual machines and the host.
- VDE (virtual distributed ethernet) networking:

port forwarding

VBoxManage modifyvm "VM name" --natpf1 "guestssh,tcp,,2222,,22"
natpf1 denotes the network card
2222 is the port on the host
22 is the port on the guest
guestssh is purely description and will be auto-generated if not present

to remove the same rule:
VBoxManage modifyvm "VM name" --natpf1 delete "guestssh"

NOTE: it is not possible to configure incoming NAT connections while the VM is running. However, you can change the settings for a VM which is currently saved (or powered off at a snapshot)
bridged networking: with bridged networking, virtualbox uses a device driver on your host system that filters data from your physical network adapter. this driver is therefore called ‘net filter’ driver. this allows virtualbox to intercept data from the physical network and inject data into it, effectively creating a new network interface in software. When a guest is using such a new software interface, it looks to the host system to the host system as though the guest were physically connected to the interface using a network cable. the host can send data to the guest though that interface and receive data from it. thismeans that yo can setup routing and bridging between the guest and the rest of your network.
running virtualbox as headless
```
VBoxManage startvm "VM Name" --type headless
```
the extra –type option causes VirtualBox to use VBoxHeadless as the front-end to the internal virtualization engine instead of the qt-front-end
the alternate syntax is:
```
VBoxHeadless --startvm 
```

Steps to install a vm headless server

create a new virtual machine

VBoxManage creative --name "Windows XP" --ostype WindowsXP --register

--registervm is optional, doing it now saves you from having to do that manually later.
--ostype is also optional but selecting this upfront results in some default options to be selected.
[/cpde]

To get the complete list

VBoxManage list ostypes

make sure that the vm is configured the guest operating system.
	1VBoxManage modifyvm "Windows XP" --memory 256 --acpi on --boot1 dvd --nic1 nat

Create a virtual hard disk for the vm (10GB)

VBoxManage createhd --filename "WinXP.vdi" --size 10000

Add an IDE controller to the new VM

VBoxManage storagectl "Windows XP" --name "IDE Controller" --add ide --controller PIIX4

set the VDI created above as the first virtual disk of the new VM

VBoxManage storageattach "Windows XP" --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium "WinXP.vdi"

attach the ISO file that contains the operating system

VBoxManage storageattach "Windows XP" --storagectl "IDE Controller" --port 0 --device 1

type

dvddrive --medium /full/path/to/iso.iso

Start the virtual machine

VBoxHeadless --startvm "Windows XP"

Notes on MongoDB / Quickstart Guide to MongoDB

rp — Fri, 23 Dec 2011 11:56:00 +0000

large system (multi-shards, config servers and mongo routers)
data model
- database holds collections
- collection holds documents
- document is a set of fields
- field is a key-value pair
- key is a name
- value is:
  - basic type
  - a document
  - or an array of values
mongo query language
BSON is a binary encoded serialization of JSON like documents. BSON was designed to be lightweight, traversable and efficient. BSON like JSON supports embedding of objects and arrays within other objects and arrays.
MongoDB uses BSON as a data storage and network transfer format for “documents”
```
$bson = bson_encode(null);
$bson = bson_encode(array('a' => 10));
```
using a database
```
	> use mydb
```
switched to db mydb

Insert data inside a collection

	> j = {name: 'mongo'}
	> db.things.save(j);
	> db.things.find();

no collection was predefined, it appeared lazily
no structure for the document to store
mongo addeds an objectID in the field _id

do stuff in a loop

	> for(var i=1;i<=20;i++) db.things.save({x:4, j:i});
	> db.things.find();

find returns back a cursor, navigation between results is possible using the ‘it’ command

c = db.things.find();
	> while (c.hasNext()) printjson(c.next());
	> db.things.find().foreach(printjson);

```
  > var c = db.things.find()
  > printjson(c[4]);
```
will show the 5th record, not right because it will load all the results unto 4 in the memory which can have scalability issues
```
> db.things.find().toArray();
```

more queries

select * from things where name = "mongo"
> db.things.find({name: 'mongo'}).forEach(printjson);
select * from things where x = 4;
> db.things.find({x:4}).forEach(printjson);
NOTE: that the _id field is always returned

allows returning only partial documents as well. for exam for name: ‘rajat’ return only the key j

	db.things.find({x:4}), {j:true}.foreach(printjson);
> findOne() - syntactic sugar
	>printjson(db.things.findOne({name: 'mongo'});
this is the same as
	>find({name: 'mongo'}).limit(1);
>printjson
function(x) {
	print(tojson(x));
}

mongo is full javascript shell, so any javascript function, syntax or class can be used in the shell.besides it defines some of its own globals like (b etc). you can see the full api at. http://api.mongodb.org/js/

select * from users where age=33 order by name
db.users.find({age: 33}).sort({name: true});

select * from users where age>33
db.users.find({'age': {$gt: 33}})
or $lt

select * from users where name like '%joe%';
db.users.find({name:/Joe/});
"Joe%" = {name:/^Joe/};
$lt = less than , $lte = less than equal to

sort order desc = .sort({name: -1});

create index myindexname on users(name);
db.users.ensureIndex({name: true});

create index myindex on users(name, ts desc)
db.users.ensureIndex({name:1, ts:-1});

select * from users where a=1 or b = 2
db.users.find({$or: [{a:1}, {b:2}]});

explain select * from users where z=2;
db.users.find({z:2}).explain();

select distinct last_name from users;
db.users.disctinct('last_name');

select count(*) from users;
db.users.count();

select count(age) from users;
db.users.find(age: {$exists: true}).count();

Commands:
the mongo db has a concept of a database command as a way to perform special operations, or to request information about its current operational status.
A command is sent to the database as a query to a special collection namespace called $cmd. The database will return a single document with the command results – user findOne(0 for that if your driver has it.
the general syntax is:
```
db.$cmd.findOne({: [, options]});
the shell provides a helper function for this:
db.runCommand({: [, options]);
```
for example to check our database current’ profile level eating
privileged commands:
```
> use admin
> db.runCommand('shutdown');
```
getting help info for a command
```
> db.commandHelp('datasize');
```

Clone Database

// copy the entire database from one name to one server
db.copyDatabase(, , );

Lock, Snapshot and unlock
sync command supports a lock option that allows to sa fely snapshot the database datafiles. while locked, all write operations are blocked, although read operations are still allowed. after snapshotting, use the unlock command to unlock the database an allows locks again.
```
> use admin
> db.runCommand({sync: true, lock: 1});
unlock command:
db.$cmd.sys.unlock.findOne();
```
while the database can be read while locked for snapshoting, if a write is attempted, this will block readers due to the database’s use of read/write lock. (issues 1423);
Index related commands:ensureIndex() is a helper function. its implementation creates an index by adding its into into the system.indexes collection
```
> use test
> db.mycollection.ensureIndex();
// same as
> db.system.indexes.insert({name: 'name', ns: 'namespaceToIndex', key: });
```
you can query system.indexes to see all indexes for a collection foo in db test.
```
> db.system.indexes.find({ns: 'test.foo'});
```

dropping indexes

db.mycollection.dropIndex();
db.mycollection.dropIndexes(); <-- drops all the indexes
e.g.
t.dropIndex({name: true});

index namespace:
each index has a namespace of its own for the btree bucket. the namespace is:
.$
this is an internal namespace that cannot be queried directly.
Last error commands:Since mongodb doesn’t wait for a response by default when writing to the database, a couple of commands exist for ensuring that these operations have succeeded. these commands can be invoked automatically with many of the drivers when saving and updating in the ‘safe’ mode.
```
db.$cmd.findOne({getlasterror:1});
or
db.runCommand('getlaterror');
or helper
db.getLastError();
```
getlaterror is primarily useful for write operation (although it is set after a command or query to) write operations by default do not have a return code. this saves the client from waiting for client/server turnarounds during write operations. one can always call getlasterror if one wants to return a code.
db.restError(); removes error that might have happened in the past, gets called before any comand is executed.
stats for a collection
```
> use comedy
> db.cartoon.validate();
```
returns stats about the collection along with the number of records etc
Mongo Metadata:
The
.system.* namespace in mongoDB are special and contain database specific information. the systems include:
- system.namespcaes list all namespaces
- system.indexes lists all indexes
- system.profile stores database profiling information.
- system.users lists users who may access the database
- local.sources stores replica slave configuration data and state.
- information on the structure of a stored object is stored within the object itself.
Collections:
mongo collections are essentially named grouping of documents. you can think of them roughly equivalent to relational database tables. It is a collection of a BSON documents.
- collection name should begin with letters and underscore. $ is reserved.
the max size of a collection name is 128 char

PuppetConf 2011 – Architecting for the Cloud: AWS Cloud Formation and Puppet – Jinesh Varia

rp — Fri, 23 Dec 2011 11:02:45 +0000

Excellent presentation, wish I had seen this before, For me slide 60 was pretty much the summary of the various cloud implementation routes, rest of it usually is just means of getting to it. Best use of your next 45mins.

AWS CloudFormation and Puppet at PuppetConf – Jinesh Varia

View more presentations from Amazon Web Services

Setting up Trac, SVN and Agilo on RHEL 5.x

rp — Fri, 23 Dec 2011 09:06:14 +0000

As part of yet another transformational project I am about to start from the very scratch. This time right from setting up the agile project management tools. I decided to go with trac (running agilo on top of it) for the agile delivery, svn for code versioning. This combination works very well together because:

You can have your product backlog, sprint backlog and tasks all in one place
Agilo manage your backlogs and provide fancy metrics of how your sprint is doing
SVN ties well into trac which means you can tie commits back into sprint tasks, which means you have a traceability matrix of the tasks/defects and the code changes made to close it
Trac also helps you manage a milestone so you can have a high-level view of how you are doing against your deadlines to meet a milestone

You can read more about them here Agilo and Trac

You first need to enable repoforge for your RHEL install and then install the required packages

[root@localhost ~]# rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
[root@localhost ~]# yum -y install subversion trac mod_dav_svn mod_python

Now you need to create the directories for svn repos, trac project and auth directory. I choose /var/www/ but you can choose anything else you want.

[root@localhost ~]# mkdir -p /var/www/svn/svnrepos #holds the svn repos
[root@localhost ~]# mkdir -p /var/www/trac #trac base folder
[root@localhost ~]# mkdir -p /var/www/svn/auth #holds the htaccess files
[root@localhost ~]# htpasswd -c /var/www/svn/auth/svn.htpasswd svnuser
[root@localhost ~]# svnadmin create /var/www/svn/svnrepos/secret-project #create svn repo

Now create the initial svn folders/files in the repo

[root@localhost ~]# svn mkdir file:///var/www/svn/svnrepos/secret-project/branches  \
file:///var/www/svn/svnrepos/secret-project/tags  \
file:///var/www/svn/svnrepos/secret-project/trunk  \
-m 'creating directories for initial import'

Committed revision 1.

Now create your first trac project

[root@localhost ~]# trac-admin /var/www/trac/secret-project \
permission add tracuser TRAC_ADMIN

You will need then require to go the settings of trac to give it the svn repo patch and then using trac-admin to call resync.

[root@localhost ~]# trac-admin $ENV repository resync "secret-project"

You would also require a post commit hook to ensure that trac gets updated every time a new commit is made. There is plenty of information available on http://trac.edgewall.org/wiki/CommitTicketUpdater, but basically you need is an svn post commit hook which calls trac-admin $ENV changeset added 'secret-project'.

Installing Agilo can be done using a plugin. If you have easy_install installed even better. Just follow the instructions as follows

[root@localhost ~]# easy_install .egg
[root@localhost ~]# cd /var/www/trac/britebill-holepunch/conf
[root@localhost conf]# trac-admin /var/www/trac/secret-project/ upgrade #upgrade trac post plugin install
Creating Product Backlog...
Creating Sprint Backlog...
Upgrade done.
[root@localhost conf]# trac-admin /var/www/trac/britebill-holepunch/ wiki upgrade #upgrade trac wiki as well

Now you can use the web interface to configure the project even more, adding your backlogs, tickets etc. Agilo also allows you to import your backlog from an xls sheet which is great as most product managers are still stuck in the xls era and refuse to let it go.

Automation for packaging websites

rp — Wed, 21 Dec 2011 00:24:55 +0000

For the ones who are following up my earlier blog post about packaging up website code, I had written about how you could also automate the process using phing/ant. I have committed a sample script up on github for anyone who is interested in looking at a sample implementation.

In order to use the phing script you would require to do the following:

Clone/download the code from git@github.com:rajatpandit/phing-website-packaging.git
Copy the debian folder across to your project
Tweak the files in there to match your projects requirements
The file names ending in .in are the template files that the phing script will take and modify before actually building the package for you
The phing script modifies the files based on two arguments that you provide build.project and build.version
The other file you need to look out for for is the install.in which contains the “from” and “to” locations of the files which will get deployed when the package is installed
The other files would required to be tweaked accordingly
Once you are happy with the changes, copy the package.xml file at the same level as the debian folder and ensure that the required packages are installed (refer to my previous post to know about the names of the packages

Execute the phing script as follows e.g.

$phing -f package.xml -Dbuild.version=1.0 -Dbuild.project=top-secret-project

In addition to the basic targets, it also has some additional target to rebuild the meta-information about your local repo. In case you don’t need one or use one, feel free to take the targets out.

Any questions, suggestions recommendations etc, as always please leave a comment.

Packaging up websites for easy deployment

rp — Mon, 19 Dec 2011 22:59:43 +0000

Over the last few years I have seen the emergence of several practices that are typically termed as ‘Best Practices for Cloud Hosting’ I have been involved in quite a few transitions of hosting providers from dedicated data centers to cloud based hosting providers and even to just a different data-center. Hence I shall be posting about these as a series.

As the first part of the series I am documenting about the common problems that most sys-admins have to face when they have to transition an application from a legacy datacenter to a different environment. Years of code/hacks usually the code base dependent on ‘assumed’ paths and ‘environment variables’. The other problem that people face is when its required to host the application on multiple servers. Which means when you need to install the latest version of your website you should be able to roll out the new code simultaneously on all the nodes in the cluster and be able to roll back similarly. This means a simple svn/git command is not going to be enough and would require well more than that.

In addition to the above problems there are other things that go hand in hand with the code deployment like your website server configurations (if your app code is dependent on things like apache modules or re-write rules etc) or any other configs like cron jobs etc. You might also want these to be updated (and corresponding services to be restarted) when the new version of code is deployed.

Packaging up the website code as a OS installer solves most of the above problems. You can make an rpm or a debian package out of your website and roll it to your local repository which can then be used to install or rollback to a particular version of your website code.

Since ubuntu is my OS of choice (atleast for now) this post will talk about how to setup a debian package, I might write a followup post for building RPM packages which turns out to equally similar to debian package.

Install the required packages

$ apt-get install dh-make devscripts

Now configure the dh_make command you updating some variables in your .bashrc file.

    $ cat >>~/.bashrc <
Checkout our code in your working directory in the structure defined below. This is  where you would be working and creating additional files that the debian package manager would use to build your package. Lets consider our project is called top-secret
 $ mkdir -p top-secret/top-secret-1.0; cd top-secret/top-secret-1.0
 $ svn co /trunk .
 $ tar -cvzf top-secret-1.0.tar.gz top-secret-1.0

The 1.0 is just an indicative version, once you have understood the process you can automate the process of creating directories with version numbers using automation scripts like phing/ant etc.
Now create the initial template files:
rp@chlorine:~/packaging/top-secret/top-secret-1.0$ dh_make -f ../top-secret-1.0.tar.gz
Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch or cdbs?
 [s/i/m/l/k/n/b] s

Maintainer name : Rajat Pandit
Email-Address   : rp@rajatpandit.com
Date            : Tue, 24 May 2011 09:08:35 +0100
Package Name    : hello-sh
Version         : 1.0
License         : blank
Using dpatch     : no
Type of Package : Single
Hit  to confirm:
Currently there is no top level Makefile. This may require additional tuning.
Done. Please edit the files in the debian/ subdirectory now. You should also
check that the hello-sh Makefiles install into $DESTDIR and not in / .

This process also builds a new tar file as well.
rp@chlorine:~/packaging/hello-sh$ ls -l
total 12
drwxr-xr-x 3 rp rp 4096 2011-05-24 09:08 top-secret-1.0
-rw-r--r-- 1 rp rp  167 2011-05-24 09:00 top-secret_1.0.orig.tar.gz
-rw-r--r-- 1 rp rp  167 2011-05-24 09:00 top-secret-1.0.tar.gz

Additionally it will also create a bunch of files under a new folder debian
debian/
├── top-secret.cron.d
├── changelog
├── compat
├── control
├── copyright
├── docs
├── files
├── install
├── postinst
├── postrm
├── README
├── rules
└── source
    └── format

These files are the ones that are used by the package manager to build a debian package.

You can learn about what each file does in the debian package managing guide http://www.debian.org/doc/manuals/maint-guide/dother.en.html which is why I won’t repeat that here.
The only important thing you need to remember is that the folder outside debian folder should be what would be the install path. Which means if your code has to go in /opt/vs/websites/top-secret the code should be in top-secret-1.0/op/vs/websites/top-secret
and also the file install should contain which folder needs to be copied where. This information should be one line per folder. For example if my code contains
├── b
│   ├── d
│   │   └── d
│   └── foo.php

and I want 'b' to be installed in /var/www/b and bla.php in /var/www/bin/bla.php then my install file would look like this:
b /var/www/b
bla.php /var/www/bin/bla.php

Building the package is also very simple. If you don't want to worry about signing your packages then you can just use the following command:
$debuild -us -uc

and you would get a .deb file in the end. You can check the contents of the debian file using
$ dpkg -c top-secret-1.0.deb

and that will list the contents of the the debian file along with its location for installation. 
Now to make this process a little more streamlined you can do a few other things:

Setup a local debian repository, which is accessible by all web servers, serve it over http ideally
Setup the local repo to point this repository so that all the webservers get to know about it
Setup some form of automation that running the commands, modifying the debian configuration files etc be done using ant or phing
Take the automation to the next level and call the script using Hudson

This will give you a nice workflow with the wonderful UI from hudson to managing building a webpackage and then deploying it to the web server. 
If you have further ideas of where this could be used please share your thoughts in your comments. I would like to credit the initial idea of packaging websites in RPM to the lovely sysadmins at IPCMedia. I modified the approach and moved from their technology to work for the ubuntu platforms.

Facebook Check-in deals

rp — Wed, 14 Dec 2011 21:27:23 +0000

Just readup about the new feature on facebook allowing individual business owners deals on facebook. its a brilliant idea for local businesses to attract, retain and build their customer base. Videos explaining what its about and the detailsabout it are here: https://www.facebook.com/deals/checkin/business/

To summarize its benefits:

Build Customer Loyalty: Award repeat customers by offering them a discount
Spread the word: Average user has 130+ friends, every check appears on their wall, helping spread the name of your business
Acquire new customers: Customers can find you based on your deals based on the location in which they are present. Offering deals gives them an incentive to come and visit you.

Apache Bench for benchmarking web servers

rp — Sun, 21 Aug 2011 09:34:29 +0000

I am nearly at the end of the project I had started for Infrastructural transformation for a client. The application is now fully functional on AWS. It has a horizontally scalable web tier with no sticky sessions, db running on RDS with A-Z Deployment and another instance running solr and memcache. It has various end points to extend itself and some day I am go to write more about the lessons learnt and the approach itself.

However more interestingly I decided to compare the performance of the initial dedicated hosting by bytemark and by the current aws setup. Now, I want to be clear about this, that I am not comparing a like for like setup. The dedicated hosting has everything running on the same machine and its probably a 8core dedicated box. On the AWS setup we have 2 core EC2 instances running the web-tier and currently two nodes behind the load balancer.

So this is what I did to get statistics.

$ ab -k -n 10000 -c 5 -g bytemark.csv http://bla.com/
$ ab -k -n 10000 -c 5 -g aws.csv http://beta.bla.com/

This gave me enough data against which I could plot some results. One of the options was to use MS Excel but lifes two short and my laptop doesn’t have a million gigs of ram, so i decided to use gnuplot for this instead.

here’s what my setup on plotter.p looks like:

set terminal png
set output "webserver-comparison.png"
set title "ab -n 10000 -c 2"
set size 1,0.9
set grid y
set xlabel "Request"
set ylabel "Response Time (ms)"
plot "bytemark.csv" using 9 smooth sbezier with lines title "nodejs", \
       "aws.csv" using 9 smooth sbezier with lines title "nodejs random timeout"

You can now go ahead and generate the required graph

$ gnuplot plotter.p

This is what the end result looks like:

What happened to new posts?

rp — Wed, 22 Jun 2011 06:35:29 +0000

I have been so far away from any of the blogging stuff for ages now. I primarily attribute that to a really busy last few months at IPCMedia and then starting a new job at Accenture. Also while I was at IPCMedia I was also working on a Infrastructural Transformation project for a startup which was very interesting.
I learnt a lot in that process, things I thought I knew were just a small piece of the puzzle, there was a lot more to it and I shall cover that over the next few posts. As the project comes to a closer to an end, I shall be writing posts with my notes about what I gathered from various software components I used during the entire process and hopefully someone would find them useful.