Connecting the dots…

Posting a quick note to myself about how to hand to handcode a multipart request. Required for a REST server/client that I am currently writing. I needed this so that I can post a json object and the associated image in the same post without the typical php name=value paradigm and read from the php://input stream.
However the effort of parsing and un-parsing multipart messages makes me wonder if its just easier to base64 encode the file and send it part of the json object and then un-serialize it at the other end. Shall write more about it later.

$mime_boundary = md5(time());
$data = '--' . $mime_boundary . PHP_EOL;
$data .= 'Content-Disposition: form-data; name="data"' . PHP_EOL . PHP_EOL;
$data .= "Some Data" . PHP_EOL;
$data .= '--' . $mime_boundary . PHP_EOL;
$data .= 'Content-Disposition: form-data; name="logo"; filename="secretfile.jpg"' . PHP_EOL;
$data .= 'Content-Type: image/jpeg' . PHP_EOL;
$data .= 'Content-Transfer-Encoding: base64' . PHP_EOL . PHP_EOL;
$data .= chunk_split(base64_encode($file)) . PHP_EOL;
$data .= "--" . $mime_boundary . "--" . PHP_EOL . PHP_EOL; // finish with two eol's!!
die($data);
$params = array('http' => array(
                  'method' => 'POST',
                  'header' => 'Content-Type: multipart/form-data; boundary=' . $mime_boundary . PHP_EOL,
                  'content' => $data
               ));

$ctx = stream_context_create($params);
$response = file_get_contents($server, FILE_TEXT, $ctx);
print '--- printing out response ---' . PHP_EOL;
print_r($response);

I finally resolved the last bit of annoyance of Virtualbox Related to sharing folders. So here are the steps

$ # Create a holder on your host osx
$ mkdir -p /websites/topsecretproject/trunk
$ # turn off the VM if its on or skip this step
$ VBoxManage controlvm 'ubuntu-x11' poweroff
$ # Share this folder via VBoxManage, ensure that your VM is turned off
$ VBoxManage sharefolder add <vmname> --name <name> --hostpath <path of the directory on host>
$ # Bootup on the virtual box as headless unless you need to use its gui as well
$ # Ignore this step and boot normally if you want to use the X Server as well on the guest, i dont
$ nohup VBoxHeadless -s 'ubuntu-x11' -v off &
$ # You would want to mount the share as owned by the user on the guest
$ # so you need to get information about the user first
rp@x11-vm:~/code$ id rp
uid=1000(rp) gid=1000(rp) groups=1000(rp),4(adm),20(dialout),24(cdrom),46(plugdev),105(lpadmin),119(admin),122(sambashare)

The last step is required to know the group id and user id to mount the share as so that the guest user would have the right set of permissions. Now open up the /etc/fstab and create the following entry. You would obviously need to change the share name and path based on your settings. Mine looks like this.

# mount the share on the host machine
code            /home/rp/code    vboxsf   defaults,uid=1000,gid=1000 0 0

Save the file and exit and then remount all the files with the following command

rp@x11-vm:~/$ mount -a
rp@x11-vm:~/code$ ls -l
total 4
-rw-r--r-- 1 rp rp 20 2010-07-14 11:05 test.php

This will remount all the shares (be careful if you have any other shares currently in use and now when you do an ls in the shared folder you will see all the files are owned by the user in guest and you would the required permissions automatically set.

One of the reasons why I had to go this route was that sshfs didn’t work very well and macfusion failed to mount (with or without keys) and I couldn’t get it work. Would be great to hear if any of you have had similar problems and what were the approaches that you took to fix it.

Documenting my ngnix configuration for new symfony projects, if you got ideas for tweaking this further do leave a comment.

server {
    listen   80;
    server_name  be.aha.sulphur.local;
    root  /websites/aha-be/trunk/web;
    index index.php;
    charset utf-8;

    access_log  /var/log/nginx/be.aha.sulphur.access.log;

    location / {
          root  /websites/aha-be/trunk/web;
          index  index.php;

          if (-f $request_filename) {
            expires max;
            break;
          }

          if ($request_filename !~ "\.(js|htc|ico|gif|jpg|png|css)$") {
            rewrite ^(.*) /index.php last;
          }
    }

    location ~ \.php($|/) {
        set  $script     $uri;
        set  $path_info  "";

        if ($uri ~ "^(.+\.php)(/.+)") {
                set  $script     $1;
                set  $path_info  $2;
        }

        fastcgi_pass 127.0.0.1:9000;
        include /etc/nginx/fastcgi_params;
        fastcgi_param SCRIPT_FILENAME /websites/aha-be/trunk/web$script;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_param SCRIPT_NAME $script;
        fastcgi_intercept_errors on;
    }

    location /sf/ {
        root /opt/vs/php-5.3.2/lib/php/data/symfony/web/;
        expires max;
    }

}

Thanks to a messed up vm, I am migrating my dev to a new 10.04 ubuntu vm (which is great since I can now move away from 8.04 JEOS vms). I am taking this opportunity to move to php5.3 and nginx with php running as FCGI where possible.
I have in my previous posts talked about how to compile and patch php to get the php-fpm interface, however manually starting it with every-reboot can be a pain, so here’s the upstart script for it.

Create a new file /etc/init/php-fpm and past the following contents inside it:

description "PHP FastCGI Process Manager"
start on {net-device-up and local-filesystems}
stop on runlevel [016]

expect fork
respawn
exec /usr/sbin/php-fpm --fpm-config /etc/php/php-fpm.conf

Obviously tweak the path and configuration to your hearts content. Hope this will help someone who is lazy like me and doesn’t like writing the same commands over and over again.

Also to start the service then, run the following command:

$ sudo initctl start php-fpm

Phew that was a long heading. Things have changed in VirtualBox 3.2.4, setting up port forwarding is lot easier now and more readable. Adding notes for setting up and configuring my linux dev vm.

Install or import a new appliance and disable gdm, if all you need to do is run a couple of services on it, no point wasting memory running gdm.
Edit grub configuration

sudo vi /etc/default/grub

and edit the following lines.

#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="text"

comment out the first line and add the second one and then update the conf

sudo update-grub2

Now setup/install ssh atleast before you configure anything else.

sudo apt-get install openssh-server

Now reboot the vm to test if it all boots fine and doesn’t start the X server. Now turn it off again and this time boot it again as headless so its just a process like any other program running on your computer and possibly more efficient in terms of memory requirement.

But before that configure port forwarding. The syntax is for network connection setup as NAT

VBoxManage     modifyvm     [--natpf<1-N> [<rulename>],tcp|udp,[<hostip>],
                                          <hostport>,[<guestip>],<guestport>]
                                              [--natpf<1-N> delete <rulename>]

So based on this, to setup port forwarding from localhost port 2222 to guest port 22, the following and start the vm headless.

mac-host:~rpandit1016$ VBoxManage modifyvm "ubuntu-x11" --natpf1 "guestssh, tcp,, 2222,, 22"
mac-host:~ rpandit1016$ VBoxHeadless --startvm "ubuntu-x11"
Oracle VM VirtualBox Headless Interface 3.2.4
(C) 2008-2010 Oracle Corporation
All rights reserved.

Listening on port 3389.

Now you can configure more services by logging in. Remember to ensure that your VM is switched off though when you want to add a new rule, otherwise it will fail everytime you try and add it.

Streams are pretty much the forgotten features in php, I don’t see many people making use of it, partially because it works transparently under the hood when we use php’s functions to read/write from the disk or network. I shall write about how you can create custom stream handlers and protocols in php in userland which is pretty amazing, however for now I am documenting one of the crazy things I noticed in php.

I am currently writing a plugin for a web-service which allows you to post files to it. Its pretty simple, your code would use the plugin and then pass the uploaded file as an argument and it will post it to the endpoint of the web-service. You can read raw POST data using the php://input format which will give you the contents of a post looking similar to a query string, however if you wanted to upload a file, you can’t use the same to read the file directly to the stream. Part of what I was trying to accomplish with streams was to be able to write the file to an S3 server, however sadly there is no way (or atleast not that I know of to intercept the file before it gets written to the disk to avoid the wasted IO overhead).

Interestingly you can always post using the libCurl functions in php, however there is a little gottcha that I didn’t know about and that’s documented as the RFC1867-posting implementation of curl. So to upload a file using the command line curl in the same way as you would do with a multipart form you can run the following command along with the response, which is basically a dump of the $_POST and $_FILES global variables.

leonardo:foo rp$ curl -F upload=@nav_logo13.png -f  http://scratch.local/foo/server.php
Array
(
)
Array
(
    [upload] => Array
        (
            [name] => nav_logo13.png
            [type] => application/octet-stream
            [tmp_name] => /tmp/phpNLOAew
            [error] => 0
            [size] => 28736
        )

)

Note: Notice the @ sign before the file name, this is where the gottcha exists. Translating this into php code.

$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
// true to return the transfer as a string of the return value
// of 'curl_exec' instead of outputting it directly
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 

curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($ch, CURLOPT_URL, 'http://scratch.local/foo/server.php');
curl_setopt($ch, CURLOPT_POST, true);
$post = array(
  'logo' => '@nav_logo13.png',
  'first_name' => 'Rajat'
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$response = curl_exec($ch);
print_r($response);

Cleanup the code to use the bits that apply in your context but this is pretty much that it involves. Pretty interesting though I wish there was a better way, one of the ways would be to possibly run my own server using stream_accept_socket and then read directly from the socket but there is all this additional layer that I can get out of HTTP that I don’t want to loose.

Does anyone have a better idea? suggestions welcome.

I have started moving my dev environments from apache to ngnix for various reasons, mostly because ngnix has a smaller footprint and lets me get more out of low memory vms. I would probably write another post later about the reasons for switching from apache to ngnix for now, here are my notes for getting the php5.3.x/php-fpm and nginix working together. I should probably automate this process using hudson/puppet/chef configuration but that’s again an exercise for later.

Notes:
Install the required basic development tools (especially if you are using a new vm/install)

$ sudo apt-get install make bison flex gcc patch autoconf subversion locate
$ sudo apt-get install libxml2-dev libbz2-dev libpcre3-dev libssl-dev zlib1g-dev libmcrypt-dev libmhash-dev libmhash2 libcurl4-openssl-dev libpq-dev --with-mysql --with-pdo-mysql

Download and patch php 5.3

$ cd /usr/local/src
$ sudo wget http://uk.php.net/get/php-5.3.2.tar.gz/from/this/mirror
$ sudo tar zxvf  php-5.3.2.tar
$ cd /usr/local/src/php-5.3.2
$ sudo svn co http://svn.php.net/repository/php/php-src/branches/PHP_5_3/sapi/fpm sapi/fpm
$ sudo ./buildconf --force
$ sudo ./configure --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --enable-pdo --with-curl --enable-inline-optimization --with-bz2  --with-zlib --enable-sockets --enable-mbregex --with-mhash --enable-xslt  --enable-zip --with-pcre-regex --with-mysql --with-pdo-mysql

Now do the regular configure, make, make test and make install

If you are a purist and want to remove all the symbols in php, run the following command, personally I leave it as it is:

$sudo strip /usr/local/bin/php-cgi

Also if you want the mecache and apc goodness then run the following commands

$ sudo pecl install memcache
$ sudo pecl install apc

Now prepare for the configuration values:

$ sudo cp /usr/local/src/php-5.3.2/php.ini-development /usr/local/lib/php.ini #if you for dev, else choose recc
$ sudo mkdir /etc/php
$ sudo ln -s /usr/local/lib/php.ini /etc/php/php.ini
$ sudo ln -s /usr/local/etc/php-fpm.conf /etc/php/php-fpm.conf

The other important that needs doing is changing the user permissions in the php-fpm.conf file. It should basically be the same user with which ngnix user runs. It was www-data in my case.

Next step is to install Ngnix. There are tons of tutorials on the internet so not going to cover it. If you are on a debian based system, you can install it quickly using the apt-get command.

$ sudo apt-get install nginx

Now its time to start the php-fpm process, which you can do by running this as root.

$ sudo php-fpm -y /etc/php/php-fpm.conf

There might be a few more tweaks that need doing in the php-fpm.conf but leave that an exercise for the reader.

Now, tweak the ngnix host files under the /etc/nginx/sites-available to your hearts content and enable the fastcgi module in it. I shall cover its configuration in detail in the next posts.

I recently had the need to parse a massive (2gb) xml file and read it back to the database, the standard approach of SimpleXML and Dom was not going to work. I wanted to use XMLReader for this purpose. So here are the quick notes for helping anyone getting upto speed of what the quick start code looks like.

Quoting from its website:

The XMLReader extension is an XML Pull parser. The reader acts as a cursor going forward on the document stream and stopping at each node on the way.

// create the reader object
$reader = new XMLReader();

// reader the XML file.
$reader->open($abms_file);

// start reading the XML File.
while($reader->read()) {
    // take action based on the kind of node returned
   switch($reader->nodeType) {
       // read more http://uk.php.net/manual/en/class.xmlreader.php#xmlreader.constants.element
       case (XMLREADER::ELEMENT):
              // get the name of the node.
              $node_name  = $reader->name;
              // move the pointer to read the next item
              $reader->read();
              // action based on the $node_name
           break;
       case (XMLREADER::END_ELEMENT):
            // do something based on when the element closes.
            break;
   }
}

Pretty handy is this parsing method since it scales well for huge documents as well as for tiny documents and the memory footprint remains consistently the same.

Screen is awesome, I am surprised I never bothered to learn about it earlier. It all stemmed from the fact that I was working on a pretty low powered VM and couldn’t afford to open multiple xterms on it. Later when I started SSH’ing into it the screen utility came in pretty handy, however when you are on OSX and ssh into a remote box and use screen, when you hit backspace, it doesn’t instead gets the screen to give you a ‘wuff — wuff’ and I am not kidding, that’s the default message.
The fix is simple, run this on the console, close and restart, sorted.

defaults write com.apple.Terminal TermCapString xterm

Had a funny problem at work today, my php.ini was set to 1G however the script would still fail saying it couldn’t allocate 32bytes of memory which was a bit odd. A little investigation showed that at lib/php/phing/Phing.php

        // should return memory limit in MB
        $mem_limit = (int) ini_get('memory_limit');
        if ($mem_limit < 32) {
            // We do *not* need to save the original value here, since we don't plan to restore
            // this after shutdown (we don't trust the effectiveness of PHP's garbage collection).
            ini_set('memory_limit', '32M'); // nore: this may need to be higher for many projects
        }

contained this interesting bit of code, so (int) ini_get('memory_limit') return 1 instead of something more sensible. Pretty failsome way to detect how much memory is allocated, assuming the entire world would be setting the memory in megabytes. Why is our php.ini set to 1G is another story.