Wednesday, June 24, 2009

Byte ordering endianness mayhem! Aahh!

Working with (binary) data structures written to files can be difficult due to the fact that a hexdump of a binary file may appear to be out of order.

The following example may illustrate the source of confusion. This simple python script will create three binary files. To each file we will write 01 02 03 04 split up in different ways.
from struct import *
open('1long.bin','wb').write(pack('L', 0x01020304))
open('2short.bin','wb').write(pack('HH', 0x0102, 0x0304))
open('4char.bin','wb').write(pack('BBBB', 0x01, 0x02, 0x03, 0x04))
We can hexdump them to see their contents:
$ hexdump 1long.bin
0000000 0304 0102 (this is the long: 0x01020304)
0000004

$ hexdump 2short.bin
0000000 0102 0304 (these are the 2 shorts: 0x0102, 0x0304)
0000004

$ hexdump 4char.bin
0000000 0201 0403 (these are the 4 chars: 0x01, 0x02, 0x03, 0x04)
0000004
Before hexdumping, one might suspect that the outputs would be the same, but they're not. Each gave a diffent ordering.

These examples were run on a regular run-of-the-mill 32-bit Intel machine. These machines have a 16-bit word size (weird, yeah, I know) and use little-endian byte ordering. Little endian byte ordering means that the lest sigificant byte of any given data type, will go in the lowest-addressed memory space (or slot in a file on disk).

The least significant byte of the long 0x01020304 is the "04". Why doesn't it appear on the far left in the hexdump such as this: 04 03 02 01?

.... Need to finish writing in here ....

Diagram that I drew showing how reordering address so they increase from right to left can help understand little-endian byte ordering:




After googling around a bit, I found a thread which talks about how to reformat your hexdump so it is in big endian format (I believe). It's quite useful:

$ od -tx1 -w16 -Ax 1long.bin
000000 04 03 02 01
000004
$ od -tx1 -w16 -Ax 2short.bin
000000 02 01 04 03
000004
$ od -tx1 -w16 -Ax 4char.bin
000000 01 02 03 04
000004





Friday, March 13, 2009

c++ runtime "symbol lookup error"

I was working on a c++ project. Everything linked and compiled fine. Upon running the executable, I got the following error:
./TestCppProgram: symbol lookup error: ./TestCppProgram: undefined symbol: _ZN12CppProgramC1Ev
I searched the internet. Two of the interesting links I found were the following:
  • http://osdir.com/ml/gcc.g++.general/2005-02/msg00061.html
  • http://www.linuxquestions.org/questions/linux-software-2/undefined-symbol-cout-263568/
For me it ended up being a bad LD_LIBRARY_PATH. The path I intended the executable to find it's needed shared library was in the LD_LIBRARY_PATH, it just wasn't before a different path which had an older version of the needed shared library. (This happened to me when I updated by bashrc with a library path and just re-sourced it).

Some cool commands in the debugging process:
  • ldd TestCppProgram (Shows you where your program is getting it's libraries from. An early-on careful inspection of this would've quickly let me to my problem!)
  • ldd -d -r TestCppProgram (Shows you any undefined symbols. There shouldn't be any undefined symbols for an executable, but there will be for a shared lib if it depends on another shared lib. Somebody please correct me if I'm wrong)
  • nm TestCppProgram | c++filt (displays unmangled symbol information)
  • nm TestCppProgram (Displays mangled symbol information. Ie: You should be able to find stuff like ZN12CppProgramC1Ev in here. In my problem above, I found which line number the undefined symbol in question was on, and then looked it up in the unmangled version to see what function it was trying to resolve. It let me know, but it didn't really help me find out what my problem was.)
  • readelf -d TestCppProgram (Shows library dependencies. similar to ldd.)

Tuesday, March 10, 2009

Wednesday, January 07, 2009

Vim search and replace, with increment

:let i=1 | g/foo/s//\=i."morestuff"/ | let i=i+1

Saturday, October 25, 2008

Tuesday, October 21, 2008

A Brief Pyjamas + Django Tutorial

Update: (June 19, 2009) Since the writing of this post, Pyjamas has been updated and the below code no longer works, though the principles discussed are still valid. The example code is now being maintained in the pyjamas Sourceforge repo, located here.


Intro:


Django is a web framework written in Python. Pyjamas is a Python port of the google web toolkit (written in Java). Pyjamas can be used with Django to create web applications.

In terms of an MVC framework, Django acts as the Model and Pyjamas acts as the Views and Controller.

The "Todo List" Application:

In this brief tutorial, we will create a very simple todo list. The primary purpose of this tutorial is to briefly demonstrate how to serve data with Django, how to create and display widgets with Pyjamas, and how to handle user events with Pyjamas.

Prerequesits:

Here is the software that is needed:
This tutorial will not go into how to install each of these components, nor their basic usage. Hopefully, some of them can be downloaded as packages for your linux distribution. One reader recommended that you have a fairly recent Django version. > 1.0 should work.

The Code:

pyjsDemo/urls.py:
from django.conf.urls.defaults import *
from django.conf import settings

urlpatterns = patterns('',
(r'^services/$', 'todo.views.service'),
(r'^site_media/(?P.*)$', 'django.views.static.serve',
{'document_root': settings.STATIC}),
)

pyjsDemo/settings.py


# ADD THIS
import os
STATIC = str(os.path.join(os.path.dirname(__file__), 'media').replace('\\','/'))

# MODIFY THIS
DATABASE_ENGINE = 'mysql' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = 'todo' # Or path to database file if using sqlite3.
DATABASE_USER = 'todo' # Not used with sqlite3.
DATABASE_PASSWORD = '' # Not used with sqlite3.
DATABASE_HOST = '' # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3.

# MODIFY THIS
INSTALLED_APPS = (
'pyjsDemo.todo',
)

pyjsDemo/todo/models.py:

from django.db import models

class Todo(models.Model):
task = models.CharField(max_length=30)

def __unicode__(self):
return str(self.task)


pyjsDemo/todo/views.py:


from django.pimentech.network import *
from todo.models import Todo

service = JSONRPCService()

@jsonremote(service)
def getTasks (request):
return [(str(task),task.id) for task in Todo.objects.all()]


@jsonremote(service)
def addTask (request, taskFromJson):
t = Todo()
t.task = taskFromJson
t.save()
return getTasks(request)

@jsonremote(service)
def deleteTask (request,idFromJson):
t = Todo.objects.get(id=idFromJson)
t.delete()
return getTasks(request)


pyjsDemo/media/TodoApp.html:



See the download for this. It's short, but I can't figure out how to paste html into my blog. I'm lazy.



pyjsDemo/media/TodoApp.py:


from ui import Label, RootPanel, VerticalPanel, TextBox, KeyboardListener, ListBox
from JSONService import JSONProxy

class TodoApp:
def onModuleLoad(self):
self.remote = DataService()
panel = VerticalPanel()

self.todoTextBox = TextBox()
self.todoTextBox.addKeyboardListener(self)

self.todoList = ListBox()
self.todoList.setVisibleItemCount(7)
self.todoList.setWidth("200px")
self.todoList.addClickListener(self)

panel.add(Label("Add New Todo:"))
panel.add(self.todoTextBox)
panel.add(Label("Click to Remove:"))
panel.add(self.todoList)
RootPanel().add(panel)


def onKeyUp(self, sender, keyCode, modifiers):
pass

def onKeyDown(self, sender, keyCode, modifiers):
pass

def onKeyPress(self, sender, keyCode, modifiers):
"""
This functon handles the onKeyPress event, and will add the item in the text box to the list when the user presses the enter key. In the future, this method will also handle the auto complete feature.
"""
if keyCode == KeyboardListener.KEY_ENTER and sender == self.todoTextBox:
id = self.remote.addTask(sender.getText(),self)
sender.setText("")
if id<0: id =" self.remote.deleteTask(sender.getValue(sender.getSelectedIndex()),self)" method ="=" method ="=" method ="=">

pyjsDemo/media/build.sh

python ~/python/pyjamas-0.3/builder/build.py TodoApp.py



A very brief walk through of how to get this running:

Extract the demo:
  • tar -xvvzf pyjamasDjango.tar.gz
Create the db:
  • mysql -u root
  • > CREATE DATABASE todo;
  • > grant all privilages to todo.* to 'todo'@'localhost'; (or possibly > grant all on todo.* to 'todo'@'localhost';)
  • > exit;
Create the tables:
  • cd pyjsDemo
  • python manage.py syncdb
Build the javascript:
  • vim media/build.sh
  • (edit this so that it points to the build.py of pyjamas)
  • media/build.sh
Run the server:
  • python manage.py runserver
Test it out:
  • In your browser, goto: http://127.0.0.1:8000/site_media/output/TodoApp.html

Here are the demo source files:

Saturday, October 18, 2008

Java to Python conversion

Pyjamas is for python, what Google Web Tooklit is for Java. Pyjamas takes python code and some widget libraries and runs it through a python-to-javascript translator. The result: Coding web appliations has never been easier! JSON calls and event handling is a breeze. Little projects that I've attempted doing in the past, using libaries such as jquery, have left me writing a lot of javascript by hand, which get's ugly.

While I absolutley love pyjamas (and gwt for that matter), I really like the looks of extjs. The good news is that they have come out with extjs-gwt, which is "a Java library for building rich internet applications with GWT."

In talking with some of the folks on the pyjamas mailing list, I've gathered that the process of making pyjamas involved hand-converting the java libaries (ie widgets) of gwt from java to python. (If you look at the source code of both pyjamas and gwt, you can see that it's a pretty straight-forward conversion).

Surprisingly, pyjamas is only about 8000 lines of source code, which is relatively long. Extjs-gwt, on the other hand, as one person on the mailing list commented, has over 33,000 lines of java code in widgets alone!

The task of hand-translating extjs-gwt would be daunting.

I'm currently investigating the possiblity of writing a java to python converter to at least help out with the translation. A google search let me to a blog post which led me to an unmaintained project called java2python.

Because I'm personally interested in the workings of such technologies, and because I couldn't get java2python to work, I've decided to write my own translator.

I found a great article by the author of antlr (the same lexer/parser/tree_generator that java2python uses) on how to do such a translation between languages.

I hope that my efforts prove fruitful!

Pyjamas