linuxjunk

Sum weights

2011-06-13T07:53:00.000-07:00

I have a file with a bunch of sequences and some weights at the top of the file:

>WEIGHTS 0.926434 1.000000 1.000000 0.926434 1.000000 0.892712 1.000000 1.000000 1.000000 1.000000 1.000000 0.892712 
>CRTC_EUGGR__Q9ZNY3 Calreticulin precursor.
XRKELWXXXXXXXXXXXXXXXXXXXXXXXXXTRWTHSTXXSDYXKFKLTSGKFYGDKAKDAGIQTSQDAKFYAISSPIASXXSXEXXXLVLQFSVKHXXXXXXGXGXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXKXEPRCEXDTLSHTYXAXXXXDXXXEVLVDQVKKESGTLEEDWEILKPKTIPDPEDKKPADWVDEPDMVDPEDKKPEDWDKEPAQIPDPDATQPDDWDEEEDGKWEAPMISNPKYKGEWKAKKIPNPAYKGVWKPRDIPNPEYEADDKVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXFYDQTNGATKDAEKKAFDSAEADKRKKEEDERKKQEEEEKKTAEEDEXXXDEXXXEDDKKDEL
>HSP47_RAT__P29457 47 kDa heat shock protein precursor (Collagen-binding protein 1) (GP46).
XRSLXXXXXXXXXXXXXXXXXXXXEAAAPGTAEKLSSKATTLAEXSXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXKXXXXSQAKAVLSAEKLRDEEVHTGLGELVRSLSNXTARNVTWKLGSXXXXXXXXSFADDFVRSSKQHYNCEHSKINFRDKRSALQSINEWASQTTDGKLPEVTKDVERTDXXLLXXAMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYXYXXXXXXXXQXVEMXXXXXXXXXXXXXXXXXXXXXRLEKXXTKXQLKTWMGKMQKKAXXISLPXGVVXVTHDLQKXXAGLGLTEAIXKNKADLSXXSGXXXXXXXXXXXXXXXEWDTEGNPFDQDIYGRXXXRSXXXXXXXXXXXXXXXXXXXXXXXXIGRLXXXXGDKMRDEL
>ENPL_PIG__Q29092 Endoplasmin precursor (94 kDa glucose-regulated protein) (GRP94) (GP96 homolog) (98 kDa protein kinase) (PPK 98) (ppk98).
XRAXXXXXXXXXXXXXXXXXXXXEVDVDGTVEEDLGKSREGSRTDDEVVQREEEAIQLDGLNASQIRELREKSEKXAFXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXELTVKXKCDKEKNLLHVTDTGVGMTREELVKNLGTIAKSGTSEFLNKMAEAQEDGQSTSELIGXXXXXXXXXXXXXXXXXVTXXHNNDTQHIWESDSNEFSVIADPRGNTLGRGTTITLXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSXKTETVEXPMXXXXAAKXEKEESDDEAAVXXXXXEKXPXTXKVEKTVWDWELMNDIKPIWQRPSKEVEDDEYKAFYKSFSXXXXXPMAYIHFTXXXXXXXXXILXXXXXXXXXLFDEYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXLNVSREXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXGVIXDHXXXXRLAKLLRFQSSHHPSDITSLDQYVERMKEKQDKIYFMAGSSRKEAESSPFVEXXXXXXXXXXXXXXXXXXXXXQALPXXXXKRFQNVAKEGVKFDESEKSKENREAVEKEFEPLLNWMKDKALXDKIEKAVVSQRXXEXXXXLVASQYGWXGNXERIMKAQAYQTGKDISTNYYASQKKTFEINPRHPLIRDMLRRVKEDEDDKXXSDLXXXXXXXXXXXXXXLLPDTKAYXXRIERMLRLSLNIDPDAKVEXXPXXXPXXTTEDTTEDTEQDDDEEMDAGADEXXQXTSETSTAEKDEL
>CRTC_HUMAN__P27797 Calreticulin precursor (CRP55) (Calregulin) (HACBP) (ERp60) (grp60).
XLXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTSXXIESXXXSDFXXFVLSSGKFYGDEEKDKGLQTSQDARFYALSASFEXXSXXXXXLVVXFXXKHXXXXXXGGGYVKLFPNSLDQTDMHGDSEYNIMFGPDIXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTYEVKIDNSQVESGSLEDDWDFLPPKKIKDPDASKPEDWDERAKIDDPTDSKPEDWDKPEHIPDPDAKKPEDWDEEMDGEWEPPVIQNPEYKGEWKPRQIDNPDYKGTWIHPEIDNPEYSPDPSIYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXAYAEEFGNETWGVTKAAEKQMKDKQDEEQRLKEEEEDKKRKEXXXAXDKEDXEXKXEDXXDXXDKXXDXXEDVPGQAKDEL

I want to sum the weights which is fine for this example with like 12 sequences. However, some of the files have a couple of hundred entries. Step in Bash:

 head -n 1 filename.txt | awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'

Giving me the sum of the weights! Woohoo.

find a random webserver using libcurl

2011-04-16T07:13:00.000-07:00

This post has moved here

Metasploit simple example (3.5.0)

2010-11-29T03:14:00.000-08:00

Download and install metasploit framwork from here: http://www.metasploit.com/framework/download/

run msfconsole type the following to run exploits:

db_driver sqlite3
db_connect
db_nmap [ip address]
db_autopwn -p -e -t

Unobservable alphabetic password entry

2010-11-27T15:43:00.000-08:00

This post has moved HERE

Unobservable pin entry

2010-11-27T05:09:00.000-08:00

This post has moved HERE

Grabbing environment variables in gnuplot

2010-11-23T08:27:00.000-08:00

This post has moved here

vim magic

2010-08-24T08:57:00.001-07:00

So I have a bunch of sequences in fasta format - and I need to rearrange the mo'fos.

To start with the entries look like this:

>F13C5.1        CE19383 WBGene00017422  status:Partially_confirmed      UniProt:O76564  protein_id:AAC64611.1

Run this vim command:

:%s:>\(\S\{4,}\)\t.*UniProt\:\(\S\{6,}\).*$:>\1_CAEEL__\2:g

And now they look like this:

>geneName_OrgID__UniProtAccNo

LaCie Network Space

2010-08-05T17:13:00.000-07:00

So.. you've hacked your network space and got root. But your lazy and you can't find up to date instructions on installing ipkg?

wget http://ipkg.nslu2-linux.org/feeds/optware/mssii/cross/stable/mssii-bootstrap_1.2-7_arm.xsh

and then sh mssii-bootstrap_1.2-7_arm.xsh

Now you have ipkg goodness.

Quick and Dirty histogram in awk

2010-07-30T08:18:00.000-07:00

awk '{histogram[int(($2*100)/100)]++;}END{for (i in histogram) { print i, histogram[i] }}' somerandomfilewithatagandcountdataincol2 | sort -n -k 1 > hist

Print every other line of a file

2010-07-30T07:56:00.001-07:00

cat blah | awk 'BEGIN{n=0}{if(n%2==0) print $0;n++;}'

Getting started with Cython on Centos 5.4 64Bit

2010-07-28T08:55:00.000-07:00

Centos is a nightmare. OK so first up I wanted to do with with Python 2.6 for which there is no official Centos package. So get the geekymedia repos which has a version of Python 2.6 which will install along side the existing 2.4 (and can be access as python26) here: here.

Then download the Cython tarball. Before you build Cython make sure you setup compile flags for 32bitness and make sure python 2.6 gets used i.e.

export CFLAGS="-m32 -march=i386"
export PYTHON="python26"

then run python26 setup.py install in the Cython directory. I had to do some other hacking of -m32 and -march=i386 on to things, but I think doing it this way should work. That should build and install Cython.

Put the following example in a file called demo.pyx:

cdef extern from "math.h":
    double sin(double)

cdef double f(double x):
    return sin(x)

print f(2.0)

Then make the following Makefile. Make sure you have tabs!


demo: demo.pyx
        cython demo.pyx --embed
        gcc -g -pthread -m32 -march=i386 demo.c -L/usr/lib -lm -lpython2.6 -o ./demo -I/usr/include/python2.6

Then just type make and it should build a binary called demo.

Digital Archaeology Tags

2010-06-14T03:12:00.000-07:00

It's fairly clear that in the future archaeology will move in to the digital domain with researchers digging through hard discs, USB keys, floppy discs, tapes, EEPROMs etc. I've often though it would be nice to drop them a message every so often, or just leave a note apologizing for my appalling spelling and grammar. So I suggest the following tags, to be encoded in 8bit ASCII when possible:

DIGIARCHNOTESTART

DIGIARCHNOTEEND

Dear future archaeologists, I shall use this in my documents from now on when I want to give you some background, point you in the direction of more information, or just say Hi, so grep away!

Grab all hrefs from a html page which have text containing View associated with them

2010-06-02T08:41:00.000-07:00

Quick and dirty perl program to grab all links from a webpage which have have anchors with the text "View" in them:

#!/usr/bin/perl

use constant false => 0;
use constant true  => 1;

use HTML::TreeBuilder;
use HTML::FormatText;

$html = HTML::TreeBuilder->new();
$html->parse_file($ARGV[0]);

my @stuff = $html->look_down( '_tag' , 'a' );

my $seqtag = "";
my $use_next = false;

for my $i (@stuff) {
  my @thing = $i->content();
  my $target = $i->attr('href');
  my $str = $thing[0][0];

  # string contains View
  if($str =~ m/View/) {
    print $target . "\n";
  }
}

samtools, bam cigar to string cigar

2010-05-04T07:03:00.000-07:00

This code fragment converts a bam style binary cigar string in to a normal cigar string.

    CigarLen = b->core.n_cigar;  // Cigar length
    uint32_t *cigar    = bam1_cigar(b);

    Cigar.clear();
    for(size_t n=0;n<cigar_len;n++) {
      uint32_t opcode = cigar[n] & 0x0000000F;
      uint32_t len    = cigar[n] & 0xFFFFFFF0;

      len = len >> 4;

      if(opcode == 0)  { Cigar += stringify(len); Cigar += "M"; }
      if(opcode == 1)  { Cigar += stringify(len); Cigar += "I"; }
      if(opcode == 2)  { Cigar += stringify(len); Cigar += "D"; }
      if(opcode == 3)  { Cigar += stringify(len); Cigar += "N"; }
      if(opcode == 4)  { Cigar += stringify(len); Cigar += "S"; }
      if(opcode == 5)  { Cigar += stringify(len); Cigar += "H"; }
      if(opcode == 6)  { Cigar += stringify(len); Cigar += "P"; }
      if(opcode == 7)  { Cigar += stringify(len); Cigar += "a"; }
      if(opcode == 8)  { Cigar += stringify(len); Cigar += "b"; }
      if(opcode == 9)  { Cigar += stringify(len); Cigar += "c"; }
      if(opcode == 10) { Cigar += stringify(len); Cigar += "d"; }
      if(opcode == 11) { Cigar += stringify(len); Cigar += "e"; }
      if(opcode == 12) { Cigar += stringify(len); Cigar += "f"; }
      if(opcode == 13) { Cigar += stringify(len); Cigar += "g"; }
      if(opcode == 14) { Cigar += stringify(len); Cigar += "h"; }
    }
  }

string to int conversion

2010-04-21T08:35:00.001-07:00

I was recently asked to write a string to int conversion function (without using library functions). I initially came up with a solution using the pow function (which is quite expensive). I had a think about it and found there were a surprising number of solutions. Briefly I came up with the following methods:

Method	Summary
Pow	my initial pow based solution (after converting a position to an in calculating 10^val)
Mul	Rather than using pow generating powers of 10 in the loop (multiplier = multipler*10)
Table	Just use a lookup table for the multipliers
Case	More of less the same as table, but encode the table in a switch statement (very ugly!)

Results are probably compiler/CPU/platform dependent. But on my Atom Z530 (1.6GHz) based netbook using GCC 4.3.3 I obtained the following results when performing the conversion 10 million times:

Method	User time
Pow	43.67s
Mul	28.21s
Table	28.22s
Case	29.13s

There's a big difference between the pow method and the others, but I was reasonably surprised that multiplier and table based methods performed similarly. It would be interesting to look at the assembler generated for these.

For reference, source code follows (note I sum and output the converted values to prevent the call to string_to_int from being optimised away). I was slightly concerned that something funky /might/ be going on in string::size() however benchmarked with this in and outside the loop and didn't observe any difference. Note: Following programs don't process signs, but in terms of benchmarking I don't believe this should be relevant.

Pow:

#include <string>
#include <iostream>
#include <math.h>
#include <stdlib.h>

using namespace std;

int string_to_int(string s) {

  int output=0;

  for(int n=0;n<s.size();n++) {
    int cval = s[n]-'0';
    output += cval*pow(10,s.size()-n-1);
  }

  return output;
}

int main() {

  // Simple tests
  cout << "1 is: " << string_to_int("1") << endl;
  cout << "10 is: " << string_to_int("10") << endl;
  cout << "14532 is: " << string_to_int("14532") << endl;

  int rsum=0;
  for(int i=0;i<10000000;i++) {
    string s;
    int numlen = rand()%11;
    for(int n=0;n<numlen;n++) {
      int rval;
      if((numlen==10) && (n==0)) { rval = rand()%2;  }
                           else  { rval = rand()%10; }
      s.push_back('0'+rval);
    }
    int v = string_to_int(s);
    rsum += v;
  }
  cout << rsum << endl;
}

Mul:

#include <string>
#include <iostream>
#include <math.h>
#include <stdlib.h>

using namespace std;

int string_to_int(string s) {

  int output=0;

  int mul=1;
  for(int n=s.size()-1;n>=0;n--) {
    int cval = s[n]-'0';
    output += cval*mul;
    mul = mul * 10;
  }

  return output;
}

int main() {

  // Simple tests
  cout << "1 is: " << string_to_int("1") << endl;
  cout << "10 is: " << string_to_int("10") << endl;
  cout << "14532 is: " << string_to_int("14532") << endl;

  int rsum=0;
  for(int i=0;i<10000000;i++) {
    string s;
    int numlen = rand()%11;
    for(int n=0;n<numlen;n++) {
      int rval;
      if((numlen==10) && (n==0)) { rval = rand()%2;  }
                           else  { rval = rand()%10; }
      s.push_back('0'+rval);
    }
    int v = string_to_int(s);
    rsum += v;
  }
  cout << rsum << endl;
}

Table:

#include <string>
#include <iostream>
#include <math.h>
#include <stdlib.h>

using namespace std;

const int powtable [] = { 1,
                          10,
                          100,
                          1000,
                          10000,
                          100000,
                          1000000,
                          10000000,
                          100000000,
                          1000000000
                        };

int string_to_int(string s) {

  int output=0;

  for(int n=s.size()-1;n>=0;n--) {
    int cval = s[n]-'0';
    output += cval*(powtable[s.size()-n-1]);
  }

  return output;
}

int main() {

  // Simple tests
  cout << "1 is: " << string_to_int("1") << endl;
  cout << "10 is: " << string_to_int("10") << endl;
  cout << "14532 is: " << string_to_int("14532") << endl;

  int rsum=0;
  for(int i=0;i<10000000;i++) {
    string s;
    int numlen = rand()%11;
    for(int n=0;n<numlen;n++) {
      int rval;
      if((numlen==10) && (n==0)) { rval = rand()%2;  }
                           else  { rval = rand()%10; }
      s.push_back('0'+rval);
    }
    int v = string_to_int(s);
    rsum += v;
  }
  cout << rsum << endl;
}

Case:

#include <string>
#include <iostream>
#include <math.h>
#include <stdlib.h>

using namespace std;

int string_to_int(string s) {

  int output=0;

  int pos=0;
  for(int n=s.size()-1;n>=0;n--) {
    int cval = s[n]-'0';

    switch(pos) {
      case 0:
        output += cval*1;
        break;
      case 1:
        output += cval*10;
        break;
      case 2:
        output += cval*100;
        break;
      case 3:
        output += cval*1000;
        break;
      case 4:
        output += cval*10000;
        break;
      case 5:
        output += cval*100000;
        break;
      case 6:
        output += cval*1000000;
        break;
      case 7:
        output += cval*10000000;
        break;
      case 9:
        output += cval*100000000;
        break;
      case 10:
        output += cval*1000000000;
        break;
    }

    pos++;
  }

  return output;
}

int main() {

  // Simple tests
  cout << "1 is: " << string_to_int("1") << endl;
  cout << "10 is: " << string_to_int("10") << endl;
  cout << "14532 is: " << string_to_int("14532") << endl;

  int rsum=0;
  for(int i=0;i<10000000;i++) {
    string s;
    int numlen = rand()%11;
    for(int n=0;n<numlen;n++) {
      int rval;
      if((numlen==10) && (n==0)) { rval = rand()%2;  }
                           else  { rval = rand()%10; }
      s.push_back('0'+rval);
    }
    int v = string_to_int(s);
    rsum += v;
  }
  cout << rsum << endl;
}

Floating point precision

2010-04-20T05:29:00.000-07:00

I was recently asked a question about the precision of floating point numbers. As most people know the representation used by most computers to store real numbers does not have continuous precision across the number line.

I thought it would be interesting therefore to plot the density of numbers across the number line. This is a nice illustration of the fact that numbers near 0 have higher precision that large numbers. The following plots are for 32bit floats (this was on a 64bit Snow Leopard Mac):

The following code was used to generate the data, which was plotted with Gnuplot:

#include <iostream>
#include <limits>

using namespace std;

int main() {


  //float smallest = 0.00000000000000000000001;
  float smallest = 0.00000001;

  float block_size = 10;
  for(float range_start=-100000;range_start<100000;range_start+=block_size) {
    float range_end = range_start+block_size;

    size_t different_n=0;
    float n;
    for(float n=range_start;n<range_end;) {

      float new_n=n;
      for(float i=1;new_n <= n;i++) {
        new_n = n + (smallest*i);
      }
      n = new_n;
      different_n++;
    }

    cout << range_start+((range_end-range_start)/2) << " " << different_n << endl;
  }
}

mpt2sas raid card from Dell r710 driver installation with Centos 5.3

2010-03-30T04:45:00.001-07:00

The REDACTED at Dell don't support Centos 5.3 on r610s raid card (Dell H200). I seriously REDRACTED hate propriety drivers and it's going to be a constant nightmare as we change kernel versions, but whatever. The problem is basically that the drivers are signed for Redhat Enterprise and Centos doesn't like that. So you have to crack open the driver package and remove the signing. A fixed package is available here: i686 x86_64.

I robbed most of the instructions to do this from here:

http://centos.org/modules/newbb/viewtopic.php?topic_id=22284&forum=37

Replace all instance of string with an incrementing number

2010-03-25T04:35:00.000-07:00

Perl solution with thanks to rmp. In this case replace all instances of MAGIC in the file small.sam:

cat small.sam | perl -pe 's/MAGIC/$c++/eg'

Pausing/Suspending a running process

2010-03-21T11:08:00.001-07:00

I had a couple of analysis jobs running they appeared to be stepping all over each other, with both jobs trashing a lot. I wanted to suspend one of the jobs to allow the other to continue in peace. You can do this using kill:

kill -STOP processid

and to continue it later:

kill -CONT processid

Found here:

http://linuxpoison.blogspot.com/2007/11/temporarily-suspend-process.html

Very simple OpenCL example

2010-03-17T09:33:00.000-07:00

This blog post has been moved HERE

netcat one line webserver

2010-03-17T04:41:00.000-07:00

Modified from the wikipedia article. Sets up a webserver on the local box (using only the netcat command) and serves a single file (called myfile) to all clients.

while true;do ( echo -e "HTTP/1.0 200 Ok\n\r"; cat myfile; ) | nc -l 8080 ;done

Samtools C example

2010-03-03T07:42:00.000-08:00

A simple samtools C api example. This program reads the bam file provided in the first argument and dumps the contents:

#include <stdlib.h>
#include <stdio.h>
   
#include "bam.h"
#include "sam.h"
 
int main(int argc, char *argv[]) {
 
  samfile_t *fp_in = NULL;
  bam1_t *b=NULL;

  fp_in = samopen(argv[1], "rb", 0);

  if(NULL == fp_in) {
     printf("Could not open file\n");
  }
 
  b = bam_init1();
  int pos=0;
  int lastpos=0;

  while(samread(fp_in, b) > 0) {
    lastpos = pos;
    pos = b->core.pos;
         
    if(pos != lastpos) {
      printf("tid  : %d\n",b->core.tid);
      printf("pos  : %d\n",b->core.pos);
      char *name  = bam1_qname(b);
      char *qual  = bam1_qual(b);

      int n=0;
      char *qseq = (char *) malloc(b->core.l_qseq+1);
      char *s   = bam1_seq(b);
      for(n=0;n<(b->core.l_qseq);n++) {
        char v = bam1_seqi(s,n);
        qseq[n] = bam_nt16_rev_table[v];
      }
      qseq[n] = 0;

      printf("name : %s\n",name);
      printf("qseq : %s\n",qseq);
      //printf("s cigar: %s\n",cigar);

      printf("qual :");
      for(n=0;n<(b->core.l_qseq);n++) {
        printf(" %d",qual[n]);
      }
      printf("\n");
    }
    bam_destroy1(b);
    b = bam_init1();
  }
  bam_destroy1(b);

  samclose(fp_in);
  return 0;
}

Print contents of all files prefixing lines with filename

2010-03-01T06:34:00.000-08:00

Print contents of all files (in this case all files starting with features.2d_table.* but not ending with png) prefixing the lines of the file with the filename:


find . -name "features.2d_table.*" | grep -v png | xargs  awk '{print substr(FILENAME,3) " " $0}' > bigtable

Change underscores in filenames to dots on a whole directory

2010-02-24T01:47:00.000-08:00

Change underscores in filenames to dots on a whole directory:

for i in *;do mv $i `echo $i | sed -e 's/_/./g'`; done

Simple SDL example in C++

2010-02-19T16:08:00.000-08:00

This blog post has moved here

Find and delete

2010-02-16T05:45:00.001-08:00

Someone didn't appreciate the existence of queueing systems like PBS and rolled their own queueing system that is supposed to take care of removing dead jobs from an app server. So I had to look for the old files that were causing the problem and see if I could delete them. Thankfully this is really easy with a simple find command.

 find ./ -type f -mtime +10

Switching the -type to d will do the same for directories. Unfortunately there is no clever hack for beating people who do this kind of thing over the internet.

n900 gnu screen in debian delete key doesn't work

2010-02-13T17:51:00.000-08:00

See title. Seems to be a long standing bug in the upstream Debian package or something: https://bugs.maemo.org/show_bug.cgi?id=3071 . Here's my complete screenrc which also fixes a problem with the enter key in vim and changes the default escape key of screen from ctrl-A to ctrl-B put this in ~/.screenrc

bindkey -a -k fe stuff ^M
bindkey -d ^@ stuff ^?
escape ^Bb

CppUnit disable exception handing

2010-02-05T11:45:00.000-08:00

By default C++ unit catches exceptions, this can make debugging a lot harder (as it often doesn't give you much information about the exception that was thrown or which line number it was thrown on). To disable the exception catching you need to remove the default protector:

// Normal test runner
CppUnit::TextUi::TestRunner runner;

// disable exceptions
runner.eventManager().popProtector();

Saving a process running in the foreground

2010-02-03T07:43:00.000-08:00

From here:

http://nickloman.wordpress.com/2010/02/03/saving-a-process-running-in-the-foreground/

Basically:

ctrl-Z
bg
disown –h PID

neat!

count all files matching some weird spec and piping though grep and stuff

2010-01-29T03:27:00.000-08:00

You could probably do all this with find, but piping though grep a) makes me happy and is exciting b) is a bit more flexible:

find . -atime +30 | grep somethinginfilename | xargs ls -l | awk '{print $5}' | awk 'BEGIN{n=0}{n += $1}END{print n/1024/1024/1024}'

Switching from tcsh to bash when you can't change your shell in /etc/passwd or whereever...

2010-01-26T05:44:00.000-08:00

Just add bash to your tcsh startup script. Edit ~/.cshrc and at the end add a single line saying: bash

When you exit you'll still have to exit twice if you do this a better option might be:

exec bash

Using gnu screens tab like functionality

2010-01-17T04:14:00.000-08:00

gnu screen is a cool tool, I normally use it for persisting shell sessions running over ssh when moving between workstations or sites. However screen also has tab like functionality which I've been looking at using after fire up screen and then use the follow to create and navigate between tabs:

Ctrl+a c: create a window
Ctrl+a n: next window
Ctrl+a p: previous window
Ctrl+a ": list all windows
Ctrl+a A: name a window
Ctrl+a k: close (kill) a window
Ctrl+a number: jump to a window

What's even better is that screen has console based cut and paste! Here's how you use it:

Ctrl+a [: enter copy mode (scrollback)

In copy mode, you can use the cursor keys (or vi navigation keys if you like) to move around. Press space to begin a selection and space again to end it.

Ctrl+a ]: paste text from buffer.

mplayer recursive play all files in directory

2010-01-15T05:01:00.000-08:00

This blog post has moved HERE

Simple python parallelisation example

2010-01-07T03:14:00.000-08:00

#!/usr/bin/python

import pp

def add_one(n):
  return n+1

job_server = pp.Server()

args1 = (1,)
args2 = (2,)
args3 = (3,)

f1 = job_server.submit(add_one, args1)
f2 = job_server.submit(add_one, args2)
f3 = job_server.submit(add_one, args3)

r1 = f1()
r2 = f2()
r3 = f3()

print "r1 is " + str(r1)
print "r2 is " + str(r2)
print "r3 is " + str(r3)

Annotating a Proxmark 3 mifare classic trace

2010-01-02T09:00:00.000-08:00

Proxmark 3s hi14areader command seems to initialise a card and send a request for block 0 using key A. It does this repeatedly a few times. I've annotated a trace from a mifare classic card.

 +      0:    :     52                                    # No idea, part of anti-collision from Koning thesis...
 +     68:   0: TAG 04  00                                # No idea, part of anti-collision from Koning thesis...
 +    -68:    :     93  20                                # Card UID request (READER)
 +    136:   0: TAG eb  5c  96  69  48                    # Card UID
 +   -136:    :     93  70  eb  5c  96  69  48  15  d4    # 93 70 followed by UID, followed by 2 CRC bytes (READER)
 +    212:   0: TAG 08  b6  dd                            # No idea, part of anti-collision from Koning thesis...
 +   -212:    :     60  00  f5  7b                        # Authentication request. 60 means auth with key A. (61 means key B). Use block 00. Last 2 bytes CRC (READER)
 +    337:   0: TAG a0  f4  b9  78                        # No idea                      *  mifarecrack tries to parse data from here 1
 +   -337:    :     52                                    # No idea - All starts again!  *  mifarecrack tries to parse data from here 2
 +      0:    :     52                                    # No idea                      *  mifarecrack tries to parse data from here 3
 +    405:   0: TAG 04  00                                # No idea
 +   -405:    :     93  20                                # Card UID request (READER)
 +    481:   0: TAG eb  5c  96  69  48                    # Card ID
 +   -481:    :     93  70  eb  5c  96  69  48  15  d4    # Reader sends UID again

Getting started with proxmark 3 (Proxmark III)

2009-12-31T11:43:00.000-08:00

This blog post has moved here

returning by reference faster than returning by value?

2009-12-04T05:57:00.000-08:00

Is returning by reference faster than returning by value? It's always been my assumption that returning by value can in many cases be optimized away. However to test this assumption I wrote a couple of small tests:

By value:

#include <math.h>
#include <iostream>
#include <vector>

using namespace std;

vector<size_t> test() {
  vector<size_t> t;

  for(size_t n=0;n<100;n++) {
    t.push_back(rand());
  }

  return t;
}

int main() {

  size_t sum=0;
  for(size_t n=0;n<10000000;n++){
    vector<size_t> t = test();
    for(size_t n=0;n<t.size();n++) {
      sum = sum + t[n];
    }
  }
  cout << sum << endl;

}

By reference:

#include <math.h>
#include <iostream>
#include <vector>

using namespace std;

void test(vector<size_t> &t) {
  for(size_t n=0;n<100;n++) {
    t.push_back(rand());
  }
}

int main() {

  size_t sum=0;
  for(size_t n=0;n<10000000;n++){
    vector<size_t> t;
    test(t);
    for(size_t n=0;n<t.size();n++) {
      sum = sum + t[n];
    }
  }
  cout << sum << endl;

}

Here are my timings on an Intel Core2Duo:

By value:
real 0m34.122s
user 0m33.219s
sys 0m0.436s

By reference:
real 0m34.147s
user 0m33.308s
sys 0m0.512s

So in this case the user times seem to be more or less the same. What's happening is the return value being optimized out as expected? Is there something else I should take in to account in my test?

i686-apple-darwin10-g++-4.2.1 was used.

http://cpp-next.com/archive/2009/08/want-speed-pass-by-value/

Also interesting comment on gcc mailing list:

Your subject isn't quite right.  It should be "Returning by reference faster than returning by value".                                                                                                                                      
                                                                                                                                                                                                                                            
C++ has RVO ("return value optimization").                                                                                                                                                                                                  
                                                                                                                                                                                                                                            
RVO is part of C++, just as the elide constructors is a part of C++.                                                                                                                                                                        
                                                                                                                                                                                                                                            
Also note:  you get RVO even if you are -O0.  It really is part of the C++ language, even for non-optimized compiling.                                                                                                                      
                                                                                                                                                                                                                                            
So your two tests as given should have the about the same performance, since the two tests are basically doing the same thing.  And, indeed, they do have about the same performance.  As expected.                                         
                                                                                                                                                                                                                                            
Read carefully the constraints upon when RVO can be applied.  (I don't have a URL handy.  You'll have to use your Google-fu, or read the ISO 14882 standard.)                                                                               
                                                                                                                                                                                                                                            
In situations where RVO cannot be applied, then for non-trivial data types, an output reference parameter will (likely) be faster.                                                                                                          
                                                                                                                                                                                                                                            
A lot of STL and Boost rely upon RVO for their template magic to be highly performant.

Reverting an svn commit

2009-12-02T04:10:00.001-08:00

To revert an svn commit do something like:

svn merge -r 4552:4551 svn+ssh://svn/path_to_svn_location
svn commit

Where 4552 is the last commit and 4551 is the one before.

Simple quicksort implementation

2009-12-01T05:58:00.000-08:00

A simple C++ quicksort implementation. This version is not in place and therefore requires O(n) extra space...

#include "math.h"
#include <vector>
#include <iostream>

using namespace std;

template<class value_type>
void quick_sort(vector<value_type> &values,size_t start,size_t end,size_t pivot) {

  vector<value_type> less;
  vector<value_type> greater;

  for(size_t n=start;n<end;n++) {
    if(n != pivot)
    if(values[n] < values[pivot]) {
      less.push_back(values[n]);
    } else {
      greater.push_back(values[n]);
    }
  }

  value_type pivot_temp = values[pivot];
  for(size_t n=0;n<less.size();n++) {
    values[n+start] = less[n];
  }

  values[start+less.size()] = pivot_temp;

  for(size_t n=0;n<greater.size();n++) {
    values[n+start+less.size()+1] = greater[n];
  }

  if(less.size()    > 1) { quick_sort(values,start      ,start+less.size(),start+(less.size()/2)); }
  if(greater.size() > 1) { quick_sort(values,less.size(),end              ,less.size()+(end/2)  ); }
}

int main() {

  vector<size_t> values;

  for(size_t n=0;n<10;n++) {
    values.push_back(rand());
  }

  cout << "start values..." << endl;
  for(size_t n=0;n<values.size();n++) {
    cout << "val[" << n << "]: " << values[n] << endl;
  }

  quick_sort(values,0,values.size(),5);
  cout << "sorted values..." << endl;
  for(size_t n=0;n<values.size();n++) {
    cout << values[n] << endl;
  }
}

Templates v subclasses v.2

2009-11-24T12:40:00.000-08:00

A slightly more complicated test similar to the previous post. In this case the templated version doesn't have such a huge advantage.

Subclassed version:

#include <iostream>

using namespace std;

class mybaseclass {
  public:

    virtual int get_i() = 0;
};

class myclass : public mybaseclass {

  public:

    myclass(int in) {
      i=in;
    }

    int get_i() {
      i++;
      return i;
    }

  int i;
};

class otherclass : public mybaseclass {

  public:

    otherclass(int in) {
      i = in;
    }

    int get_i() {
      return i*i;
    }

  int i;
};

int main(void) {
  myclass *c = new myclass(2);

  double sum=0;
  for(size_t n=0;n<2000000000;n++) {

    mybaseclass *cc = c;

    sum += (static_cast<double>(cc->get_i())/2.2);

  }
  cout << sum << endl;
  delete c;
}

Templated version:

#include <iostream>

using namespace std;

template<class t>
class myclass {

  public:

    myclass(int in) {
      i=in;
    }

    inline int get_i() {
      i++;
      return i;
    }

  int i;
};

template<class t>
class otherclass {

  public:

    otherclass(int in) {
      i = in;
    }

    inline int get_i() {
      return i*i;
    }

  int i;
};

int main(void) {
  myclass<int> *c = new myclass<int>(2);

  double sum=0;
  for(size_t n=0;n<2000000000;n++) {

    sum += (static_cast<double>(c->get_i())/2.2);

  }
  cout << sum << endl;
  delete c;
}

$ g++ small_classed.cpp -O3;time ./a.out
9.09091e+17

real 0m21.151s
user 0m20.909s
sys 0m0.072s

$ g++ small_templateclassed.cpp -O3;time ./a.out
9.09091e+17

real 0m20.736s
user 0m20.527s
sys 0m0.064s

Are templates faster than subclasses?

2009-11-24T11:30:00.000-08:00

Are template classes faster than using a derived class? Intuition would tell me that the template class should be faster. However to test if this is the case I wrote the following small programs:

Subclassed version:

#include <iostream>

using namespace std;

class mybaseclass {
  public:

    virtual int get_i() = 0;
};

class myclass : public mybaseclass {

  public:

    myclass(int in) {
      i=in;
    }

    int get_i() {
      return i;
    }

  int i;
};

class otherclass : public mybaseclass {

  public:

    otherclass(int in) {
      i = in;
    }

    int get_i() {
      return i*i;
    }

  int i;
};

int main(void) {
  myclass *c = new myclass(2);
  long int sum=0;
  for(size_t n=0;n<2000000000;n++) {
    mybaseclass *cc = c;

    sum += cc->get_i();

  }
  cout << sum << endl;
  delete c;
}

Templated version:

#include <iostream>

using namespace std;

template<class t>
class myclass {

  public:

    myclass(int in) {
      i=in;
    }

    int get_i() {
      return i;
    }

  int i;
};

template<class t>
class otherclass {

  public:

    otherclass(int in) {
      i = in;
    }

    int get_i() {
      return i*i;
    }

  int i;
};

int main(void) {
  myclass<int> *c = new myclass<int>(2);

  long int sum=0;
  for(size_t n=0;n<2000000000;n++) {

    sum += c->get_i();

  }
  cout << sum << endl;
  delete c;
}

$g++ small_templateclassed.cpp -O3;time ./a.out
4000000000

real 0m0.008s
user 0m0.000s
sys 0m0.001s

$ g++ small_classed.cpp -O3;time ./a.out
4000000000

real 0m5.908s
user 0m5.853s
sys 0m0.018s

My first effort was IO bound, but in this case it appears that templatized code is substantially faster.

Nabble thread is here.

Renicing synology nfsd so it has priority over other services

2009-11-20T02:07:00.000-08:00

We needed nfs to have priority over other services on a synology DS509+. To do this we reniced the nfsd process. Synologies are basically just linux boxes. So enable ssh on the synology and ssh to it (ssh synology -lroot) use your normal admin password for the password. Then edit /usr/syno/etc/rc.d/S83nfsd.sh. Where it currently reads:

/sbin/portmap
/usr/sbin/nfsd
/usr/sbin/mountd -p 892
;;
stop)

change it to read:

/usr/sbin/nfsd
/usr/sbin/mountd -p 892
renice -10 `pidof nfsd`

;;
stop)

Commands used to install golang

2009-11-16T17:44:00.000-08:00


Commands I used to install Go language (golang) on Ubuntu amd64:

sudo apt-get install mercurial bison gcc libc6-dev ed make
cd ~
export GOROOT=$HOME/go
export GOARCH=amd64
export GOOS=linux
mkdir $HOME/gobin
export GOBIN=$HOME/gobin
export PATH=$PATH:$GOBIN
hg clone -r release https://go.googlecode.com/hg/ $GOROOT
cd $GOROOT/src
./all.bash

To compile something make a test program e.g.:

package main

import "fmt"

func main() {
       fmt.Printf("hello, world\n")
}

and compile with:

8g test.go ;8l test.8 ;./8.out

Version

2009-11-16T03:55:00.000-08:00

Hi,

Probably did this before - but administered lost and found computers has made me need this info again.

Version of linux.

cat /proc/version

Gives you:

Linux version 2.6.28-15-generic (buildd@rothera) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #52-Ubuntu SMP Wed Sep 9 10:49:34 UTC 2009

And then you need the release info for the distribution.

lsb_release -a

Gives you:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 9.04
Release:        9.04
Codename:       jaunty

And of course uname -a. Which is a start. Would love to hear other tips on determining the state of a box that has previously been abandoned.

Update: Naming of the actual machine and shit as well.

Running hostname should give you the name of the actual machine. Not entirely sure how this works when you have multiple "machines" kinda running off the same physical machine.

user@machine:~$ hostname
machine-name

You can then check this the other way round:

user@machine:~$ host name.domain.tld
ip address .in-addr.arpa domain name pointer name.domain.tld

list installed packages on debian and uninstall those not correctly removed.

2009-11-15T16:31:00.001-08:00

dpkg --get-selections

Will give you a list of packages currently installed on a debian system (possibly works for Ubuntu too?). You may see a few which are listed as "deinstall". They've not uninstalled correctly. To uninstall them use this command I found on the debian mailing lists to purge them:

dpkg --get-selections | grep deinstall | cut -f1 | xargs dpkg -P

Turning a Ubuntu install into a Window terminal server dumb terminal

2009-11-13T07:49:00.000-08:00

I've written a quick script to term a clean Ubuntu (I'm using 9.04 netbook remix) into a dumb terminal to connect to a windows terminal server.

You can use it like this:

wget http://sgenomics.org/~new/term.tar
tar xvf term.tar
sudo ./termsvr MYTERMINALSERVERNAME MYDOMAINNAME

Under the hood, the script looks like this:

if [ $# -ne 2 ]
then
echo "Usage: `basename $0` "
exit 1
fi

apt-get -y --force-yes remove gnome*
apt-get update
apt-get install rdesktop
cp ./rc.local /etc/rc.local
sed "s/\*SERVERNAME\*/$1/" ./xinitrc > ./xinitrc.fixed0
sed "s/\*DOMAINNAME\*/$2/" ./xinitrc.fixed0 > ./xinitrc.fixed1
cp ./xinitrc.fixed1 /etc/X11/xinit/xinitrc

where xinitrc contains:

#!/bin/bash
# $Xorg: xinitrc.cpp,v 1.3 2000/08/17 19:54:30 cpqbld Exp $

# /etc/X11/xinit/xinitrc
#
# global xinitrc file, used by all X sessions started by xinit (startx)

while [ true ]
do
rdesktop -f *SERVERNAME* -d *DOMAINNAME* -u "" -k en-gb
done

# invoke global X session script
. /etc/X11/Xsession

and rc.local contains:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

startx

exit 0

Upgrading eclipse on Ubuntu

2009-11-13T05:38:00.000-08:00

After totally borking my eclipse install I seriously suggest just removing every mention of the damned thing before installing the new version. By removing I mean move to a new dir in /tmp/ of course.

Things to look out for:

/home/user/.eclipse
/home/user/.mozilla/eclipse
/foo/bar/eclipse

Seriously this should be a lot easier. Anyway - the other thing to attempt is exporting your old software update sites list and then importing this into the new install. This should save an ungodly amount of time reinstalling plugins - which vastly outweighs the amount of time installing takes.

strings in the n900 firmware

2009-11-11T10:08:00.000-08:00

The n900 isn't out yet, but it's firmware is! I ran strings over it and looked at the longer, more interesting strings as follows:

strings RX-51_2009SE_1.2009.42-11_PR_F5_MR0_ARM.bin | awk '{if(length($0) > 50) print $0}' | more

Some interesting finds:

Uses glibc, good to know... GLIBC_2.4

"Major hack here, this needs to be spawned from a proper class"

Always good to know. :)

Skype:

usr/share/locale/fr_FR/LC_MESSAGES/skype-ui.mo

Lots of these:

TMS320C6x COFF Linker Unix v6.0.7 Copyright (c) 1996-2006 Texas Instrument

''C:/users/Prasad/Nokia/CHK_100_V_H264AVC_D_BP_OMAP3430_1_Q

5320C6x COFF Linker Unix v6.0.7 Copyright (c) 1996-2006 Texas Instrumentl

Looks like they are making extensive use of the DSP on the OMAP3430. I guess for accelerating graphics etc. This is good news in terms of functionality, should make the whole experience smoother. But bad news for open source zealots as there's no open source TMS320C6x compiler.

Some familiar apps:

usr/share/doc/osso-fm-transmitter-l10n-cscz/changelog.gz

/usr/share/applications/hildon/chess_startup.desktop
/usr/share/applications/hildon/mahjong_startup.desktop
/usr/share/applications/hildon/osso_lmarbles.desktop
/usr/share/applications/hildon/osso_pdfviewer.desktop
/usr/share/applications/hildon/osso_calculator.desktop
/usr/share/applications/hildon/osso_rss_feed_reader.desktop

What does it use prolog for?!?

usr/share/doc/swi-prolog/changeD

No IPv6?

8Removing IPv6 stuff because it is no longer found in fremantle BUT only ifR

echo "Welcome to your upstart powered internet tablet."

Upstart powered!!

Custom bootloader:

INokia OMAP Loader v1.4.13 (Oct 5 2009) running on %s %s (%s

Wonder if this is the same bootloader they use on the n97...

%s: Checksum mismatch (stored 0x%04x, calc. 0x%04x). Who cares, right?

Yea! Who cares!

Yup, there it is. Stand by for some Linux pleasure.

That's right bitches! Bring on the pleasure.

Make directory if it doesn't exist return true so you can use it in a make file

2009-11-09T03:03:00.000-08:00

[ -d /opt/foo/bin ] || mkdir -p /opt/foo/bin

Routertech AR7RD ppp interface not coming up

2009-11-08T06:14:00.000-08:00

The ppp0 interface on my RouterTech router wasn't coming up on reboot, I have no idea why. To get it to come up I telneted in to the router (login as root, with normal admin password). Then executed the following:

/etc/ppp/ip-up

vector bounds checking in GCC

2009-10-12T06:20:00.000-07:00

Robbed from somewhere on the internet. I can't try this is my application because my libraries are not compiled with GLIBCXX_DEBUG but I believe this option should enable bounds checking on stl vectors etc. Here as a note for future use:

Pedantic debug mode checks not only for errors, but also for usages that rely on GNU extensions and that, while correct, may not be portable to other compilers. In pedantic mode, for example, constructing an std::string with a null pointer will result in an abort, not an exception. Pedantic debug mode is not the default. To enable pedantic debug mode, compile your program with both -D_GLIBCXX_DEBUG and -D_GLIBCXX_DEBUG_PEDANTIC.

Robbed from here: http://lists.apple.com/archives/xcode-users/2006/Feb/msg00209.html

Submit a bunch of jobs to SGE using a file as a template

2009-10-04T12:25:00.000-07:00

This script takes a filename from the third whitespace separated field of each file of the file "work-to-do" it uses this as the argument of a command (at /home/user/stuff/bin/stuff). The second argument of this command is the directory of the first argument. The third is a name generated from the current date. These commands are generated and then submitted to an SGE queue using qsub (the queue is called myqueueq). As always probably useless to anyone but me.

awk '{print "cd /home/user/stuff;./bin/stuff "$3" `dirname "$3"` auto-`date +%Y-%m-%dT$H-%M-%S`"}' work-to-do > sgecommands

awk '{print "echo \"" $0 "\" > /tmp/regen_run;qsub -q myqueueq /tmp/regen_run"}' sgecommands

Windows 95/98/Me registry kludging

2009-10-03T13:22:00.000-07:00

There is a neat trick that puts Windows 9x back into setup. This lets you resetup all your drivers and can be useful in recovering a dead system (for example one that will not boot into Windows normally). It also keeps all your application settings.
Basicly what you do is merge a registry file called "SYSTEM.1ST" which is stored in the root of the C: drive ontop of your registry. To do this do the following:

Reset the computer, when it says "Starting Windows..." press F5.
You should be presented with a C:\> prompt type:

attrib -h -r -s C:\SYSTEM.1ST
regedit /L:C:\SYSTEM.1ST /E C:\1ST.REG
regedit C:\1ST.REG

next time you boot Windows you will be presented with the setup screen. Your serial number sould already be displayed and you should just be able to click through most of the options.

If your registry is completely corrupt then you may want to just copy C:\SYSTEM.1ST over C:\WINDOWS\SYSTEM.DAT

Misc RM Connect 2.x fixes

2009-10-03T13:12:00.000-07:00

Is anybody still stupid enough to use RM Connect? I hope not but in the event that they are here are a random collection of bug fixes:

No packages install

Symptoms

No packages install on any workstations, even clean builds. No error is displayed at all.

Solution

Last time I encountered this it was due to the Nimda virus messing up Share permissions on our network. Check that the "Propagate" share on your PDC has the correct permissions. The share should be located at D:\RMVA\Manage\pkgctrl and permissions should be (as for all RM based shares) Everyone/Full Control. File permissions should be "Propagator/Full Control".

If this problems has been caused by Nimda trashing you file permissions you will probably find that many other shares have been distroyed as well. I've written some scripts that will reset the share permissions to there correct values, please contact at: help@asknav.com if you would like more information.

Users exceed quota due to large Internet History folders

Symptoms

Users (on a 200/300 station network) come to you on an almost daily basis saying that their file area is full and they can't logon, but when they look in their file area they can't find anything to delete. When you you examine their file area it's AppData is huge, and cookies/History information is using up their quota.

Solution

I've written a small program, which you run from the login script of all users, to solve this problem. Basicly it will delete everything in the Internet folder of a user as they login, this should keep the filearea down to a reasonable size.

The fix is available from the link below:

delHist2.exe

I remove NetBEUI from a client and it continues to work, but I get an error from licence manager when I bootup

You want to get rid of NetBEUI, good idea! NetBEUI produces far too much network traffic, and is not recommended for use on networks larger than 30/40 computers.

The problem you are getting is caused by the fact that RM licence manager is configured by default to use NetBEUI to check for licences. The good news is however that RM licence manager can be configured to use TCP/IP to do this. I've created a package to do this and it can be downloaded from the link below. To install the package on your network simply unzip the folder and copy it to D:\RMVA\manage\packages\32bitWin or the share \\PDCNAME\manage\packages\32bitWin. Then allocate it to all your workstations using application wizard.

remNetB.zip

Licence Manager doesn't work even on a clean build

Well, it sounds like licence manager is broken on the server. The simplest thing to do is reset it, and add some new licences. Licence manager is really a tool for you to keep track of licencing, so if you are having problems you can add as many licences as you want.

Step 1: Stop the "RM Licence Manager" service on all of your servers.
Step 2: Delete the file called "C:\WINNT\rmlicman.lic" or "C:\WINNT\System32\rmlicman.lic".
Step 3: Start the "RM Licence Manager" service on all of yout servers.
Step 4: Run the "RM Licence Manager" program from Admin Tools on the start menu, it may take some time to startup. Add as many licences as you want to each of your servers.

I'm running Sophos with file Protector, but everytime a user logs in it reinstalls itself, but I've got the correct exculsions setup

Solution

(Quick and Dirty tobe updated soon)

Your problem is that File Protector recovers the registry from a backup (taken when a package is allocated or when you "Backup key files") when the system boots up.

The easiest way to solve this is to stop File Protector from replacing the registry at boot. The downside is that any changes that a user makes could stick (which includes, if they know how to do it, disabling File protector).

OK what you need to do is edit a batch file in the "FileProt" package. This is located at: D:\RMVA\Manage\packages\32BitWin\FileProt on your PDC. Or mapped to L:\Packages\32BitWin\FileProt on Admin accounts. You need to edit the file in the STATION\HDS directory called recover.bat load it into notepad. Find the lines under "rem System.Dat" and alter them so they read:

rem System.Dat
rem c:\windows\command\attrib -r -h -s c:\Windows\System.DAT
rem c:\windows\command\attrib -r -h c:\BackUp\System.da0
rem copy c:\BackUp\System.da0 c:\Windows\System.DAT >> %logfile%
rem c:\windows\command\attrib +r +h +s c:\Windows\System.DAT
rem c:\windows\command\attrib +r +h c:\BackUp\System.da0

rem User.Dat
rem c:\windows\command\attrib -r -h -s c:\Windows\User.DAT
rem c:\windows\command\attrib -r -h c:\BackUp\User.da0
rem copy c:\BackUp\User.da0 c:\Windows\User.DAT >> %logfile%
rem c:\windows\command\attrib +r +h +s c:\Windows\User.DAT
rem c:\windows\command\attrib +r +h c:\BackUp\User.da0

Instead of:

rem System.Dat
c:\windows\command\attrib -r -h -s c:\Windows\System.DAT
c:\windows\command\attrib -r -h c:\BackUp\System.da0
copy c:\BackUp\System.da0 c:\Windows\System.DAT >> %logfile%
c:\windows\command\attrib +r +h +s c:\Windows\System.DAT
c:\windows\command\attrib +r +h c:\BackUp\System.da0

rem User.Dat
c:\windows\command\attrib -r -h -s c:\Windows\User.DAT
c:\windows\command\attrib -r -h c:\BackUp\User.da0
copy c:\BackUp\User.da0 c:\Windows\User.DAT >> %logfile%
c:\windows\command\attrib +r +h +s c:\Windows\User.DAT
c:\windows\command\attrib +r +h c:\BackUp\User.da0

The run Application wizard and allocate "FileProt", choose to re-allocate to workstations already setup.

Registry changes should new stick and Sophos should install itself only once.

If you still have problems make sure the follow exclusions are set in file protector:

C:\InterChk.chk
C:\Program Files\Sophos Sweep

Windows NT/2000 password recovery

2009-10-03T13:10:00.000-07:00

I've found two options for cracking Windows 2000 passwords (this probably still works for XP and maybe even Vista).

The first is to buy LockSmith and NTRecover from Wininternal, these cost a hell of a lot (about 200quid I think). They will however almost definitely work as the people who write them really know their stuff.

The second (cheaper option) is to use a free program that's available on the web (http://home.eunet.no/~pnordahl/ntpasswd/). I've saved many systems using this but comes with no guarantees.

The required files are available on the above website, with local copies below. Download them into a directory on your computer and then from the commandline do the following:

C:
CD C:\
rawrite2

When prompted for the disk image source file name enter "bd010114.bin" the target drive will of course be A:. Insert a disc and the program should create you a bootable floppy which is able to reset the Administrator password.

When you boot from this disc on the workstation you are cracking you should be able to accept the defaults for most of the options. You probably won't need to probe for SCSI devices, and you need to answer yes when prompted to write the SAM.
bd010114.bin
rawrite2.exe

Internet Explorer and Acrobat print incorrectly. But Word and notepad do.

2009-10-03T13:08:00.000-07:00

Symptoms

You can print from Microsoft Word and notepad correctly, but when you try printing from Internet explorer you get an error message similar to:

Line: 539
Char: 2
Error: Invalid argument.
Code: 0 URL: res://C:\WINDOWS\SYSTEM\SHDOCLC.DLL/preview.dlg

You may also (or possibly only) find that documents in Adobe Acrobat don't print correctly (words run into each other etc). The printer you're trying to print to is an HP printer.

Solution
These errors most commonly occur with HP printers, usually they are down to the PCL 6 driver that is supplied with the printer. Your best option is to download a PCL 5e, or PS driver from the HP website.

Appariently the Internet Explorer problem can also be caused by a faultly DLL (although I haven't come across this), details are available at: http://support.microsoft.com/support/kb/articles/Q293/1/76.ASP

Network configuration for older brother (1260/1660) printers/Emulex configuration

2009-10-03T13:04:00.000-07:00

I just rediscovered the internet archive backup of a website I had back in the 2000 odd. Most of the stuff from there is old and no longer relevant but may just be of some use to someone.

Network configuration for older brother (1260/1660) printers/Emulex configuration

Step 0 resets the card, you can probably skip this for a new card.

0. Remove the Emulex NetJet card from the printer.
0.1 Set Jumper JX1 to the 1-2 position (nearest the connector).
0.2 Put the card back in the printer and switch it on for 30 seconds.
0.3 Switch the printer off and remove the card.
0.4 Set JX1 to 2-3 and put the card back in the printer.
0.5 Switch the printer on.
1. Make a note of the MAC address of the printer (writen on the back of the NetJet card).
2. At a cmd prompt on the workstation type:
"arp -s ipaddress mac" for example: "arp -s 192.168.3.50 00-00-c9-09-22-67". Telnet to the printer
4. Enter the default password which is "access"
5. Type "su", when prompted for a password type "system"
6. You should now have a "Server>>" prompt.
7. Type "define subnet yoursubnet" to set the subnet.
8. Type "define node ip your_ip" to set the ip address. for example "define node ip 192.168.3.50"
9. Type "logout".
10. You should now be able to ping the printer from any workstation on the network.
11. You'll need to setup an LPR port on the server to connect to the printer, under Windows NT4/2000 use generic LPR.
12. The queue name will be "PASSTHRU", and protocol should be LPR not RAW.

A short script to move files out of the way retaining full path

2009-10-03T12:59:00.000-07:00

Find all files named *.fast4 under the current directory. Create replicate the path under ./obsolete and copy the file to the associated location. This script creates the required commands in a file called "moveobsolete" which you can then check over and execute.

find . -name "*.fast4" > obsoletefiles
awk '{print "mkdir -p ./obsolete/`dirname "$1"`; mv " $0 " ./obsolete/" $0}' obsoletefiles > moveobsolete
chmod 777 moveobsolete

Rename files matching given spec to incrementing filenames

2009-09-27T15:00:00.000-07:00

This one liner renames all files starting with shred_ to shred_0.tif, shred_1.tif etc.

ls shred_* | awk 'BEGIN{n=0;}{print "mv " $0 " shred_" n ".tif"; n++;}' | sh

My .mailcap

2009-09-23T14:43:00.000-07:00

Used with mutt to view gifs and jpegs as ascii graphics!!!! requires aview package on ubuntu/debian:

image/gif; asciiview '%s'
image/png; asciiview '%s'
image/jpg; asciiview '%s'
image/jpeg; asciiview '%s'

Robbed off some random place on the internets.

Simple pysam example

2009-09-22T08:55:00.000-07:00

This python code uses pysam to parse a bam file and print out all the alignment contained therein.

import pysam

# Open our sam file
samfile = pysam.Samfile()
samfile.open( "sim.bam", "rb" )

# Grab an alignment from the sam file

def m_fetch_callback( alignment ):
  print str(alignment)

num_targets = samfile.getNumTargets()

for n in xrange(num_targets):
  samfile.fetch(samfile.getTarget(n),m_fetch_callback)

Simple zlib file reading example

2009-09-18T04:00:00.000-07:00

This C++ example can read from both compressed and compressed files. It will read from testfile.gz (which can either be compressed or uncompressed) and write the contents to the standard output (screen).

#include "zlib.h"

#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
#  include <fcntl.h>
#  include <io.h>
#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
#else
#  define SET_BINARY_MODE(file)
#endif

#include <iostream>

using namespace std;

int main(void) {

  gzFile f = gzopen("testfile.gz","rb");

  char buffer[1001];

  for(;!gzeof(f);) {

    int e = gzread(f,buffer,1000);
    buffer[1000] = 0;
    cout << buffer << endl;
  }

  gzclose(f);

}

Compile with g++ testprog.cpp -lz -o testprog

Count files matching a given type/find spec

2009-09-18T03:27:00.000-07:00

Count file size in bytes of all files matching *fast4 modified in the last 90 days.

x=`find . -atime 90 -name \*fast4 -ls | awk '{print $7 " +"}' | xargs echo`; echo "$x 0" | bc

examine irssi scrollback

2009-09-18T03:26:00.000-07:00

I always find this a pain to lookup. But to scrollback in irssi: esc then p, forward esc then n.

iWorks '09 Keynote hangs loading a ppt file potentially losing all your work

2009-09-09T15:45:00.001-07:00

I had a problem. I'd been working all day on a presentation which I hadn't saved (yea I know). Then I tried to load a powerpoint file and Keynote just hung trying to load the file. If I couldn't get it to unhang I would have lost a days work as Keynote doesn't autosave. The solution is to attach gdb to the Keynote and get it to jump out of whichever loop it's stuck in. Like so:

news-macbook:MacOS new$ ps -ef | grep Keynote
 501 13290    82   0   1:22.74 ??        46:51.60 /Applications/iWork '09/Keynote.app/Contents/MacOS/Keynote -psn_0_1552763
news-macbook:MacOS new$ grep -p 13290
grep: invalid option -- p
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
news-macbook:MacOS new$ gdb -p 13290
GNU gdb 6.3.50-20050815 (Apple version gdb-1344) (Fri Jul  3 01:19:56 UTC 2009)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".
/Applications/iWork '09/Keynote.app/Contents/MacOS/13290: No such file or directory
Attaching to process 13290.
Reading symbols for shared libraries . done
Reading symbols for shared libraries ................................................................................................................................................................................................................................ done
0x942ed26d in ___CFBasicHashFindBucket1 ()
(gdb) bt
#0  0x942ed26d in ___CFBasicHashFindBucket1 ()
#1  0x942edd5b in CFBasicHashFindBucket ()
#2  0x942edcd4 in -[NSClassicMapTable objectForKey:] ()
#3  0x942edca8 in NSMapGet ()
#4  0x008dcf77 in -[ADReference reference] ()
#5  0x008db594 in -[ADNode firstChildWithName:xmlNamespace:objcClass:resolveReference:] ()
#6  0x008db3d2 in -[ADNode firstChildWith:resolveReference:] ()
#7  0x008db369 in -[ADNode firstChildWith:] ()
#8  0x009036b6 in -[ADSStyleSheet(ADInternal) parentSheet] ()
#9  0x00903f22 in -[ADSStyleSheet(Private) recursiveStyleWithIdentifier:] ()
#10 0x00903613 in -[ADSStyleSheet ensureOwnStyleWithIdentifierExists:] ()
#11 0x005f50a7 in +[PIOGraphicStyleMapper mapMasterStylesFromPptSlide:toStyleSheet:withState:] ()
#12 0x005fbe13 in -[PIOMasterSlideMapper mapMasterStyles] ()
#13 0x005fbf9d in -[PIOMasterSlideMapper map] ()
#14 0x00610ea9 in -[PIOThemeMapper mapMaster:includingGraphics:] ()
#15 0x00610f94 in -[PIOThemeMapper mapMaster:] ()
#16 0x006111b1 in -[PIOThemeMapper mapMasters] ()
#17 0x00611285 in -[PIOThemeMapper map] ()
#18 0x005f28c1 in -[PIODocMapper(Private) mapThemes] ()
#19 0x005f238b in -[PIODocMapper map] ()
#20 0x005f9467 in -[PIOImporter readFromFile:binaryDirectory:templatePath:localizer:] ()
#21 0x0022e0e2 in ?? ()
#22 0x0022e2d0 in ?? ()
#23 0x0022f789 in ?? ()
#24 0x0022ecea in ?? ()
#25 0x000a8c5f in ?? ()
#26 0x000a875f in ?? ()
#27 0x96c08769 in -[NSDocument initWithContentsOfURL:ofType:error:] ()
#28 0x96c0830d in -[NSDocumentController makeDocumentWithContentsOfURL:ofType:error:] ()
#29 0x000163ed in ?? ()
#30 0x00096fb4 in ?? ()
#31 0x02ddac2e in -[SFAppApplicationDelegate(Private) pOpenDocumentWithContentsOfFile:] ()
#32 0x02dd822d in -[SFAppApplicationDelegate application:openFiles:] ()
#33 0x001f7f55 in ?? ()
#34 0x96c0637d in -[NSApplication(NSAppleEventHandling) _handleAEOpenDocumentsForURLs:] ()
#35 0x96b44104 in -[NSApplication(NSAppleEventHandling) _handleCoreEvent:withReplyEvent:] ()
#36 0x9432346c in -[NSAppleEventManager dispatchRawAppleEvent:withRawReply:handlerRefCon:] ()
#37 0x94323230 in _NSAppleEventManagerGenericHandler ()
#38 0x90905de6 in aeDispatchAppleEvent ()
#39 0x90905ce5 in dispatchEventAndSendReply ()
#40 0x90905bf2 in aeProcessAppleEvent ()
#41 0x9061a381 in AEProcessAppleEvent ()
#42 0x969bddd6 in _DPSNextEvent ()
#43 0x969bd40e in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#44 0x9697f5fb in -[NSApplication run] ()
#45 0x0000f13f in ?? ()
#46 0x0000f0b8 in ?? ()
#47 0x0000ee50 in ?? ()
#48 0x0005cc79 in ?? ()
(gdb) ret
Make selected stack frame return now? (y or n) y

Keep 'ret'ing until you reach 'readFromFile'. At this point type 'c'. Keynote will then tell you it couldn't open the file and you will be able to save your existing document.

Postfix won't start

2009-09-03T06:31:00.000-07:00

Just noticed I wasn't receiving system mail for the last few weeks. Coming back after holiday. Anyway, one of the few services not to restart after the computer had been switched off in my absence was postfix.

Couldn't for the life of me work out the problem, it was complaining about not finding the host. Therefore I had a look in /etc/hosts and commented out the line below:

#
# hosts         This file describes a number of hostname-to-address
#               mappings for the TCP/IP subsystem.  It is mostly
#               used at boot time, when no name servers are running.
#               On small systems, this file can be used instead of a
#               "named" name server.
# Syntax:
#
# IP-Address  Full-Qualified-Hostname  Short-Hostname
#

127.0.0.1       localhost


# special IPv6 addresses
#::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

The one about IPv6. Dunno what that was about - but it works now.

Program for calculating E

2009-08-28T04:30:00.001-07:00

This post is now here

Remove any linefeed from line which did not end in capital or special character (vim)

2009-08-24T16:09:00.001-07:00

Remove any linefeed from line which did not end in capital or one of ./?<>:'

Like so:

%s/\([^./?<>":'A-Z]\)\n/\1/g

Twitter botnet/remote access

2009-08-19T07:00:00.000-07:00

Be the envy of your friends! Remote control your PC via twitter! Add the following code to cron to send all tweets with CMD in them to the shell.

curl twitter.com/new299 | html2text | grep "CMD" | awk '{$1="";$0=substr($0,2)}1' | sh

Simulate dataset with samtools, align with Bowtie, BWA, MAQ and pileup

2009-08-19T03:43:00.000-07:00

This post has moved here

Reset author to filename on a bunch of pdf files

2009-08-12T07:22:00.000-07:00

This script will reset the author to metadata on all files in the current directory to be the same as the filename of the file. It will write new files to a sub directory (which should already exist) called fixed.

Quick hacky script to so that papers are index more sensibility on my Sony Reader.

for i in *.pdf
do
        echo $i
        echo "InfoKey: Author" > data.txt
        echo "InfoValue: " $i >> data.txt
        cat data.txt
        pdftk ./$i update_info data.txt output ./fixed/$i
done

ö (O with umlaut) on Mac OS X

2009-08-06T03:20:00.000-07:00

This post has moved here

Running samtools pileup on some aligned reads

2009-08-05T08:43:00.000-07:00

samtools needs a sorted file for pileup.. doesn't give you an error if it isn't sorted so watch out!

./samtools faidx phi.fa
./samtools import ./phi.fa.fai /scratch/nava/lane7.phi.sam /scratch/nava/lane7.phi.bam
./samtools sort /scratch/nava/lane7.phi.bam /scratch/nava/lane7.phi.bam.sort
./samtools pileup -f ./phi.fa /scratch/nava/lane7.phi.bam.sort.bam > pile

vim/vi inverted replace

2009-08-05T06:30:00.000-07:00

Replace any character that isn't the ^ character with nothing:

%s/[^^]//g

n810 g++ installation (os 2008.1 diablo)

2009-08-01T06:28:00.000-07:00

Install the following repositories by editing /etc/apt/sources.list.d/hildon-application-manager.list

add the following:

deb http://repository.maemo.org/ diablo sdk/free sdk/non-free tools/free tools/non-free
deb http://repository.maemo.org/extras/ diablo free non-free
deb http://repository.maemo.org/extras-devel/ diablo free non-free

apt-get update
apt-get install g++

done

Openmoko DNS settings

2009-07-28T06:35:00.000-07:00

After several promising starts, I seem to have networking working reasonably reliably now. This is all available in the wiki on the site - so this is really just to remind me what the settings are for the resolv.conf after my office network moved to openDNS recently.

Home (wireless config)

echo nameserver 208.67.222.222
echo nameserver 208.67.220.220

Work (usb to computer)

search my-work.domain
nameserver 10.1.1.11
nameserver 10.1.1.12
nameserver 10.1.1.11
nameserver 10.1.1.12

No idea as yet how to automagically set these. Though this should get better. The network setup at work isn't helping though.

Quick bash script (check zero sized file) for reference

2009-07-17T03:46:00.000-07:00

This is a quick bash script I wrote. Pasting it here for reference. It reads a bunch of ipar intensity files processes them with swift, and calibrates them with the swift calibrator. It checks if the file is zero sized (that's because I was updating a set of files some of which had failed). The:

   if [ ! -s /scratch/user/"$lane"_"$tile".fastq.pf.calibrated ]

That checks that a file /is/ zero sized. And only runs swift if it is.

sourcedir=/ADIRECTORY

for lane in {1..8}
do

for tile in {1..100}
do
 if [ ! -s /scratch/user/"$lane"_"$tile".fastq.pf.calibrated ]
   then
   nice cp "$sourcedir"/s_"$lane"_*0"$tile"_int.txt.p.gz /scratch/user/input.gz
   nice gzip -d /scratch/user/input.gz
   nice ./ipar2gapipelineint /scratch/user/input "$sourcedir"/s_"$lane"_*0"$tile"_pos.txt /scratch/user/gaint
   nice ./swift --purity_threshold 0.56 --intfile /scratch/user/gaint --ref ./phi.fa --optical_duplicates_distance 6 --optical_duplicates_mismatches 8 --fastq /scratch/user/"$lane"_"$tile".fastq --phasing_window 10 --phasing_iterations 3 --align_every 50
   nice ./QualityCalibration/quality_remap --in /scratch/user/"$lane"_"$tile".fastq.pf --mapfile ./QualityCalibration/firecrest_1_3_2_fixedtable --out /scratch/nava/"$lane"_"$tile".fastq.pf.calibrated
   rm /scratch/user/input
   rm /scratch/user/gaint
   rm /scratch/user/*.fastq.pf
   rm /scratch/user/*.fastq.nonpf
 fi
done
done

Palm Pre uses glibc and has busybox

2009-06-10T04:42:00.000-07:00

This post has moved here

Making a C++ object printable to standard output

2009-05-17T10:03:00.000-07:00

You want to make a C++ object printable to the standard output. For example you have an object call ScoredSequence and you want do be able to do: ScoredSequence s("BLAHBLAH"); cout << s; You need to add a << operator to ScoredSequence, for example:

class ScoredSequence {
public:

  ScoredSequence(string s) : sequence(s) {
  }


  string get_sequence() {
    return s;
  }

  /// \brief Extractor which prints the whole sequence.
  inline friend std::ostream& operator<<(std::ostream& out, const ScoredSequence& s) {
    out << s.get_sequence();
    return out;
  }

private:
  string sequence;
};

delete all linefeeds from a file

2009-05-12T06:13:00.001-07:00

like this:

tr -d "\n" < gaps2 > nogaps

Connect to Freerunner Android from Opensuse

2009-05-07T01:31:00.000-07:00

Plug in the freerunner, and yast2 should prompt you to configure a new network device of type usb. Follow the instructions, and configure using ifup (Not network manager) and use a static IP address for the freerunner. Assign it to 192.168.0.200.

The adb program is available here: http://people.openmoko.org/sean_mcneil/adb

Download it and put it somewhere and make it executable.

Then save this set of commands as a bash script with a name like connect or whatever.

#!/bin/bash

$/usr/local/share/adb kill-server
$ADBHOST=192.168.0.202 /usr/local/share/adb devices
$/usr/local/share/adb shell

Make this executable and then run it. It should drop you automatically into the shell interface on your freerunner.

Multi column table in latex

2009-04-30T09:26:00.000-07:00

You want a table the spans across pages and raps in 2 columns. You need supertabular and to stick everything in a \twocolumn. Do it:

\pagebreak
\twocolumn
\begin{supertabular}{ |l|c|l| }
\hline
stuff1 & stuff2 & stuff3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
1 & 2 & 3 \\
\hline
\end{supertabular}
\onecolumn
\pagebreak

latex "File ignored" not found on nfs ( isilon )

2009-04-17T05:28:00.000-07:00

You are unable to compile latex files (i.e. latex myreport.tex) on an nfs drive but they compile correctly from the local disk. You receive errors such as "File ignored" or "no found". In my case this occurred when using an Isilon. The solution is to switch file ID size from 64bit to 32bit on the isilon configuration. "File Sharing"->NFS->"Edit Settings" on the Isilon web configuration.

Growing a VM

2009-04-03T08:44:00.000-07:00

I started a VM with the default 8Gb for a linux install. I have successfully added additional HDD to the VM but eventually I hit a problem that required editing the actual size of the main disk. I used the vmware-vdiskmanager (-x 12Gb Kubuntu.vmdk) to successfully resize the VM disk. Then I booted the VM into a gparted image using the ISO as a CD (option in the VMware server). This worked fine. I then edited the partition. First increase the size of the swap. Then shift it to the right and then make it smaller again, before resizing the main partition. Click apply changes and then wait. Or in my case head up to the terrace and enjoy beer and "Schnittchen". The KDE session does appear borked after all this. It complains about some DCOP setting not running. I tried changing permissions on the tmp dir and following general advice on the internet - however moving .kde to .kde_old solved the problem for me. The partition is now showing up as 12 gigs and all the files seem to be okay.

Aluminium Mac Keyboard disassembly

2009-03-07T10:29:00.000-08:00

This post has been moved here

avr-gcc read I/O on single port with AT90USB162

2009-03-06T15:14:00.000-08:00

I want to repeatedly read from and write to a single port on a AT90USB162. This is to implement a key matrix scanner and pins will be repeated reconfigured as input or output. I ran in to trouble, reads seems to overlap as if they were being cached.

For example, I have pins 1 and 2 shorted.

pin 1 is set as an output. pin 2 an input.

pin 2 is set to pullup.

I output logic 0 on pin 1.

I read the value from pin 2 and store the result in a variable VAL1.

Now I set pin 1 as a input (and pullup).

I set another pin, say pin 3 to logic 0 (not shorted to anything).

I output logic 0 on pin 3.

I read the value from pin 2 in to a variable VAL2.

Now logically VAL1 should be 0 and VAL2 1. However that doesn't seem to be the case. If I do this I often read 1 for both VAL1 and VAL2. Removing the second read however makes VAL1 revert to it's correct value, EVEN THOUGH BOTH READS SHOULD BE INDEPENDENT!! I've not seen this with the AT90USBKey so it's a bit odd.

The solution I've found is to place a delay loop after the port configuration but before the read, in my code:

inline unsigned char get_scan(unsigned char pin) {

  unsigned char p = 1 << pin;

  DDR = 0x00 ^ p;
  PORTD = 0xFF ^ p;

  for(volatile int i=0;i<100;i++);

  return ~p & PIND;
}

The volatile is important, if you omit that it doesn't work. I guess the compiler optimizes it out. (volatile tells the compiler some other external code could change this variable so explicitly check it where ever it's referenced).

jvterm: text based cut and based

2009-03-04T12:47:00.000-08:00

I've been trying to rid myself of the mouse. The final thing holding me back is cut and paste for terminal applications. Turns out that most Linux terminal programs don't have any support for keyboard based cut and paste. And in fact the only mainstream terminal app I've seen it in the The OS X Tiger Terminal.app (it seems to have been removed from later versions).

Anyway, after some hunting I found a Linux terminal program that does text based cut and paste. It's called jvterm. The downside is that though it's an X11 application it's not really designed to run in concert with X. It only works full screen and you can't alt-tab between it and X applications. Anyway, I couldn't find the documentation for text based selection so here's typically how you might use it:

Press Shift twice to enter selection mode

Use the cursor keys to move to the start of the selection region

Press Shift again

Move to the end of the selection region (text will be highlighted)

Press space

The text will now be on the clipboard. To paste it press Shift Twice and then Enter.

This is a really neat feature, and I hope it finds its way in to other terminal applications.

(protip: Read the man page. I found this out from guessing, but it's actually in the man page).

AHHH! You can do something similar with screen but how do I get it to share the X clipboard?!?

There's a urxvt (rxvt-unicode) perl script for this!

I found it in google cache and have copied it here:

HERE

To use it press alt-y to activate. Move using vi style keystrokes (hjkl) then press space to start selection and space to end. The text appears on you x buffer via xclip! Neato!

To install the script copy it to ~/.urxvt The copy the following in to you ~/.Xdefaults:

URxvt.keysym.M-y: perl:mark-and-yank:activate_mark_mode
URxvt.keysym.M-u: perl:mark-and-yank:activate_mark_url_mode
URxvt.perl-lib: /home/new/.urxvt/
URxvt.perl-ext-common: tabbed,mark-and-yank
URxvt.urlLauncher: firefox

Gaussian and Binomial distributions

2009-03-01T11:25:00.000-08:00

I'm most probably being entirely stupid here...

I've been trying to understand the Gaussian distribution and it's relation to the Binomial distribution. I've kind of decided that the Gaussian distribution is a continuous representation of the Binomial distribution. Is this correct?

Anyway, here is some C++ which creates a binomial distribution by adding n random values (-1,0 or 1) to a fixed initial value. I think for this as a kind of 2D random walk. Here's the code:

#include <vector>
#include <iostream>
#include <math.h>

using namespace std;

int main(void) {

  vector<size_t> space(100,0);

  size_t tries=1000000;
  size_t offset=50;
  for(size_t n=0;n<tries;n++) {

    size_t moves=1000;

    int final=50;
    for(size_t t=0;t<moves;t++) {
      int val = rand()%3;
      val -= 1;

      final +=val;
    }
    space[final]++;
  }

  for(int n=0;n<space.size();n++) {
    cout << n << " " << space[n] << endl;
  }

  return 0;
}

Here's the gnuplot generated from the output:

If you have any pointer on the link between Binomial and Gaussian distributions explained in a way a computer scientist can understand please add them below.

Simple XY plot with gnuplot

2009-02-28T16:09:00.000-08:00

This blog post has moved here

LaCie NAS Drive

2009-02-22T12:55:00.000-08:00

I recently purchased a NAS ethernet drive from LaCie. They are pretty cheap these days - about a hundred quid. I'm sick of finding out that I downloaded a particular file to my laptop rather than my desktop or whatever and this seemed like a good solution. It works flawlessly on the MacBook through the LaCie provided software. Less so, with the openSuSe.

I ended up installing everything in yast2 that even tangentically mentioned samba or netbios. I then opened up my firewall to the fullest extent possible, sacrificed two virgins, restarted netbios and samba a couple of times and lo and behold it worked. I then switched the firewall back on with samba and netbios selected as allowed services for the external zone and it seems happy enough with that. No real lesson learned here, just black magic. This is really just a note for other to let them know that it could in theory work.

Mirroring a mediawiki to another server

2009-02-14T13:15:00.000-08:00

There are lots of good guides on the interwebs on how to do this but I wanted to get a few notes down so I remember my preferred/installed method. These instructions are for Debian/Ubuntu and assume the target server doesn't have an existing mediawiki or mysql installation. All this needs to be done as root.

1. Install mediawiki on the target server, install the same version.

2. Delete /var/lib/mediawiki1.7 on the target server.

3. Dump the mysql permissions database on the source server:

mysqldump -u root -pYOURPASSWORD --opt mysql > ~/mysql.sql

4. Copy that file over to the target file. Import it using the following command.

mysql -pYOURPASSWORD -h 127.0.0.1 mysql < ~/mysql.sql

5. Setup a cron job on the source host to dump the mediawiki sql database every night. Type crontab -e to edit the crontab and add the following line:

0 0 * * * mysqldump -u root -pYOURPASSWORD --opt wikidb > ~/wikidb.sql

6. On the target host, add cron jobs to copy across the database dump, the mediawiki config/uploaded files and to import the mediawiki database. Again do crontab -e and add the following:

0 5 * * *  rsync -oglpvtr root@SOURCEHOST:/var/lib/mediawiki1.7 /var/lib > ~/backups/SOURCEHOST_mediawiki.log
0 5 * * *  rsync -vtr root@SOURCEHOST:/root/wikidb.sql ~/wikidb.sql
0 6 * * *  mysql -pYOURPASSWORD -h 127.0.0.1 wikidb < ~/wikidb.sql

Protip: you'll need your ssh keys setup so you can do passwordless authentication.

The Viterbi algorithm

2009-02-14T13:10:00.001-08:00

For your viewing pleasure I present an implementation of the Viterbi algorithm. I implemented this to help me understand the algorithm as presented in "Biological Sequence Analysis" by Durbin et al. The code solves the "occasionally dishonest casino, part 2". The example displays the output as copied from the book as well as it's own calculation so you can verify the results.

Code here

tkz-graph example (A latex graph drawing package)

2009-02-04T03:34:00.000-08:00

This blog post has moved here

PGF and tkz-graph installation in Ubuntu

2009-02-04T02:58:00.001-08:00

Ubuntu likes to keep it's texlive packages at least a year out of date, so if you want to install tkz-graph you need to install it and pgf manually. Create a directory called texmf in your home directory (mkdir ~/texmf). Create a subdirectory called tex (mkdir ~/texmf/tex). Create subdirectories of this called generic and latex. From the pgf tarball copy pgf-2.00/generic into the generic subdirectory. Copy pgf-2.00/latex to the latex subdirectory (you now have ~/texmf/generic/pgf and ~/texmf/latex/pgf. Download the tkz-graph style (tkz-graph.sty) and copy it to ~/texmf/latex. I copied it to ~/texmf/latex/prof I don't think that matters... Next "sudo texhash". That should do it.

Download a bunch of papers from a journal

2009-02-03T07:55:00.001-08:00

Say you want to download a bunch of pdf files from a webpage. You want to do this remotely over ssh so those cool firefox extensions are our of the question. Use wget. For example:

tsocks wget -r -l1 -t1 -N -A.pdf -erobots=off http://www.springerlink.com/content/x49g43t85772/

Does the job.

Openmoko freerunner with vodafone DE sim card

2009-02-03T05:30:00.000-08:00

Just a note to mention for anyone curious that I managed to get a 3G german vodafone sim working with OM 2008.12 by flashing the gsm firmware to moko10. Whilst the wiki advises using shr I found it didn't make any difference. Still have to test this with 3 cards from the UK but hopefully some of them will work now too.

Silc client won~t accept the / (backslash) key

2009-02-03T05:21:00.000-08:00

I use silc as a replacement for things like irc. Most of the time I think its great. It provides secure and simple encryption and allows me to waste time instead of actually working. Well that last bit is a tad unfair as I learn a lot of things on there.

Anyways, for the last while, it has been impossible for me to enter the backslash character in a session. This is regardless of the keyboard layout on the client computer, the OS and was a seemingly random error. Today I erased the .silc/silc.conf file and restarted the silc client. Success - I am now able to connect to the server once again. The most infuriating aspect of this is that the error occured during a running session. When I copied urls into that session none of the slash characters would be displayed on the screen. Which was funny at first but very annoying once I restarted silc only to discover that I couldn't now type /connect. Which was funny for about 30 seconds. So there you go. If you are ever in the situation where you can't enter the slash key in a silc client just remove the configuration file.

Replacing a string in vim to fix an dxf file

2009-02-03T02:14:00.000-08:00

I have an DXF file that has entries that look like this:

LCD.1
70
64
62
150
6
CONTINUOUS
0
LAYER
2
LCDMOUNT.1
70
64
62
252
6

Autocad doesn't like this file... I have no idea what created it. After much screwing around I've discovered that it doesn't like the "."s in the entity (or whatever they're called) names. Load the file in vim and use the following regexp to get rid of the dots.

%s/\([A-Z]\)\.\([0-9]\+\)/\1\2/g

Fixed file looks like this:

LCD1
70
64
62
150
6
CONTINUOUS
0
LAYER
2
LCDMOUNT1
70
64
62
252
6

FYI, the  are delimiters in the search sequence. \1 and \2 reference back to these.

Addition: The file in question was produced by a program called "Camera 3D" or "3D Camera" or something... Just in case anyone else has the same problem with this code.

Merging in Subversion

2009-01-26T07:49:00.000-08:00

Merging in subversion doesn't work like you'd think it'd work. You don't say "take this branch and apply all the changes to trunk". No... you say "calculate the differences between this branch and and older revision then apply these changes to trunk". That's very different. Because of this you need to note the revision at which you're branch was made and use this when merging. Here's a quick example:

Make a branch:

svn copy trunk ./branches/nw3

NOTE THE REVISION NUMBER (basenum) AND LEAVE A USEFUL LOG COMMENT. OK, now make edits commit them etc on the branch. At some point you want to merge the changes in to trunk (run this while in the trunk directory):

svn merge -r basenum:currentrev https://swiftng.svn.sourceforge.net/svnroot/swiftng/trunk

Where basenum is the revision number at which the branch was made and currentrev is the current revision.

Beagleboard GPIO Input (driverless)

2009-01-18T15:34:00.000-08:00

This blog post has moved HERE