Sum weights

I have a file with a bunch of sequences and some weights at the top of the file:

>WEIGHTS 0.926434 1.000000 1.000000 0.926434 1.000000 0.892712 1.000000 1.000000 1.000000 1.000000 1.000000 0.892712 
>CRTC_EUGGR__Q9ZNY3 Calreticulin precursor.
XRKELWXXXXXXXXXXXXXXXXXXXXXXXXXTRWTHSTXXSDYXKFKLTSGKFYGDKAKDAGIQTSQDAKFYAISSPIASXXSXEXXXLVLQFSVKHXXXXXXGXGXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXKXEPRCEXDTLSHTYXAXXXXDXXXEVLVDQVKKESGTLEEDWEILKPKTIPDPEDKKPADWVDEPDMVDPEDKKPEDWDKEPAQIPDPDATQPDDWDEEEDGKWEAPMISNPKYKGEWKAKKIPNPAYKGVWKPRDIPNPEYEADDKVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXFYDQTNGATKDAEKKAFDSAEADKRKKEEDERKKQEEEEKKTAEEDEXXXDEXXXEDDKKDEL
>HSP47_RAT__P29457 47 kDa heat shock protein precursor (Collagen-binding protein 1) (GP46).
XRSLXXXXXXXXXXXXXXXXXXXXEAAAPGTAEKLSSKATTLAEXSXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXKXXXXSQAKAVLSAEKLRDEEVHTGLGELVRSLSNXTARNVTWKLGSXXXXXXXXSFADDFVRSSKQHYNCEHSKINFRDKRSALQSINEWASQTTDGKLPEVTKDVERTDXXLLXXAMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYXYXXXXXXXXQXVEMXXXXXXXXXXXXXXXXXXXXXRLEKXXTKXQLKTWMGKMQKKAXXISLPXGVVXVTHDLQKXXAGLGLTEAIXKNKADLSXXSGXXXXXXXXXXXXXXXEWDTEGNPFDQDIYGRXXXRSXXXXXXXXXXXXXXXXXXXXXXXXIGRLXXXXGDKMRDEL
>ENPL_PIG__Q29092 Endoplasmin precursor (94 kDa glucose-regulated protein) (GRP94) (GP96 homolog) (98 kDa protein kinase) (PPK 98) (ppk98).
XRAXXXXXXXXXXXXXXXXXXXXEVDVDGTVEEDLGKSREGSRTDDEVVQREEEAIQLDGLNASQIRELREKSEKXAFXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXELTVKXKCDKEKNLLHVTDTGVGMTREELVKNLGTIAKSGTSEFLNKMAEAQEDGQSTSELIGXXXXXXXXXXXXXXXXXVTXXHNNDTQHIWESDSNEFSVIADPRGNTLGRGTTITLXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXSXKTETVEXPMXXXXAAKXEKEESDDEAAVXXXXXEKXPXTXKVEKTVWDWELMNDIKPIWQRPSKEVEDDEYKAFYKSFSXXXXXPMAYIHFTXXXXXXXXXILXXXXXXXXXLFDEYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXLNVSREXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXGVIXDHXXXXRLAKLLRFQSSHHPSDITSLDQYVERMKEKQDKIYFMAGSSRKEAESSPFVEXXXXXXXXXXXXXXXXXXXXXQALPXXXXKRFQNVAKEGVKFDESEKSKENREAVEKEFEPLLNWMKDKALXDKIEKAVVSQRXXEXXXXLVASQYGWXGNXERIMKAQAYQTGKDISTNYYASQKKTFEINPRHPLIRDMLRRVKEDEDDKXXSDLXXXXXXXXXXXXXXLLPDTKAYXXRIERMLRLSLNIDPDAKVEXXPXXXPXXTTEDTTEDTEQDDDEEMDAGADEXXQXTSETSTAEKDEL
>CRTC_HUMAN__P27797 Calreticulin precursor (CRP55) (Calregulin) (HACBP) (ERp60) (grp60).
XLXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTSXXIESXXXSDFXXFVLSSGKFYGDEEKDKGLQTSQDARFYALSASFEXXSXXXXXLVVXFXXKHXXXXXXGGGYVKLFPNSLDQTDMHGDSEYNIMFGPDIXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTYEVKIDNSQVESGSLEDDWDFLPPKKIKDPDASKPEDWDERAKIDDPTDSKPEDWDKPEHIPDPDAKKPEDWDEEMDGEWEPPVIQNPEYKGEWKPRQIDNPDYKGTWIHPEIDNPEYSPDPSIYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXAYAEEFGNETWGVTKAAEKQMKDKQDEEQRLKEEEEDKKRKEXXXAXDKEDXEXKXEDXXDXXDKXXDXXEDVPGQAKDEL

I want to sum the weights which is fine for this example with like 12 sequences. However, some of the files have a couple of hundred entries. Step in Bash:

 head -n 1 filename.txt | awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'

Giving me the sum of the weights! Woohoo.

Find and delete

Someone didn't appreciate the existence of queueing systems like PBS and rolled their own queueing system that is supposed to take care of removing dead jobs from an app server. So I had to look for the old files that were causing the problem and see if I could delete them. Thankfully this is really easy with a simple find command.

 find ./ -type f -mtime +10 

Switching the -type to d will do the same for directories. Unfortunately there is no clever hack for beating people who do this kind of thing over the internet.

Silc client won~t accept the / (backslash) key

I use silc as a replacement for things like irc. Most of the time I think its great. It provides secure and simple encryption and allows me to waste time instead of actually working. Well that last bit is a tad unfair as I learn a lot of things on there.

Anyways, for the last while, it has been impossible for me to enter the backslash character in a session. This is regardless of the keyboard layout on the client computer, the OS and was a seemingly random error. Today I erased the .silc/silc.conf file and restarted the silc client. Success - I am now able to connect to the server once again. The most infuriating aspect of this is that the error occured during a running session. When I copied urls into that session none of the slash characters would be displayed on the screen. Which was funny at first but very annoying once I restarted silc only to discover that I couldn't now type /connect. Which was funny for about 30 seconds. So there you go. If you are ever in the situation where you can't enter the slash key in a silc client just remove the configuration file.

Disk space usage

My VM keeps getting filled up, and then crashing cos the programs that write logs are not being correctly handled by the underlying FS. So I am interested in generating useful infos on the disk to find out where all the disk space is at. The command du is a big help here. But I frequently just want to know in which directories the most space is being used.

du -h --max-depth=1

That command will give you a listing of which directories under the current dir are using all the space. So in the home dir or at the usr dir run this and find out where all the unneccessary crap is.

Giving me in the /usr dir the following output:

1.1G    ./share
40K     ./games
1.4G    ./local
7.1M    ./sbin
125M    ./src
36K     ./X11R6
136M    ./bin
14M     ./include
1.6G    ./lib
4.3G    .

And this tells me that I ought to tidy up the /usr/local dir. If you ran the above command without the max-depth option then the resulting output will probably overrun your cache/history in your bash shell and besides the output would be impossible to make sense of.

Count chars

We might have had this one before but I couldn't find it.

cat commafile.txt | tr -dc ',' | wc -c

That could be replaced by any file in the commfile.txt. So, basically count the number of occurrences of the expression ',' in the file and print it out. Why? you ask - cos I gotta.

Nokia 6630 & Obextools

How to get your stuff off of your nokia 6630 onto your linux computer. I realise its now an old phone and probably no-one cares but I thought I'd stick it up anyway. Its a decent phone, worth picking up second hand.

Get Obextools. I managed to get a few of the other tools like Kmobiletools working with the phone but they couldn't actually do a while lot.

Add the following to your apt sources.list

#Obex/ Mobile Phone stuff
deb http://www.stud.uni-karlsruhe.de/~ubq7/debian extra main
deb-src http://www.stud.uni-karlsruhe.de/~ubq7/debian extra main

Then run apt-get and install the following:

apt-get install tcllib tablelist bwidgets obextool

You may have to get tablelist separately. I got my copy from the ubuntu forums as a .deb package and installed it separately.

Now run obextools:

obextool --obexcmd "obexftp -u 1"  

I had to run this as root for reasons that are beyond me. I guess its probably some sort of group permission thing for my user account. But I had accessed the obexftp fine using the command line, so I'm not entirely sure.
Anyway its working now - so that I can download pictures from the phone. Woo hoo.

useful bash shortcuts

ctrl-a: jump to beginning of line
ctrl-e: jump to end of line
alt-f: jump forward a word
alt-b: jump back a word

alt-d: delete word
alt-t transpose two words

ctrl-r: search back for a command, hit ctrl-r again to search back further
ctrl-xx: jump back to your last edit, again to get back to original position

For ctrl and alt commands press control AND the key indicated at the same time. More random bash here:
http://blog.webhosting.uk.com/2007/04/08/using-bash-shell-shortcuts/

Extracting part of a field in awk

I had a file where the lines look like this:

all_initstring0010010100111000/1dca.rule118.iter100.score all_initstring0010010100111000/1dca.rule140.iter100.score: 0
all_initstring0010010100111000/1dca.rule128.iter100.score all_initstring0010010100111000/1dca.rule122.iter100.score: 122312
all_initstring0010010100111000/1dca.rule113.iter100.score all_initstring0010010100111000/1dca.rule143.iter100.score: 3213

I wanted to extract the value after rule and before the . in the 1st and 2nd fields and also print the third field. I used awk and the substitution function to replace everything but the required value using a regular expression. Here's the code:

gawk '{gsub(/^.*rule/,"",$1); gsub(/[^0-9].*/,"",$1); gsub(/^.*rule/,"",$2); gsub(/[^0-9].*/,"",$2); print $1 " " $2 " " $3}' myfile

There is actually another solution which is probably nicer:

awk '{ split($1,a1,/\./) ; split($2,a2,/\./); print substr(a1[2],5), substr(a2[2],5), $NF; }' myfile 

eps2latex

quick and dirty bash script to create a latex document from a bunch of eps files.

#!/bin/bash

echo "\documentclass{article}"
echo "\usepackage{graphicx}"
echo "\begin{document}"

for file in $1/*.eps
do
  echo "\begin{center}"
  echo "\begin{minipage}{\textwidth}"
  echo "\includegraphics{$file}"
  echo "\end{minipage}"
  echo "\end{center}"
  echo "{\footnotesize $file}"
done

echo "\end{document}"

Run preview easily from bash on Mac OS X

add the following to your .profile:

preview(){
    open -a /Applications/Preview.app/ "$1"
}

Remote browsing

Say your trying to access regionalised content or may be a journal a different university has access to and you happen to have a box on that remote network, how do you access websites as if you were coming from there network?

There are a few solutions:

1. Just run firefox over an X session. ssh will setup your environment variables making this straight forward. Simply do: ssh my.computer.com -X -l username (unfortunately this is usually very slow)

2. Run lynx! Because text based browsing rocks! (no, no it actually sucks ass)

3. Run a proxy on the remote workstation on a local port (such as squid). Then use ssh port forwarding to redirect the proxy to a local port for example: ssh -L 3128:localhost:3128 my.computer.com -lusername. Then reconfigure your webrowser to use a proxy at localhost port 3128. All done!

4. Use ssh remote port port forwarding to forward port 80 on a specific host to a port on your local computer. Unfortunately this means you can only browse that host (basically this sucks)

Right now those are all the solutions I know of. It would be great if you could forward all outgoing connections from a given application using ssh port forwarding, but as far as I can see... you can't. Another nice solution would be to use a simple user mode proxy server, if anyone konws of one leave a comment.

powerTop

Really just a link to this site:
http://www.linuxpowertop.org/powertop.php

Worth a look though.

Counting the number of instances of a letter in a file

To explain why this might be necessary:

First I have an xml file with some sequences in it. I grep the xml file finding the instances I am interested in and save these to a file, stripping out the xml using vim.

grep -A 5 \<type\>Dis filename.xml | grep \<sequence\> > file_raw_seq.txt

Alternatively I could use the fasta file and strip the headers using grep -v \>.

I am now left with the raw text of the sequences from the xml file that I am interested in.

I can now run the following command to check out the number of times say M appears in the file;

tr -dc M < input.fa | wc -c

Easy. The alternative was to write a C++ program to do all this - which would have taken considerably longer. Especially given that some people haven't committed working versions of their code. I'm looking at you Mr. TreeCreate and Mr IntVector.

Sticky

Further to the back up stuff I been documenting recently, I was told of this useful (?) little titbit. I have a spare disk called /dataStore that is mounted locally on my workstation. Other people log in to my computer to compile stuff, maybe run some stuff etc. So I am letting them back their stuff up locally on my computer. In order to give them the appropriate permissions on the data disk I have made a directory in this disk called imaginatively enough backup.

I have made this directory group writable so that everyone can use this disk. However, in order to ensure that no one fucks up anyone else's data I have made the directory a sticky. To do this type:

chmod +t dir

This is as close to an explanation as I could find:
http://www.uwsg.iu.edu/UAU/files/sticky.html

ssh bash completion in Ubuntu

SSH bash completion doesn't currently work in 6.10. This is because hostnames are now stored as hashes in ~/.ssh/known_hosts. To disable storing hostnames as hashes and allow ssh hostname completion to work correctly you need to edit: /etc/ssh/ssh_config and change HashKnownHosts to No.

Thanks to Ubuntu forums for the answer.

Set up subversion

If you are comfortable with CVS - changing over to svn shouldn't be too hard. First make sure you download the latest version of svn from your package manager.

Then to create the new svn repository type:

svnadmin create /foo/bar/repo

Then to check it out, you need to create a working directory and know the url of the svnserve. In my case this is a local server running on just my machine so the following will work:

mkdir svndir

svn checkout file:///foo/bar/repo svndir

You can now move into the svndir and start working.

Its probably well worth reading up on the differences between cvs projects and svn directories. Its probably easiest to think of creating a new svn repository for what would have been a cvs project. Also its worth bearing in mind that changes in a svn repository tag the whole project rather than a single file.

You don't exist, go away!

This post has moved: HERE

Information about your session

Tips for finding out about your permissions and rights in a session, either on your own computer or a server.

whoami: Displays the username that you are currently logged in as.

who: Displays a list of users currently logged in to the computer. Useful for getting the ip address of the computer you have logged into from.

ifconfig: Displays the ip address of your computer

groups: Displays group membership for the current user.

echo $ENV_NAME: Display the setting for the environment setting ENV_NAME. For example to check which cvs server/root you are currently using type:

echo $CVSROOT

This will display the cvsroot information you are currently using. (CVS commands to follow in a new post)

uname -r

Displays the current kernel version.

More to come here.

Default editor

To change the default command line editor from say vim to nano/pico enter the following in the .bashrc file in your home directory.

export EDITOR=/usr/bin/pico

or you can do this by issuing a setenv command during a session.

eg:

setenv EDITOR=/usr/bin/pico