systemcall dot org

Another normal day here at EBI when I change a variable called GZIP from local to global (via export on Bash) and I got a very nice surprise: all my gzipped files have gzip itself as a header!!!

Let me explain… I have a makefile that, among other things, gzip some files. So, I’ve created a variable called GZIP that is the same as “gzip –best –stdout” and on my rules I do:

%.foo : %.bar
$(GZIP) < $< > $@


So far so good, always worked. But I had a few makefiles redefining the same command, so I though: why not make an external include file with all shared variables? I could use the @include for makefiles but I also needed some of those variables for shell scripts as well, so I decided to use “export VARIABLE” for all make variables (otherwise they aren’t caught) and called it a day. That’s when everything started failing…

gzip environment

After a while digging the problem (I was blaming the poor LSF on that) I found that when I hadn’t the GZIP variable defined all went well, but by the moment I defined GZIP=”/bin/gzip –best –stdout” even a plain call to gzip was corrupted (ie. had the binary gzip as a header).

A quick look on gzip’s manual gave me the answer… GZIP is the environment variable that gzip stores all default options. So, if you say that GZIP=”–best –stdout”, every time you call gzip it’ll use those parameters by default.

So, by putting “gzip” on the parameter list, I was always running the following command:

$ /bin/gzip /bin/gzip --best --stdout < a.foo > a.bar

and putting a compressed copy of gzip binary together with a.foo into a.bar.

What a mess can a simple environment variable do…

Popularity: 11% [?]

Share This

Long way ago, when I started using Linux, lots of people laughed at me: “What an absurd! You have to compile your own kernel, what do they want with that? They’ll get nowhere!”. Well, things have changed a bit in the last decade and Linux grew up as a very mature, modern and user-friendly operating system as we (not them) all expected.

OS companies didn’t believe at start but with time Linux became a nuisance, than a problem and now it’s real competition. Not only Linux (or rather GNU/Linux) but all free software and all the free licenses like GPL, FreeBSD, CC, etc. Linux is real business, it’s more stable, faster, better designed and change so much faster than any other OS in existence both for security patches and new features. Lots of companies today contribute to free software without charge or restrictions, just because we gave them so much without charge or restrictions (and it turns out as profit too!).

But last year something I wasn’t expecting happened… The biggest OS company for the last 15 years did a move so stupid that I couldn’t believe. Windows Vista was not an operating system, it was a joke, a *very bad joke* indeed. It reminded me the first upgrades of the first Linux distros back in 94, it was a nightmare.

Well, seems like the free software community learnt a lot about deployment, user interfaces, quality assurance, software development strategies. On the other hand, Microsoft seems a bit amateurish when trying to fix the previous mistakes. Every round it gets worse, I wonder where the good programmers they use to have are now…

Well, better for us, Ubuntu seems to be the new OS of choice for many previous Windows users and with recent Microsoft moves it may become more and more often… Luckily they’ll force everyone out of XP (the last minimally decent thing they did) as they did to Win2000 (the only reasonably decent thing they did) and people will migrate to Ubuntu instead of Vista… Let’s see the outcome by next year…

Popularity: 14% [?]

Share This

Recently I’ve posted this entry about how NFS cache was playing tricks on me and how sleep 1 kinda solved the issue.

The problem got worse, of course. I’ve raised to 5 seconds and in some cases it was still not enough, than I’ve learnt from the admins that the NFS cache timeout was 60 seconds!! I couldn’t sleep 60 on all of them, so I had to come with a script:


timeout=60
while [ ! -s $file ] && (( $slept < $timeout )); do sleep 5; slept=$(($slept+5)); done


In a way it’s not ugly as it may seem… First, the alternative is to change the configuration (either disable cache or reduce timeout) in the whole filesystem and that would affect others. Second because now I just wait for the (almost) correct amount of time and only when I need (the first -s will get the file if there is no problem).

At least, sleep 60 on everything would be much worse!

Popularity: 9% [?]

Share This

Multics arose from the dead in the source code shape! MIT has just released its source and now you can see with your own eyes how it was back in ‘64!

It’s not easy to retrieve the whole code (no tarballs) but it’s a good exercise to read its parts if you can understand the structure, of course. If you couldn’t, don’t worry, start here.

Popularity: 8% [?]

Share This

I use LSF at work, a very good job scheduler. To parallelize my jobs I use Makefiles (with -j option) and inside every rule I run the command with the job scheduler. Some commands call other Makefiles, cascading even more the spawn of jobs. Sometimes I achieve 200+ jobs in parallel.

Our shared disk BlueArc is also very good, with access times quite often faster than my local disk but yet, for almost two years I’ve seen some odd behaviour when putting all of them together.

I’ve reported random failures on processes that worked until then and, without any modifications, worked ever after. But not a long time ago I figured out what the problem was… NFS refresh speed vs. LSF spawn speed using Makefiles.

When your Makefile looks like this:

bar.gz:
$(my_program) foo > bar
gzip bar


There isn’t any problem because as soon as bar is created gzip can run and create the gz file. Plain Makefile behaviour, nothing to worry about. But then, when I changed to:


bar.gz:
$(lsf_submit) $(my_program) foo > bar
$(lsf_submit) gzip bar


Things started to go crazy. Once every a few months in one of my hundreds of Makefiles it just finished saying:


bar: No such file or directory
make: *** [bar.gz] Error 1


And what’s even weirder, the file WAS there!

During the period when these magical problems were happening, which I was lucky to streamline the Makefiles every day so I could just restart the whole thing and it went well as planned, I had another problem, quite common when using NFS: NFS stale handle.

I have my CVS tree under the NFS filesystem and when testing some perl scripts between AMD Linux and Alpha OSF machines I used to get this errors (the NFS cache was being updated) and had to wait a bit or just try again on most of the cases.

It was then that I have figured out what the big random problem was: NFS stale handle! Because the Makefile was running on different computers, the NFS cache took a few milliseconds to update and the LSF spawner, berzerk for performance, started the new job way before NFS could reorganize itself. This is why the file was there after all, because it was on its way and the Makefile crashed before it arrived.

The solution? Quite stupid:


bar.gz:
$(lsf_submit) "$(my_program) foo > bar" && sleep 1
$(lsf_submit) gzip bar


I’ve put it on all rules that have more than one command being spawned by LSF and never had this problem again.

The smart reader will probably tell me that it’s not just ugly, it doesn’t cover all cases at all, and you’re right, it doesn’t. NFS stale handle can take more than one second to update, single-command rules can break on the next hop, etc but because there is some processing between them (rule calculations are quite costy, run make -d and you’ll know what I’m talking about) the probability is too low for our computers today… maybe in ten years I’ll have to put sleep 1 on all rules…

Popularity: 10% [?]

Share This

SciCortex is to launch their cluster-in-a-(lunch)-box with promo video and everything. Seems pretty nice but some things worries me a bit …

Of course a highly interconnected backpane and some smart shortest-path routing algorithms (probably not as good as Feynman’s) is much faster (and reliable?) than gigabit ethernet (myrinet also?). Of course, all-in-one chip technology is much faster and safer and more economic than any HP or IBM 1U node money can buy.

There are also some eye-candy like a pretty nice external case, dynamic resource partitioning (like VMS), native parallel filesystem, MPI optimized interconnection and so on… but do you remember Cray-1? It had wonderful vector machines but in the end it was so complex and monolithic that everyone got stuck with it and never used it anymore.

Assembling a 1024-node Linux cluster with PC nodes, Gigabit, PVFS, MPI etc is hard? Of course it is, but the day Intel stops selling PCs you can use AMD (and vice-versa) and you won’t have to stop using the old machines until you have a whole bunch of new ones up and running transparently integrated with your old cluster. If you do it right you can have a single cluster beowulf cluster running alphas, Intel, AMD, Suns etc, just bother with the paths and the rest is done.

I’m not saying it’s easier, nor cheaper (costs with air conditioning, cabling and power can be huge) but being locked to a vendor is not my favourite state of mind… Maybe if they had smaller machines (say 128 nodes) that could be assembled in a cluster and still allow external hardware to be connected having intelligent algorithms to understand the cost of migrating process to external nodes (based on network bandwidth and latency) would be better. Maybe it could even make their entry easier to existent clusters…

Popularity: 11% [?]

Share This

It has been a while since Linux became more popular among the non-hackers. Projects like Ubuntu, OpenOffice and Firefox really brought the spotlight to Linux as just another option for your desktop, but it’s hard to compete with money-driven brands such as Apple’s Macs. Not anymore!

According to the register, “Versions 2.3 [of the OpenOffice] is available for Windows, Linux, and Solaris, but Mac mavens won’t get an update until sometime next year” [source]. Funnily enough, IBM’s symphony (their office suite) which also uses the same Open Document format as OpenOffice “is available free of charge to Windows and Linux users” [source].

So, now… what about Mac users? Why should they have to wait that much more to have something that is clearly multi-platform and intended to anyone in the world, for free? Is it that more difficult to develop Mac applications? Well, thank god Linux is the best development environment and it may not touch you (if you’re a non-hacker) but the benefits are showing up these days…

It’s time for non-hackers to start thanking us with our complicated Linux boxes because they will be using them (much simpler than we did) in the very near future.

Popularity: 9% [?]

Share This

The first big favour Microsoft did to the Linux community was to delay Windows Vista over and over (and over) again. That put into highlight some Linux distributions such as Ubuntu. Not that it was the only reason, of course but it did help a lot. Ubuntu got many deals, in special with Dell, selling per-installed Ubuntu systems to the general public.

Now it’s even better! They’re (thinking of) using a black screen of death security system that if your copy is not legal it’ll lock your computer and die. Pretty much what used to happen with the previous blue screen of death, only intentional.

It’s a mighty gift, much better than the One Ring to Boromir, it’ll force all people that use pirate copies of Windows (the vast majority) to buy legal copies or crack the system.

Now let’s be serious, how many people you know will actually buy a real copy? If you live in the US, most. In Europe? Some. In the third world countries? One or two… A hack will eventually be available but the more they force people out of piracy the more people will be using Linux, and that’s the best scenario for us, freedom supporters!

They are so dependent in piracy (and they know it) that the less piracy there is, less market they gain, less people the get brainwashed, less of a monopoly the industry becomes, the more freedom we all have… It’s a bless!

Again, thank you very much!

Popularity: 17% [?]

Share This

Wandering around my Linux filesystem I found a weird directory in /home …


drwxr-xr-x 2 root root 4096 2007-08-19 12:03 eb588afc0325b12eeb074fd6


Ok, I thought, I didn’t create that. If it’s a virus, it’s the most stupid virus in existence, but, we never know… Then I got inside and see what files it had, and found this:


$ l eb588afc0325b12eeb074fd6/
total 956
-rw-r--r-- 1 root root 865822 2007-08-02 21:41 mrt.exe._p
-rw-r--r-- 1 root root 96216 2007-08-02 21:34 mrtstub.exe
-rw-r--r-- 1 root root 45057 2007-08-19 12:03 $shtdwn$.req


Mamma mia, if it really is a virus, it’s even more stupid trying to put .exe files in my Linux box! Anyway, The Oracle would know the answer… Searching for mrtstub, the first hit is this page, directly from the enemy’s site. Not too far I found the origin:

mrtstub is part of the Malicious Software Removal Tool. It is responsible
for copying mrt.exe to the correct location and launching it.

Long story short: I have dual boot (which I never use but my son plays sometimes) and my Linux home directory is mounted using an ext3 driver for Windows. Microsoft asked me to install this Malicious Software Removal Tool which I denied 10 times asking every bloody time NEVER TO INSTALL IT IN THE FUTURE until the 11th was my son that wasn’t even asked but turned it off as he always do and Microsoft stealthily installed this piece of crap in my computer.

That’s enough, I’ll spend a fiver and buy a cross-over software to run my son’s games on Linux and remove this crap out of my computer once and for all.

Popularity: 100% [?]

Share This

The first editor I’ve used on Unix was VI. Since then, I’ve been using lots of different editors for both code and text files but I still can’t find a replacement for VI.

VI, now called vim, is the most powerful and simple editor in existence (Yes! Emacs users, it *is* simpler than Emacs). Of course, there are simpler or more powerful editors around but not both. At that time (early 90’s) VI wasn’t so complete and powerful but it was simple and widely available on Unix world and that’s what made it famous.

But before using VI for coding, I used Borland’s fantastic Turbo C (for DOS) and the need for a smarter IDEs was something I always had in mind. It began, then, the search for a TC-like IDE. Borland made later several great IDEs for Windows but once coding on Unix it’s very hard to turn back and code on Windows, so I had to find a good IDE, for Linux.

Early tries

After coding for so long in VI I was feeling like it was a natural choice to use VI every time I wanted to edit a file, whatever it was. I never bothered to find other text editors (such as joe or emacs) but I did use a bit of pico (later nano) and it was terrible.

When Gnome and KDE came to substitute WindowMaker they came with lots of text editor but they were, after all, notepad clones. Later they became a bit better but still not as good as VI so, why bother change?

Well, one good reason to change was that, every time I need to edit a file I had to go to the console and open the VI. That was not such a bad thing because I always have a console open somewhere and navigating through the filesystem is easier anyway, but a few times it was annoying and I used Kate (from KDE, my WM of choice). Anyway, it was around that time that VI gained a nice brother, gvim: the graphical editor! One reason less to not use VI.

Kate was really good in fact but I found out that I had lots of “:wq” (the command to save and close VI) on my files when using any other editor. I also tried to use Quanta for HTML but it was so cluttered and I had so much “:wq” on my pages that I just gave up.

Java?

When I started programming in Java I found out the Eclipse IDE. A fantastic tool with thousands of features and extremely user friendly editor and all gadgets that a coder would want to have! And it was free and faster than any other Java IDE available at the moment. And it was free! too good to be true?

Nah, for the Java community it was *that* good, but for the rest of us it was crap. The C++ plug-in was (and still is) crap, as well as the Perl plug-in. It didn’t understand classes, inheritance and most important, didn’t have all nice features as for Java for refactoring and understanding the code.

So, why use a gigantic (still fast) IDE that doesn’t speak your language? If it’s not to speak the same language I very much prefer VI! So I went back, once again. Also, by that time, VI got a wonderful feature: tab-completion (CTRL-N in fact).

KDeveloper

The most promising rival is KDeveloper and it’s almost as good as I wanted to be, but not quite enough. It have CVS integration (not much easier as using the console), class structure information, integrated debugger, etc etc etc. But, it’s still a bit heavy (as expected) and not useful for all development projects.

VI re-birth

For a while I only used VI at work and for text files at home, specially while I was busy trying all possibilities of KDeveloper, and that’s because I still missed one very important feature of an IDE that VI didn’t have: tabs.

Editing with tabs is so much simpler than switching buffers or splitting windows. That’s why I revisited Kate a few times later than have abandoned it and that’s why I didn’t use much VI for a long time in my personal projects.

But than VI 7.0 came out, with lots of improvements and the long wanted tab support. It was like one of those amazing sunsets in the country with birds singing and all that stuff. Also, the tab-completion (still CTRL-N) is really smart, it understands includes, class, defines, typedef, everything and have a very simple interface to use.

VI, or now vim is complete! And I’m happy!

Thanks Bram Moolenaar for this amazing piece of software!

Popularity: 4% [?]

Share This

« Previous entries