Hand-made random seldomly blog-like thoughts
which might be worth entering Google's index
atom

git-svn rulz

Out of a repo with nearly 20,000 revisions:

    [gc@meuh ~/src/soft/trunk.svn] time svn log --revision=5000 2>&1|tail -2|head -1
    0.08user 0.00system 0:22.77elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
    [gc@meuh ~/src/soft/trunk.svn] time svn log --revision=5000 2>&1|tail -2|head -1
    0.07user 0.01system 0:13.99elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k

    [gc@meuh ~/src/soft/trunk.git] time git svn log --revision=5000 2>&1|tail -2|head -1
    0.46user 0.12system 0:00.75elapsed 78%CPU (0avgtext+0avgdata 0maxresident)k
    [gc@meuh ~/src/soft/trunk.git] time git svn log --revision=5000 2>&1|tail -2|head -1
    0.46user 0.11system 0:00.64elapsed 90%CPU (0avgtext+0avgdata 0maxresident)k
  

Transcoding videos from HTC Magic

When shooting a video in "portrait" mode, we need to rotate it, or else using it in blogs or so forth it will be incorrectly displayed. Even if mplayer properly replays the videos, when transcoding, the video is made much too fast. After much investigations, trials and errors, upgrades of ffmpeg and mplayer to latest from cooker, I discovered it's probably due to the video lying about its FPS (or not saving the information). Adding "-fps 17" to mencoder commandline workarounds the problem.

    mencoder sourcefile.3gp -oac mp3lame -ovc lavc -vf rotate=1 -o outputfile -fps 17
  

Watching Kaamelott show on HTC Magic

Recently I've been watching again the Kaamelott DVD's from first book (season) at home - this french hilarious show everyone should own in his DVDtheque. To optmize a bit, I figured out I could transfer some of my DVD's to my HTC Magic to watch while commuting by train.

First step is to rip the DVD and create one VOB file per chapter; I use dvdrip, check "All" in "Specify Chapter Mode" in "RIP Title" tab, then rip.

Second step is to transcode in a format suitable for my HTC Magic; the aspects being slightly different, the default look is not good (too high forms); between adding vertical pads and cropping horizontally, I chose the latter, in order to use the whole screen area; I also raise volume (256 -> 640) because maximum volume on the phone is not enough to overcome train noise.

    ffmpeg -i sourcefile -s "560x320" -vcodec mpeg4 -b 600k \
        -cropleft 40 -cropright 40 -acodec libfaac -ac 2 -ar 32000 \
        -vol 640 -r 13 -ab 64k outputfile
  

If videos sometimes slow down, it might be interesting to install Task Manager to kill background tasks not closed.

Network slowing reminder

It's just a reminder for trickle, the very useful userspace bandwidth shaper.

Nicer text outline in Gimp

Say you want to use the Text Tool in Gimp. Most of the time, an outline in a solid color improves readability and is nice to see. Given your text (zoomed here), naively you could use the Select by Color Tool, grow the selection by a few pixels, and fill with some color in a separate layer, but the result doesn't look so good because the outline edges are too sharp.

Trading the Select by Color Tool for Layer/Transparency/Alpha to Selection, the selection will take advantage of the translucent edges of the text, and the result will look better.

Transcoding videos for my HTC Magic

    ffmpeg -i sourcefile -s "480x320" -vcodec mpeg4 -b 512000 \
        -acodec libfaac -ac 1 -ar 16000 -r 13 -ab 32000 -aspect 3:2 outputfile
  

GTK+ with default X composition

Just upgraded to Mandriva 2010.0, which uses Xorg 1.6.5. I use a qwerty-us keyboard, with the right Windows key configured as the Compose key. It wasn't working anymore and keyboarddrake didn't seem to propose the option, so I had to copy-paste the following section in /etc/X11/xorg.conf:

    Section "InputDevice"
        Identifier "Keyboard1"
        Driver "kbd"
        Option "XkbModel" "pc105"
        Option "XkbLayout" "us"
        Option "XkbOptions" "compose:rwin"
    EndSection

valgrind reminder

I always forget the needed parameter for the useful use of valgrind which is getting the backtrace of segmentation faults together with getting information on memory leaks. So here it is:

    # valgrind --log-file=logfile.out --tool=memcheck --leak-check=full ..binary..

Telling svn to ignore space changes

This one is not magic but I often forget the exact command. The aim is to use the -b switch of the diff command, to ignore changes in the amount of white space. It is useful when commiters reindented stuff.

svn diff -x -b ...

Setting Bash's IFS to newline only

When I need that for i in `cat foo`; do ...; done iterate over each line of the file, I always forget how to set Bash's IFS to newline only, and I never find it immediately with Google, so here's my personal cache.

IFS=$'\n'

Diffing ZIP files

When deploying a new WAR file in production, I like to view exactly what are the changes to the current one; it helps spot what bugfixes/features are actually in the new WAR file, and avoid misleadingly deploying wrong versions or with unwanted changed.

To do so, I've written a script to output diff from ZIP files. It is cool.

PostgreSQL implements POSIX regexp, Java doesn't

For Java's Pattern class, foo{ is an incorrect regular expression, but for PostgreSQL's POSIX regular expression operator ~, it is totally fine.

For Java's Pattern class, foo$* is a perfectly valid regular expression, but for PostgreSQL's POSIX regular expression operator ~, it is incorrect!

Ouch.

Deinterlacing mini-DV with mplayer

I used to use the simplest / most suggested way to deinterlace my mini-DV videos with mplayer e.g. use -vf lavcdeint. After reading some stuff on linuxfr and making some experiments, I came to the following conclusions:

• it's much better to use -vf yadif=0, in particular to obtain sharper images;
• the yadif filter has some magic in it, as it can allow to produce 50 fps videos of quite high quality; try to watch your mini-DV videos in 50 fps, videos feel much more "realistic"..

Some details with screenshots can be found here.

PostgreSQL JDBC: setting an Array value

It's not possible to easily use #setArray on a PreparedStatement object with PostgreSQL's JDBC driver, as no provided class implements the required interface java.sql.Array (and it seems tough to implement it for that use).

There are two handy workarounds (at least). The first is to use an existing SQL ARRAY value: you can craft an SQL query returning an array, then use that value directly with #setArray; in that situation, the first query will look like:

SELECT ARRAY( SELECT lanname FROM pg_language ) AS foo;

The second possible workaround is to use the function string_to_array. For example, you can use e.g. #setString("internal,sql") on a query like:

SELECT * FROM pg_language WHERE lanname = ANY( string_to_array(?, ',') )

Publishing Git repositories on a shared server

So you are the happy admin or user of a shared Linux server. You know that Git is very cool, and you want to install what's necessary on the shared server to host user repositories there. You're like me: you've read some Git documentation, but you just haven't found some sort of executive summary comprehensively listing how to do that. This place may help!

Fun with Google Translate

Here are a series of translations given by Google Translate:

Sadly, from German to French, Madame Dupont isn't returned :/.

Help your Windows friends and increase your karma

By default, Microsoft Outlook doesn't do proper quoting of replies - e.g. each line of reply is not prefixed with "> "; however, it should, to improve readability and allow for faster typing of multiple responses "inline" the replied message: if the replied message is left as is, the person has to manually "prefix" each part of his response with some special text, or manually set a style or coloring; however, when it's already prefixed, it's as easy as moving inside the replied message and type your text on a new line.

It's possible to configure Outlook to perform proper quoting of replied messages.. if you know where the option is buried. I bet that most of our Windows friends would be happy to use more standard and more efficient emailing.

• In the Tools menu, choose Options. On the Preferences tab, click E-mail Options.
• Under When replying to a message, choose Prefix each line of the original message.
• Click OK twice.

Eclipse 3.3.0 and broken Show Tooltip Description key binding

I personally try to use the keyboard as much as possible, opposedly to the mouse, when editing source code: I think it's much faster. When using Eclipse to edit Java, Java errors and warnings are displayed "inline" (as spelling mistakes in a word processor: a curly underline below the offending words); details about each problem can be obtained as a tooltip by hovering the mouse, but also with the Show Tooltip Description key binding. Strangely enough, when upgrading from Eclipse 3.2 to 3.3.0, the key binding stopped working. After a lot of head scratching, with no information on the net about this, I finally found that opening once the Problems window after starting Eclipse workarounds the problem. Quite strange.

Ruby equivalent to Java's synchronized {} block

In Ruby, the base synchronization tool is the Mutex class. You can use its synchronize method and pass it a block which will be guaranteed to not be interrupted by another Thread also synchronizing on this mutex. Unfortunately, the synchronize method doesn't like to be called from a Thread already owning the lock. If you look for something easier to use, as Java's synchronized {} blocks are (in my personal experience, calling/recursing into another method which also wants to synchronize on the same resource is a common use-case), the solution is to use Ruby's Monitor class. It's even possible to mix it in any other class and then synchronize on the object representing the resource which should be synchronized, as we can do with every Object in Java.

    require 'monitor'

    whateverobject.extend(MonitorMixin)
    whateverobject.synchronize { ... }

There *is* an official list of MIME types

My daily work involves dealing with MIME types quite a lot, as I maintain a web application and as I work with MMS data. I've been unhappily questing for a bright light each time I struggle with finding the official/standard MIME type for any content; is there any official/standard list for this pile of ***?; and finally the great Odie found the grail: this actually is the official list of registered MIME types, provided by the IANA. It is so kind to provide RFC links when applicable.

Zattoo on Mandriva

The Zattoo Linux installation procedure is manual. Here are the steps which worked for me:

• download their player, untar it in /tmp (it goes to /tmp/dist)
• download their xulrunner, untar it in /tmp/dist (it goes to /tmp/dist/xulrunner)
• rename all xulrunner shared libs: in /tmp/dist/xulrunner, do for i in *.so; do mv $i ${i}.0d; done
• urpmi libgtkglext-1.0_0
• have data files accessible to the player: sudo ln -s /tmp/dist/usr/share/zattoo_player /usr/share
• start it with (in one line):

    # LD_PRELOAD=/tmp/dist/usr/lib/zattoo/libavcodec.so.51:
                 /tmp/dist/usr/lib/zattoo/libavformat.so.50:
                 /tmp/dist/usr/lib/zattoo/libavutil.so.49:
                 /tmp/dist/usr/lib/zattoo/libfaad.so.0.0.0
      LD_LIBRARY_PATH=/tmp/dist/xulrunner /tmp/dist/usr/bin/zattoo_player

identify sucks for reading EXIF orientation of a JPEG file

My web-album software booh is great, but somehow kinda slow. Most of the slowness comes from XML parsing (since it's done with REXML, a pure Ruby XML parser, which is not too fast) and image resizing (since it's done with convert of ImageMagick, which is not too fast either). Though, there's also the fact that I use identify from ImageMagick to read the EXIF orientation tag of photos; and actually, on my Pentium-4 2.8 GHz, it takes exactly one second to extract the orientation of a 3 MB digital photo! that sucks! I've made a comparison with commandline's exif tool on a directory with 428 photos, and identify absolutely cannot stand:

    # time identify -format "%[EXIF:orientation]" *.jpg >/dev/null
    07:34.42 elapsed
    # time exif --tag Orientation *.jpg >/dev/null
    00:05.73 elapsed

tcpdump reminder

I always forget the needed parameter for a useful probe with tcpdump. So here it is:

    # tcpdump -s65535 -w/tmp/tcpdump.out host blabla.bla and port 22

Then of course, launching wireshark and clicking on "Follow TCP Stream" are a must.

kill -QUIT's brother

It's possible to show interesting GC related information from a running java program with the jmap utility.

    [gc@meuh ~] jmap -heap 12377
    Attaching to process ID 12377, please wait...
    Debugger attached successfully.
    Client compiler detected.
    JVM version is 1.5.0_09-b03

    using thread-local object allocation.
    Mark Sweep Compact GC

    Heap Configuration:
    [...]

Notice that in this information, it's possible to view if you're close to reaching an out of memory because of a too small permanent generation.

Verifying a S/MIME signed email

Openssl's smime utility will do.

    [gc@meuh ~] openssl smime -verify -in mailfile -signer /tmp/signer.pem >/dev/null
    Verification successful
    [gc@meuh ~] openssl x509 -noout -text -in /tmp/signer.pem
    [...]

What threads are running inside my Java application?

kill -QUIT triggers an stdout-based full thread dump. Including a backtrace of all threads, with precise locations where they are currently locked if applicable!

Ruby presentation at Linux Days

This morning, I gave a Ruby presentation (in french) in the LinuxDays event, Geneva. Was kinda cool, room was pretty nice, in a building close to the Palais des Nations of Geneva. About 20 to 30 attendees. The french presentation, made with OpenOffice.org, is available.

Linux + Laptop + Radeon + Beamer = 640x480

Well, at least on the "ATI Radeon 9250 and earlier" of the HP pavilion ze4300 (PCI 1002/4337), when plugging a beamer, Xorg (X Window System Version 1.3.0, Release date 19 April 2007, X protocol version 11, revision 9, release 1.3) says:

    (WW) RADEON(0): Failed to detect secondary monitor DDC, default HSync and VRefresh used

And then nearly all modes are removed because of default low frequencies, and both laptop and beamer start up at 640x480 (laptop screen is 1400x1050). This is not lethal for OpenOffice.org/Impress which will scale slideshows, but for the rest of your applications, you will probably end up pretty much screwed (however, this is ideal for Frozen-Bubble, which is probably the most important).

The solution I found on a forum is to force timings. I am not sure how large you can set, and if there is a possibility to break your beamer, so don't use too large values, but the results of my experiments, with a 4 years old beamer, are (within the Device section of the X configuration file) that the following setting allow 1024x768 output.

    Option "CRT2HSync" "28-50"

When using 60 instead of 50, the device chooses 1280x1024, and when using 70, 1400x1050 (but that's unreadable on the beamer output, of course).

PostgreSQL: VACUUM and VACUUM FULL

Due to PostgreSQL approach to MVCC, the databases need to be regularly VACUUMed and ANALYZEd, else the disk usage and performance will decrease to a crawl. Unfortunately, if databases have not been VACUUMed regularly enough in the past, a new run of VACUUM will not completely recover performance, due to a large number of unused space within actual data. I went to the extend of submitting a documentation patch to make it more clear in PostgreSQL documentation; fortunately, after the sterile troll that occurred in the thread, the patch was accepted upstream by a more sensible guy than the futile trollers who answered in the ML.

Notice that unfortunately, there is no easy way to know if there is a lot to gain from VACUUM FULL prior to trying it - and as it fully locks tables (and the DB connection which is performing the request, even if used for a totally different statement), you badly need to know for large tables. You may:

• on the production database, you can use contrib/pgstattuple. If the free_percent reported for interesting tables is large, and free_space is large compared to 8K, then VACUUM FULL should be very helpful.


• another way is to dump your production database, restore it somewhere, issue VACUUM VERBOSE on a given table on both databases (in production, and on the restore) and compare the reported number of pages needed. The difference is the potential benefit of running VACUUM FULL.

  production_db=# VACUUM VERBOSE table;
      [...]
  INFO:  "table": found 408 removable, 64994 nonremovable row versions in 4395 pages
  
  restored_db=# VACUUM VERBOSE table;
      [...]
  INFO:  "table": found 0 removable, 64977 nonremovable row versions in 628 pages
  

  In the above example, 6/7 of the data is unused. Scanning the table will be 7 times faster (or more - lower data amount means higher cache hit rate) after a VACUUM FULL!

Generating a DVD to play on a DVD device

It's somehow not so simple to generate a DVD to play on a regular DVD device (the one below your TV). There are details to make right, or the device will refuse to play it, jerk sound or things like that. It turned out that I didn't manage to properly generate a DVD with mencoder or transcode, but I could do it with ffmpeg. Here it is:

    # ffmpeg -i source.avi -target dvd -bf 2 target.mpeg

Don't forget to deinterlace interlaced material such as DV with -deinterlace. You can also add -threads 2 if you have a hyperthreading or dual core CPU. Personally, I dislike the little black borders left and right of the image when coming from a DV, so I also use -aspect 4:3.

Of course, I only showed the .mpeg-generation phase. Now you need to author the DVD. Under Linux, most tools suck bigtime. I use qdvdauthor, which looks promising, but which seems to still be unable to easily generate a menu with a video background, and to properly save project parameters between runs.

Java out-of-memory error with plenty of memory left

Happened on me that my web application running inside tomcat received the out-of-memory exception, although it was obvious it didn't run out of memory (java heap space). When looking closer at how the memory is handled, it turns out the GC of the JVM has several locations where objects are put, and classes and methods are actually loaded inside a specific one, the permanent generation. In my case, it turned out that I had a lot of classes loaded, and after some time the default size for the permanent generation (which is 64 MB, and is not altered by setting the heap space with -Xmx) was not enough to hold them all. Why the JVM (1.4.2) is not able to properly hint in the out-of-memory exception in that case is a pity, but life's sometimes full of unexpected things :). The solution was to enlarge the permanent generation with -XX:MaxPermSize.

What are the currently running SQL statements?

In an application using PostgreSQL, it might not be obvious to list the currently running SQL statements, together with the date they started. It is particularly useful if you see in the processes that you have running statements (ps auxw | grep postgres: | grep -v idle), and your machine is loaded, but you don't know where the load is coming from. Set:

    stats_command_string = on

in postgresql.conf, then you may see the running statements with:

    db=# SELECT current_query, query_start FROM pg_stat_activity WHERE current_query <> '<IDLE>';

strncat sucks

Contrary to popular belief, using strncpy and strncat in C programs is not the panacea. While strncpy would probably be still acceptable, strncat is truly braindead. As I could not find a GPL implementation of OpenBSD's strlcat, I wrote my own (for inclusion in the Frozen-Bubble server). So, this is GPLv2:

    size_t strconcat(char *dst, const char *src, size_t size)
    {
        char *ptr = dst + strlen(dst);
        while (ptr - dst < size - 1 && *src) {
            *ptr = *src;
            ptr++;
            src++;
        }
        *ptr = '\0';
        return ptr - dst;
    }

Notice it doesn't check src and dst are not NULL.

UTF-8 Byte Order Mark?

Some sucking Windows software add a BOM in front of UTF-8 files, even if it makes no sense. The BOM in UTF-8 is 0xEFBBBF. As a consequence, you might end up needing to remove such byte sequence in front of UTF-8 text documents you receive from the wild - especially when you feed it to an XML parser, because anything in front of the XML declaration will trigger a parse error.

Guillaume Cottenceau
atom