Extract links from web pages

One or two weeks ago, I tried to download some videos from a website under Linux. Of course it was not possible to simply download the videos using the browser - the videos were secured. A challenge for a hacker. ;) After a quick search in the Ubuntu Software Center I found get-flash-videos. It is a small Perl command line program, capable for retrieving flash movies from a wide variety of movie sites. But sadly it just worked one or two times for me. On the other day, it tried the program, it stopped working. And so began the search for a solution...

For Windows I am using URL Snooper 2 from Mouser software. It is a great utility. A very powerfull tool in combination with a download manager like the Firefox extension down them all. You can get every video from the web with ease.

But I did not find any tool like URL Snooper for Linux and so - I searched the web for some "inspiration". And finally - I found ngrep. It is an command line tool - a bit old (not maintained since 2006) - but easy to use.
Combined with some Perl commands and "down them all" it is quite powerfull. Not as easy and powerful like URL Snooper, but good enough to download the movies I wanted.
And the best: the command fits into one single line. ;) (okay, if you have a big display, where 177 chars will fit... but it is one line). Of course it could be optimized further and some chars could be saved.

Because this code is for the extraction of movies from an adult website, I will not post the link here. ;)
Here is the "proof of concept":

sudo ngrep -n 1 -q -d eth0 '(/key.*flv).*Host: ([\d.])' 'tcp and dst port 80' | perl -ne 'm/.*(\/key.*flv).*Host: (\d+.\d+.\d+.\d+).*/; if ($2){ print "http://$2$1\n"; exit; }'

Take it as a source for your inspiration. Try to figure out what the line of code is doing. Ngrep and Perl can be your friends!

Notes/wish list/TODO's...

... for me.
Some of these projects will be posted here in some weeks or months - i hope.

Finished projects:

• uploading the executable and source of "bool_solver 2"
• "The (dirty) 7zip password cracker" with exe and source
• brute forcing [code snippet]

Unfinished projects:

• Presentation of the project "Dieselpunk PC"
• Some words about the Open Logic Sniffer
• Creating a VBS (Video Baseband Signal) with an 8051 (+ Open Logic Sniffer)

Future projects:

• Reverse engineering the Canon EF protocol with the Open Logic Sniffer

Creating random numbers on a µC [code snippet]

This is a small C function, to feed srand() with a random seed. I wrote it some weeks ago just to test if it would be possible to get random seeds with an 8051 microcontroller without additional hardware. And yes, it is possible. I was using an AT89C51ED, but the code should work with every 8051 compatible microcontroller.

How does it work and why? I took a closer look in the data sheet of the µC and i saw that some bits and bytes in the special function registers (SFRs) are not initialized with "1" or "0". Some bits were marked with "x" and those are the interesting ones. The value of those bits can be "0" or "1" after power up - their initial value is random. All i had to do was to iterate over all those SFR-bytes and XOR them. I am sure generations of 8051 coders have had the same idea, but i did not study all available information regarding 8051 and random numbers.

Just call the function like that "srand(getRandom());", and you will have a new random seed, after each power up of the microcontroller. (A normal µC reset will not give you a new random seed!)

unsigned char getRandom( void )
{
     unsigned char *random;
     unsigned char k;
     unsigned char value;

    random = 0x80; //Starting address of the SFRs

    for (k=0; k <= 127; k++) //iterate over the 128 bytes of the SFRs:
        value ^= random[k]; //XOR the values
    return value; //return the random seed
}

Parsing source code files for function headers and function calls

Some weeks ago I wrote a small program, which is able to print a list of functions and function calls used in a program. It can parse a directory recursively for a given file type.

Here is the source code: functiondiagram.d
And here is the source as a "quote" (a function to present formated source code is missing on blogger.com...):

/*
    This console apllication written in D is able to parse C, C++, D and many more file types.
    It scans the files for functions and function calls and prints a list with all matches for every file.
    As first parameter you must specify a directory name, where the search will start. This folder will be scanned recursively for matches.
    The type of file must be specified by the second parameter.
  
    This code stands under the do-with-it-what-you-want license.
  
    Author: Energized
    E-mail: undervoltage@safe-mail.net
    Blog: http://electric-handicraft.blogspot.com
*/


import std.file;
import std.stdio;
import std.stream;
import std.regexp;

/*
TODO:
- Self defined data types are not recognized
- Comments will be parsed too. This can result in false "positives".
*/


/*
    Function is searching in a line of text/code for function calls and function headers:
*/
string[] returnFunctions( string data )
{
    string regex = r"([\w\d]+)\s*[(]";
    string[] rv;
  

    foreach( m; RegExp( regex, "ig" ).search( data ) )
    {
        if ( (m !is null) && (m.match(1) !is null) ) //is something was found:
        {
            //writefln("> '" ~ m.match(1) ~ "'");
            if ( RegExp( r"^(if|for|foreach|return|while|finally|try|catch)$" ).match( m.match(1) ) is null ) //is the match a reserved word?
            {
                rv ~= m.match(1); //if not, it will be part of our result

                //if no datatype is in front of the match, it is a function call:
                auto n = search( m.pre, r"^\s{0,}(string|bool|int|byte|char|long|short|float|double|void|signed|unsigned|HBITMAP)", "i" );
                if ( n is null )
                    rv ~= ";";
                else
                {
                    /*
                        If an assignment is done (function to variable), false positives will be produced.
                        Then the function call will be misinterpreted as a function header. Thats why we are searching for a
                        semicolon here. Is a semicolon in this line, it is a function call (or a forward declaration).
                    */
                    if ( std.string.find( m.post, ';' ) != -1 ) rv ~= ";";
                }
            }
        }
    }
    return rv;
}


/*
    Shows the result:
*/
void printTree( string[] data )
{
    if (data.length == 1)
        writefln( "\nFunction \"%s()\":", data[0] );
    else
    {
        foreach( t; data )
            if (t != ";") writefln("\t%s();", t);
    }
}


void main( char[][] parameters )
{
    string path = parameters[1];      //as first parameter the program needs a path name
    string extension = parameters[2]; //extension (like "*.cpp")
    string[] result = listdir( path, extension ); //crawl through all files with the given extension in the given directory and all sub directories and store the filenames in the "result" array.
  
    foreach( filename; result ) //scan one file after the other:
    {
        BufferedFile file = new BufferedFile( filename, FileMode.In ); //open the file for reading

        writefln( "\nFile: " ~ filename ); //print the filename
      
        foreach(ulong n, string line; file) //go through the lines and scan them for function names or function calls:
        {
            string[] functions = returnFunctions( line );
          
            if (functions !is null) printTree( functions ); //if the end of file is reached, print the result
        }
        file.close();
    }
}

 And here is an example, how the result looks like:

File: C:\functiondiagram.d

Function "returnFunctions()":
        RegExp();
        search();
        match();
        writefln();
        match();
        RegExp();
        match();
        match();
        match();
        search();
        done();
        call();
        find();

Function "printTree()":
        writefln();
        s();
        writefln();
        s();

Function "main()":

Function "extension()":
        listdir();
        BufferedFile();
        writefln();
        returnFunctions();
        printTree();
        close();

The program is not perfect yet. Some false positives are possible like in the example the function "s()" which is not a function but a regular expression.

New firmware for the thermometer

Here is an up-to-date version of the clock-thermometer firmware: Thermometer4_-_LCD.c
Because of school I did not have much time to make improvements.

Some current and finished projects...

Last month i rewrote my bool solver from scratch. This time i wrote it in pure C - without C++ elements. It has almost all functions of the old version plus some new ones, but the new program is much shorter. Documentation and comments are in german... thats why i do not upload it.

Because the old firmware (version 4) of the AtTiny clock was pretty ugly, i rewrote and changed many parts of the firmware. The new version has the number 5. The new code runs like a charm and as a bonus it eliminates the problems i had with the buttons (it was not a hardware problem).
Release will be soon. Only a calibration function for the quartz is still missing. ^.^°

And one last thing: i disassembled my old Acer travelmate 220 notbook and under the sticker of an IC i found an Atmel AtTiny25 as part of the charging circuit. It communicates directly with the laptop battery through TWI/I²C. The foto was made with my Canon EO350 / Digital Rebel XT / Kiss Digital N with an hand hold lens in retro position - the IC was so damn small:

UART - Video

In this small video, you can see a simple counter made with my UART board. The program for counting up the LEDs is running on a PC. The µC only communicates with the PC and is responsive to his commands.

UART - again

Here is a nice driver for Win2k + WinXP to enable the inportb and outporb commands under those systems: porttalk
This can be quite usefull, if you have old software, and you must work with those software - somehow.
The driver is quite nice. It works perfectly with my UART Testboard  example program (number 3).
I converted it from linux to Open Watcom (16 bit Dos).

Here is the output:

C:\Dokumente und Einstellungen\xxxx>D:\Programme\porttalk\AllowIo.exe D:\Program
me\WATCOM\projects\uart\test3\test3.exe
AllowIO for PortTalk V2.0
Copyright 2002 Craig Peacock
http://www.beyondlogic.org
Executing D:\Programme\WATCOM\projects\uart\test3\test3.exe with a ProcessID of
2089937184
PortTalk Device Driver has set IOPM for ProcessID 2089937184.

C:\Dokumente und Einstellungen\xxxx>

~RS232 Testboard~
Testing all possible combinations...

Handshake is: off
2400 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
2400 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
2400 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 0, hs: false... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 1, hs: false... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 2, hs: false... Device error: '0x00' UART error: 0x0000
Handshake is: on
2400 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
2400 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
2400 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
4800 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
9600 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
14400 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
19200 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
28800 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
38400 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
57600 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 0, hs: true... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 1, hs: true... Device error: '0x00' UART error: 0x0000
115200 Baud, p.: 2, hs: true... Device error: '0x00' UART error: 0x0000
End.

UART toy - version 1.2

I made some modifications for the RS-232 Testboard.
Now the baud rate can be changed and the parity check can be enabled. Tons of other modifications, a new exciting sample program and lots of bug fixes were made too.

You can download the whole package (AVR source code, PC source code + libraries, drawing and the command list) here:
project.zip

"RS-232 Testboard" - more software

Just finished some libraries and tests programs for the "RS-232 Testboard" . You can download the package here: test-software.zip
I wrote a low level UART library, a high level library, and two example programs, showing the usage of the high level library (and as a bonus they have some nice effects).
All code was tested with Linux - but it should work with DOS too. If you want to compile it under Linux, you will need the ncurses developer library for the second example program.
Hm... perhaps i will upload some videos soon.

UART toy - additional comments...

I forgot to mention, that the Linux program must be executed as root. Otherwise it will not work, because it needs low level access to the serial port.
And the program must be compiled with optimization turned on.

The correct preprocessor command for GCC under Linux is "__linux__" - with two underscores and in small letters.

All commands in the instruction set are stated as ASCII characters. The return values and the parameters are in hex.


If someone got the circuit to work with Windows XP, please inform me. ;)

UART toy - part 4

And here is a list with the instruction set plus a short list of error codes for the AVR part:

http://sites.google.com/site/species0x2118/commands.pdf?attredirects=0&d=1

Enjoy. ^.^

UART toy - part 3

And another part (not the last one) of my UART series.
Here is a piece of code for the PC to establish a connection to the RS232 testboard and to communicate with it..
http://sites.google.com/site/species0x2118/testboard.zip

It is just an example how to interface with the board and not a complete library. The example application sets the outputs, reads in the inputs, reads the analog input and calculates the voltage on that pin. Everything with hardware handshake.
But you need the right cable for the handshake. Otherwise change the "true" in the uartSend commands to "false" and delete the handshake activation.

The software is tested under DOS with TurboC and under Ubuntu Linux with GCC. The source compiles with Open Watcom too - but the Watcom binary is not tested under DOS yet.
I changed the compiler preprocessor macros for the recognition of operating systems and compilers several times - but did not test every combination with every operatin system and every compiler. Perhaps you need to figure out the correct setup for your system (or just delete the stuff you do not need).

It took me several hours to figure out, that linux needs an activated FIFO control register. Otherwise the data transmission will fail in 99% of the cases.

UART toy - part 2

Here is the code for the AVR as well as some pictures.
The C code: RS232_Testboard.zip
The code is public domain.
I am using Peter Fleury's AVR UART library for this project.

The first picture shows a test application, written on an 80286 with Borland TurboC that communicates with the RS232 testboard:

 The second picture shows the circuit itself:

New self made toy for UART programming

I have had very few time over the last weeks, so all hobby projects were paused in this time.
But now i have some spare time again - thx to Xmas and holidays!

The current theme in the technical engineering school i visit is "UART register programming". Sounds great, right? Just good enough for an nerd like me.
Sadly we only "program" the UART on paper... sounds sick - right? I was very unsatisfied with this fact and so i designed a small circuit arround an AVR. Now i am able to test the UART programming on the PC side. I learned much by this project.

The features of the board:

• Serial communication using RS232, 9600 baud, 8 data bits, no parity, 1 stop bit.
• 5 digital outputs (connected to LEDs)
• 6 digital inputs
• 1 analog input (8 bit ADC, Umax = 2,56 V)
• 10 build in commands
• hardware handshake can be activated
• 4 free ports on the AVR

First the circuit diagram:

Game

The development of the level generator is almost finished. I have uploaded some screenhots of the generated levels. After programming this small demonstration- and test-application (the first three pictures) i made a very versatile class out of it and hacked it into my existing game. The game was heavily optimized too. There is only one bug left in the level generator. And after that i will add some functions for adding doors, treasures, monsters and so on. On the last screenshot you can see how the generated level looks like in the game. Now some usefull links for building your own random maps:

http://steelgolem.wordpress.com/2008/01/28/random-level-generator/

http://roguebasin.roguelikedevelopment.org/index.php?title=Dungeon-Building_Algorithm

(http://roguebasin.roguelikedevelopment.org/index.php?title=Main_Page)

http://en.wikipedia.org/wiki/Rogue_%28computer_game%29

Enjoy! ^^

 

My game...

Because i had some free time last week, i started a small game project.
The game itself will be a mixture of "Castle of the winds" and "Diablo".
A project of my dreams. ;P
Round based Hack and slay, with tons of monsters, randomly generated maps, lots of weapons and armory...

Basic movement works, collision detection works, tile loading on two layers works, very basic map generation works.
Next on the list: a random map generator.
After that: interface tuning, adding monsters, adding weapons/armory/magic, adding sound ...

New software

Last weekend i wrote a small application for calculating truth tables based on boolean logic in C++. (/me was bored)
The software is inspired by Forth (the programming language) because it makes use of an stack and the reverse Polish notation.

The quickly written readme is currently available in German only.

This is an example for an XNOR with three inputs:
Formula:

a !not b !not c !not And And a b c And And Or a b !not c !not Or Or And

Or the simplified (optimized) version:

a !not b !not c
!not And And
a b c And And Or

Calculated truth table:

a b c Z
-------
0 0 0 1
1 0 0 0
0 1 0 0
1 1 0 0
0 0 1 0
1 0 1 0
0 1 1 0
1 1 1 1

Download:
Bool_solver.zip

Some C helper routines...

I wrote a small application for counting the number of lines (of code) in *.c files today.
As a side effect i have now some handy functions for different applications.
Luckily i had the passion to write them.


Lets take a look at the functions:

void findFiles( char *, void (char *, char *) );
Finds all files in a given folder + subfolders and calls a user defined callback function.

const char *getFileType( char * );
Returns the file extension of a given file.

const char *right( char *, int );
Returns 'i'  chars from the right side of an string (same function as the command 'right' in VBS).

Works with PellesC 6.00.4.
I used some non standard C functions in the code. So it will not work with every compiler without modifications.

Here is the library: utils.zip


(God damn...! The layout on blogger.com is so crapy)