Parallel make & optimizations

Here's a bunch of makefile notes for creating valid parallel makefiles. The problem with writing makefiles and having then executed in parallel is that you'll have to know your source dependencies and resolve them so make does not attempt to build your in the wrong order.

I have header a lot about how bad make is at solving dependencies, and hvor other tool like the latest build in Visual Studio build is much better etc. I strongly disagree. VC hides the dependencies behind a "pointy clicky" scheme. Make dosent assume anything it leaves the dependency problem up to the developers, meanins that the people working on the source should know how to solve their sources dependencies!

As an example, the common idiom for writing recursive makefiles is that directory traversing, or any traversion for that matter, is implemented as a shell loop, or by using the make function "foreach". As both of these methods solve the traversion, they also both have their drawbacks.

The drawback with the shell method is that firstly you'll invoke a separate shell for each directory in the make process, and make cannot know whats going on in these sub shells and therefore make cannot solve any dependencies in them, which in the end usually causes the build to break. The "foreach" method forces you to create a function based makefile, which again makes writing dependencies hard(er).

Makes primary way to handle dependencies is using targets. The usual way is to say i.e.

all: prepare: depend

The above says that to build the all target, you'll have to build first prepare, which in turn depends on depend, meaning that make will have to build depend before prepare. Simple, and as i may add easy to read. For make this means that to start any processes it'll first build depend, and any parts of depend that can be executed in parallel will be so, again this goes for prepare and all. But how do you then traverse a set of subdirectories? Simple, you'll just make your dub directories a target i.e.

SUBDIRS:= src lib interface doc

.PHONY: $(SUBDIRS)
$(SUBDIRS):
            $(MAKE) $(MAKEFLAGS) --directory=$@ $(MAKECMDGOALS)

.PHONY: all
all: $(SUBDIRS)

The snipplet above executes make in the SUBDIRS list, using a make target instead of the common "for" approach. Not each of these SUBDIRS may have a set of dependencies that needs to be resolved, these dependencies are solved in the respective SUBDIRS directory, where they belong.

Lets say, that for the sake of illustration that lib depends on src and interface. Meaning that you cannot build the lib part of the project before both src and interface are finished. All you'd have to do to tell make these dependencies is to add a new target line stating these dependencies i.e

lib: src interface

There, the lib target is now depended on the src and interface targets meaning that make cannot execute the lib target in parallel with either src and interface. So, because the dependencies must be resolved, make will execute src, interface and doc in parallel and once it's done build the lib target. Just as you specified.

Let's say that the src directory is a huge directory, containing roughly 100 different software components for the build. Some set of these components are interdependend. Then, again you'll have to create a makefile containing a SUBDIRS target to traverse the various src components. Now, even though there are 100 components you still have to know which ones are depending on each other, and which can be build in parallel.

Usually, solving your source dependencies are pretty easy, but it could be a major writing task. Because it may only be 4 of the 100 components that must be build first. You can solve this in various ways, by creating 2 targets like SUBDIRS making the higher level components dependend on the lower levels i.e.

LOWLEVEL:=hw io drv os

These are the components lowest level, they must be build prior to other components in the src directory. For the sake of argument, hw contain the memory map, a interrupt layout etc. io contain hardware io, you know hardware ;), drv contain drivers needed by the os to communicate with the hw etc.

Next, the src directory contain a core components part, usually some components that are part of a framework and running on top of the lower levels, this could again be a subset of the 100 directories.
 

CORE:=menu network error

And finally, the rest of the 100 components are applications that all must be build after LOWLEVEL and CORE

APPLICATIONS:=$(filter-out $(LOWLEVEL) $(CORE), $(shell ls -d ./*/))

The applications are found using a shell command, this is because listing everyfile in this directory would be a pain. But you should remember that automatically finding things like this can have its drawbacks.Once you have all the targets in your makefile, you'll have to resolve the dependencies i.e.

$(APPLICATIONS):$(CORE):$(LOWLEVEL)

Should do the trick. Here's a listing of the short make file I used to explore these issues independently with.

SHELL:=/bin/sh
ALL_DIRS:=$(shell ls -d ./*/)
ENV:=${ENV}
LOWLEVEL:=src elektra
STUFF:=Documents Music
CORE:=$(filter-out $(LOWLEVEL) $(STUFF),$(ALL_DIRS))
SUBDIRS:=$(STUFF) $(CORE) $(LOWLEVEL)

.PHONY:
all: $(SUBDIRS)
    @echo $(ENV)

.PHONY: $(SUBDIRS)
$(SUBDIRS):

.PHONY: $(LOWLEVEL)
$(LOWLEVEL):
    @echo "lowlevel-> $@"

.PHONY: $(STUFF)
$(STUFF):
    @echo "stuff-> $@"   

.PHONY: $(CORE)
$(CORE):
    @echo "core-> $@"

$(CORE):$(LOWLEVEL) $(STUFF)

.PHONY: lowlevel
lowlevel: $(LOWLEVEL)

.PHONY: core
core: $(CORE)

.PHONY: stuff
stuff: $(STUFF)

References: makefile_tricks, Recursive make considered harmful

Flexin' your Bison

Code that writes code that writes code

The problem I'd like to solve in this post is a simple network structure generation scheme. Say we have a number of messages that we'd like to send from one machine to another. Being programmers we'd also like to have a visual indication of our network transactions, for this I'd use "Wireshark". And we'd like to have valid documentation representation, say, in html or generated by doxygen for that matter.

To keep this example as simple as possible, I'll address only the C structure generation and the wireshark "Lua" extention generation. This pair should be enough to display the general idea behind code generation. What I'd like to achieve is to produce a set of outputs based on a single input, helping me maintain the code base. This approach is similar to the "Strategy" pattern, if you're into patterns.

The tool we're writing is a command line tool for linux. I'd like the following options:
-l creates lua output, -c creates c output and -a creates all targets. The tool should take a single file in as argument with -f and produce a single file with -o output i.e

# ./yantlg -c -f telegram.ntl -o telegram.h
# ./yantlg -l -f telegram.ntl -o telegram.lua
# ./yantlg -a -f telegram.ntl

The above examples are what I'd like the final program to use, in this simple version the focus will be on the Lex and Yacc parts of the project. As using gnu opts for passing and error reporting program options in a program is worthy of a separate blog post.

The Lex and Yacc tools are explained in detail in may places here is, IMHO, a good introduction.

Network telegram language

Lex needs a file it can use to tokensize the language. But first we must define a simple network telegram structure language or "NTL" for short. I'm sure I'm not the first implementer of something like this, there's most likely a bunch of tools out there, but here's what I consider rational for solving the problem at hand.

The first  and simple version of NTL will be supporting the data types listed below, and of these data types, except the end of message marker, need a name to identify the variable of that type, which gives us:

M <name> - message
B <name> - byte (8 bits)
W <name> - word (16 bits)
D <name> - double word (32 bits)
Q <name> - quart word (64 bits)
E - end of message

An example of a test NTL telegram could, therefore, be:

M telegram
B version
B size
W word
D dword
Q qword
E

This will be stored in a file called "telegram.ntl" and this file will serve as the input for our NTL generator (compiler/encode/writer/whatever) .

Flex'ing tokens with lex - "or lexing tokens with Flex"

Lex needs a token it shares these tokens with the parser, translate file. Lex input files are usually identified by the extention "l" (that's lower case L), at least that's the extension recognized by make, so we'll use this.

%{
#include <stdio.h>
#include <stddef.h>
#include "yyltype.h"
#include "parser.h"

/* definitions for reporting errors, see a lex and yacc doc/book for details */
int yycolumn = 1;
#define YY_USER_ACTION yylloc.first_line = yylloc.last_line = yylineno; \
    yylloc.first_column = yycolumn; yylloc.last_column = yycolumn + yyleng - 1;\
    yycolumn += yyleng;

%}
%option yylineno

%%
m                     { yylval.type = MESSAGE; return MESSAGE;}
e                     { return END; }
b                     { yylval.type = BYTE; return BYTE; }
w                     { yylval.type = WORD; return WORD; }
d                     { yylval.type = DWORD; return DWORD; }
q                     { yylval.type = QWORD; return QWORD; }
[a-zA-Z][a-zA-Z0-9_]+ { yylval.name = strdup(yytext); return NAME; }
\n                    { yycolumn = 1; } /* ignore end of line */;
[ \t]+                /* ignore whitespace */;
[^mebwdq]             { yyerror("'%s' unknown symbol",yytext); yyerror;}  /* Error on unknown tokens */
%%

The above is a *simple* lexer for our NTL generator, the only file that seem a bit strange is in the header section where #include "parser.h" is added. This file is generated by Yacc, and it must be included for the token names to be defined in the final compilation.

Notice every data type is assigned to a token. A token is identified by a token type and a name (except from the end token).  Tokens are composed by the [a-zA-Z][a-zA-Z0-9_]+ regular expression.

This expression states that a name must begin with a character either upper or lower case, this is defined by the first []: [a-zA-Z] part. Followed by either upper or lower characters a numeric value or the underscore character, defined by the second []: [a-zA-Z0-9_]+ part.

The final tree expressions are to ignore white spaces and newlines, and forces error reporting for unknown token characters.

Tokenized code generation

The code generation part is handled with Yacc (Bison). This tool needs a file with the extension "y", again an extension recognized by make. This file must contain the various specific C code parts for our NTL generator program.

Like the Lex file for tokenization this file is divided into a number of sections. Where the percentage curly braces contains the C code part, "%{ C code %}", including header file definitions etc. The %tokens contain the token definitions, and, finally, the percentage percentage section, "%% bnf %%", contain the BNF notated language syntax.

%{
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>

#include "token.h"
#include "yyltype.h"

extern int yyleng;           /* Defined by the lexer used to output the length of the column */
extern int yylineno;         /* Defined by the lexer to get the current line number */
extern char *yytext;         /* Defined by the lexer to get the current token stream */
struct YYLTYPE yylloc;       /* This is the lexer location structure, it is used to produce the parser error messages */
struct Token *root,*current; /* This is the parsed token tree, it is collected by the parser and used later to produce the output code */

void yyerror(char *s, ...)
{
  va_list ap;
  va_start(ap, s);

  fprintf(stderr,"error: line: ");

  if(yylloc.first_line) {
      fprintf(stderr, "%d.%d-%d.%d ", yylloc.first_line, yylloc.first_column,yylloc.last_line, yylloc.last_column);
  }
  vfprintf(stderr, s, ap);
  fprintf(stderr, "\n");
}

void lyyerror(YYLTYPE t, char *s, ...)
{
  va_list ap;
  va_start(ap, s);

  fprintf(stderr,"error: line: ");

  if(yylloc.first_line) {
      fprintf(stderr, "%d.%d - %d.%d ", yylloc.first_line, yylloc.first_column,yylloc.last_line, yylloc.last_column);
  }
  vfprintf(stderr, s, ap);
  fprintf(stderr, "\n");
  va_end(ap);
}

int yywrap()
{
  return 1;
}

%}

%locations

The above C code section contain error parts, see the yacc/bison documentation or check a book on the subject for details. We will focus on the next part, where the actual parsing definitions are placed.

%union {
    int type;
    char *name;
}

%token <type> MESSAGE
%token <type> BYTE
%token <type> WORD
%token <type> DWORD
%token <type> QWORD
%token <name> NAME
%token END

%%
message: /* empty */
   |
   message data;

data:  byte | word | dword | qword | end;

message: MESSAGE NAME { root = current = create($1,$2); };
 | NAME { lyyerror(@1,"'%s' undeclared identifier",$1); /*YYABORT*/;};
 | error '\n'

byte: BYTE NAME { current = current->next = create($1,$2); };
 | NAME { lyyerror(@1,"'%s' undeclared identifier",$1); /*YYABORT*/;};
 | error '\n'

word: WORD NAME { current = current->next = create($1,$2);};
 | NAME {lyyerror(@1,"'%s' undeclared identifier",$1); /*YYABORT*/;};
 | error '\n'

dword: DWORD NAME { current = current->next = create($1,$2);};
 | NAME {lyyerror(@1,"'%s' found where d was exspected",$1); /*YYABORT*/;};
 | error '\n'

qword: QWORD NAME { current = current->next = create($1,$2);};
 | NAME {lyyerror(@1,"'%s' undeclared identifier",$1); /*YYABORT*/;};
 | error '\n'

end: END { current = NULL;};
%%

The above yacc code defines the token type, the input as it is read fom yyin to be either a int, in the case of token type or a char* in the case of token names.  These are used to create a token table (a simple linked list) from the files token.h and token.c

The next part, all the %token <foo> bar, definitions defines the tokens. Finally, the %% message: section defines the NTL language BNF.

Basically, what the BNF does it defines that a message must contain a set of data, taht can be of the types byte, word, dword, qword or end.

Then, each of the token types are assigned some code, which in a correctly formatted scenario feeds a token generator, a simple c linked list, with the token type and name. If there is an error in the syntax this will be issued to the user and the parser will read to the next newline.

The token linked list

Heres a listing of the token linked list structure for creating the simple linked list. First the header then the implementation file

#ifndef token_h_
#define token_h_

struct Token {
    int type;
    char *name;
    struct Token *next;
} *root,*current;

struct Token *create(int type, char *name);

#endif // token_h

The above defines a token structure, used to generate code from.

#include <stdlib.h>
#include <string.h>
#include "token.h"

struct Token *create(int type, char *name)
{
    struct Token *t = malloc(sizeof(struct Token));
    memset(t,0x00,sizeof(struct Token));
    t->type = type;
    t->name = name;
    return t;
}

The above defines a function that creates and assigns the token type and name to a simple linked list. This list is used to generate the code from.

The code generation for this project is also fairly simple. The principle is that a single function generates the code for each section. This function is then used from the main program as a function pointer, providing the polymorphic ability for generating code.

The benefit of using function pointers is that to generate a new output all you have to do is write a function for the generation, and register this function. The other parts of the program stays the same.

#ifndef C_GENERATOR_H
#define C_GENERATOR_H

enum OUTPUT_TYPE {
    GENERATE_NONE = 0x00,
    GENERATE_TREE,
    GENERATE_C,
    GENERATE_LUA,
    GENERATE_MAX = GENERATE_LUA,
};

void c_generator(char *filename);
void tree_generator(char *filename);
void lua_generator(char *filename);
typedef void (*generator)(char *);

#endif /* C_GENERATOR_H */

The above defines a set of functions, one for each output type, and a set of constants for the corresponding function.

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include "token.h"
#include "file.h"

void tree_generator(char *outfile) {

    outfile = strcat(outfile,".tree");
    fprintf(stdout,"Generating: %s\n",outfile);

    struct Token *next;
    fprintf(stdout,"Generating: %s\n",outfile);
    fprintf(stdout,"Token: %s Type: %d\n",root->name,root->type);
    for(next = root->next; next!=NULL; next = next->next) {
    fprintf(stdout,"\t |-- Type: %d - Token: %s\n",next->type,next->name);
    }
}

void c_generator(char *outfile)
{
    struct Token *next;

    outfile = strcat(outfile,".h");
    fprintf(stdout,"Generating: %s\n",outfile);

    char *type_words[] =  {"","struct","uint8_t","uint16_t","uint32_t","uint64_t"};
    char *hton_functions[] = {"","","","htons","htonl","undefined_htoll"};
    char *ntoh_functions[] = {"","","","ntohs","ntohl","undefined_ntohll"};
    FILE *ntlout = file_open(outfile,"w","couldn't open temp for writting\n");

    fprintf(ntlout,"#ifndef %s_h\n",root->name);
    fprintf(ntlout,"#define %s_h\n\n",root->name);

    fprintf(ntlout,"struct %s {\n",root->name);
    for(next = root->next;next!=NULL;next = next->next) {
    fprintf(ntlout,"\t%s %s\n",type_words[next->type],next->name);
    }
    fprintf(ntlout,"\n");
    fprintf(ntlout,"\tvoid hton();\n");
    fprintf(ntlout,"\tvoid ntoh();\n");
    fprintf(ntlout,"};\n\n");

    fprintf(ntlout,"void %s::hton()\n{\n",root->name);
    for(next = root->next;next!=NULL;next = next->next) {
    fprintf(ntlout,"\t%s = %s(%s)\n",next->name,hton_functions[next->type],next->name);
    }
    fprintf(ntlout,"}\n\n");
   
    fprintf(ntlout,"void %s::ntoh()\n{\n",root->name);
    for(next = root->next;next!=NULL;next = next->next) {
    fprintf(ntlout,"\t%s = %s(%s)\n",next->name,ntoh_functions[next->type],next->name);
    }
    fprintf(ntlout,"}\n\n");

    fprintf(ntlout,"#endif // %s_h\n",root->name);

    fclose(ntlout);
}

void lua_generator(char *outfile) {
    outfile = strcat(outfile,".lua");
    fprintf(stdout,"Generating: %s\n",outfile);
}

Disadvantages

Lets face it, writing code that writes code has both its advantages and disadvantages.
The advantages are many, and discussed deeply at various resources like: "The Pragmatic Programmer" or "The Art Of Unix Programming". The Disadvantages are usually not addressed as they're not the key selling point, and their consequences are usually ignored. Remember:

"If you write code that writes bad code, there's no end to what you can't generate!"

If you just want to know how jump to the code section. If you want to know why, the following section lists some of the pitfalls I've met during my travels in professional programmer land.

The disadvantages I personally have experienced with code generation, have all been in the same area.

1. The outputted code was impossible to debug and use due to bad code design.
2. The input was so advanced that it was a programming language on it's own.
3. Lack of tokenization after the lexical analysis.
4. Choosing the wrong tool for the job.

Well, the first item leads to the kind of code you just want to, for the lack of a better word, refactor. Bad code should be removed immediately as it's consequences is your projects greatest risk. The first item is also related heavily to the second item.

The second leads to your project having to educate its programmers to the specific code generation language. This is expensive as you cannot hire an expert. You'll have to, as a project or company, educate your own experts.

That will be hard as the experts your project educate cannot use the expertise elsewhere, and this might cause a motivational issue in your project. Plus, your project will have to maintain the full infrastructure related to a programming language like: usage examples, and documentation. Expensive.

Item three is a common mistake, a code generation step that is often overlooked in the process of producing the code generator output. This may be due to lack of experience, or, even worse, due to a poor programming work environment. The latter of these is the worst as the environment enforces a fast process with out any consideration for the side effects. A side effect could be porting of the generated code to a new platform, or documentation generation in a different format.

The fourth item is a common mistake in many technical matters. Choosing a tool to preform a specific task is often not trivial. But in many cases some of the new parsers etc. are often misused due to lack of understanding in the problem domain. One of the things I have seen is the usage of Apache ant and not make.

This is not a criticism of either Apache ant or make. I'm certain that they both have their specific problem to solve. Nevertheless, I believe that the purpose of Apache ant can be solved by make, but that is just my personal opinion, I honestly believe in the approach taken in the "The Art Of Unix Programming" book. Where the suggested solution is to make use of the tools available to you already, and creating only the smallest possible solution to perform the task needed.

Re-sizing the C drive of your vmware installation.

1 Re-sizing the VM disk
1.1 Power oww your virtual XP installation
1.2 Select VM menu item, and press settings in the drop down list
1.3 Select the Hard Disk entry, and press the utilities button in the lower right corner of the dialog
1.4 Select Expand from the utility list, set the desired size in the dialog (I went for 100 GB)
1.5 Wait until the virtual machine has expanded the drive size, then click save

2 Re-sizing the virtual XP disk
2.1 install a partition tool on your machine, I'm using EASEUS partition magic professional
2.2 Follow your tool's re-size procedure (Mine is listed below)
2.2.1 Select the C drive and click the re-size button.
2.2.2 Drag the allocated space pointer to the desired size (100GB)
2.2.3 Press apply and close the partition tool
2.2 Reboot your virtual XP image

3 Check new partition size
3.1 After rebooting your virtual XP image, press "windows key" + e
3.2 Right click the Local Disk(C:), and select properties

Missing altgr keystrokes in VMware

If your VMWare keyboard mapping prevents you from using special keys such as Alt Gr, just add the following the your /etc/vmware/config:

xkeymap.keycode.108 = 0x138 # Alt_R xkeymap.keycode.106 = 0x135 # KP_Divide xkeymap.keycode.104 = 0x11c # KP_Enter xkeymap.keycode.111 = 0x148 # Up xkeymap.keycode.116 = 0x150 # Down xkeymap.keycode.113 = 0x14b # Left xkeymap.keycode.114 = 0x14d # Right xkeymap.keycode.105 = 0x11d # Control_R xkeymap.keycode.118 = 0x152 # Insert xkeymap.keycode.119 = 0x153 # Delete xkeymap.keycode.110 = 0x147 # Home xkeymap.keycode.115 = 0x14f # End xkeymap.keycode.112 = 0x149 # Prior xkeymap.keycode.117 = 0x151 # Next xkeymap.keycode.78 = 0x46 # Scroll_Lock xkeymap.keycode.127 = 0x100 # Pause xkeymap.keycode.133 = 0x15b # Meta_L xkeymap.keycode.134 = 0x15c # Meta_R xkeymap.keycode.135 = 0x15d # Menu   Complete steal from: http://draptik.wordpress.com/2009/07/28/vmware-keyboard-mapping-issues-alt-gr-solved/    Copied because the distribution customers @work needs this info, and I didn't relyon the resource being available. Thanks @draptik

Bash one liners

Automatically remove dead "java" processes, simply replace process with java below

#ps aux | grep <process> | grep <optional other process> | tr -s " " | cut -d" " -f2 | xargs kill -9

The key point here is that the output from the ps command is cluttered with spaces and tabs when used together with cut. So, tr can be used to trim the lines, what the expression does is exactly that . It replaces all spaces with a single space character, allowing us to use this character as a delimiter to cut.

Testing memory maps on embedded systems

Here's the thing, sharing constant values between FPGA hardware (HW) and software (SW) can be hard for both sides. The constants that need sharing are memory addresses and masks for various registers. The main reason that sharing constants is hard is that no simple framework, at least to my knowledge, exist.

Usually, both sides of constants can be generated from some common format, or the HW side have to create a interface header containing the constants needed for the SW. Either way when a constant is changed it needs to tested properly.

But what happens when the HW side changes and the new FPGA code is released? Mayhem, since the HW doesn't have the complete SW source to test against, as their FPGA project could be in a different system, in a different source control etc.

In this case the HW should at least create a simple program that verifies their naming conventions, addresses and masks are the same as the last release. This is IMHO the smallest possible release test of a new FPGA image that can be run before releasing the HW. I'm not stating this is enough to actually test the HW, but it is enough to ensure that the SW does not break due to new names and wrong constants.

For this I have written a small C program. This program uses the MACROS defined by the HW and tests these. Firstly against C compilation, this step is important because changing a name would break compilation on the SW side, and changes like these should not be possible. Secondly, changing masks and address values should also be verified, at least against the previous version.

This will ensure that the HW side will know exactly what they break on the SW side if they change anything. And they'll clearly have a visual indication of what fails, enabling them to report any changes.

/***
  name: macro dumper
  version: 0.1
  author: john
  description:
  This program is a simple interface test for the defined memory map.
  The program dumps the macro's name and value to stdout.

  The program purpose is to test all generated macros against their generated
  name and value to ensure an interface test of the generated HW memory map

  The program will fail to compile any of the macros change their name according
  to the first version of the HW interface.

  The test program below is supposed to fail in the deliberate fail macro! This
  line may be commented in a release version of the simple test program

  Remember that new test cases must be added when new HW is introduced
  When you're introducing new macros in the hardware interface the following
  regular expressions will assist you in getting the macros from the interface
  header and into valuable C code that you can paste into the source's test
  sections below.

  The comment format change for a reason, namely the regular expression characters
  contain both * and / which is comment end ;)
***/
//  Regular expressions to:-
//  List all macros in the interface with all other things stripped:
//
//  egrep "#define [a-zA-Z0-9_]+ +" macro_dumper.c | perl -p -e "s/#define +(.+) \((.*)\)/NAME: \$1 VALUE: \$2/g"
//
//  Replace all macros with a corresponding test call for inserting into the name test:
//
//  egrep "#define [a-zA-Z0-9_]+ +" macro_dumper.c | perl -p -e "s/#define +(.+) \((.*)\) ?.*/TEST_NAME(\$1);/"
//
//  Replace all macros with a corresponding test call for inserting into the value test:
//
//  egrep "#define [a-zA-Z0-9_]+ +" macro_dumper.c | perl -p -e "s/#define +(.+) \((.*)\) ?.*/TEST_VALUE(\$1,\$2);/"
#include <stdio.h>

/*
  This test result construction is meant for the test to produce an output for any other command line tool.
  The purpose is that the test_result returns >0 in case of errors and report these to the OS.

  A return value of zero indicates a failed test
*/
static int override_return_value = 0;
static int test_result = 0;
static int test_case = 0;
int assign_test_result(int value,int varname)
{
    if(value != varname) {
    test_result++;
    return 0;
    }
    return 1;
}

#define TEST_NAME(varname) fprintf(stdout, "%s = %x\n", #varname, (varname));
#define TEST_VALUE(varname,value) fprintf(stdout, "%d - %s = %s\n",++test_case, #varname, (assign_test_result(varname,value))?"ok":"error");

/* #include "memorymap.h" */
/* The macros below are ment to be placed in the memorymap file provided by HW, here they are simply a test of the concept */
#define TEST1_MACRO (0x010000000)                 /* emulating some start address */
#define TEST2_MACRO (0x2000)                      /* emulating some offset */
#define TEST3_MACRO (TEST1_MACRO + TEST2_MACRO)   /* Check to see that the macro names are reported correctly when combined */
#define TEST_MACRO_MASK (0x0010)                  /* Test for some kind of mask */
#define DELIBERATE_ERROR_MACRO (0xff)             /* The check for this macro should not be 0xff as we deliberately want this check to fail */

int main(int argc, char** argv)
{
    override_return_value = argc-1;

    fprintf(stdout,"%s","This program checks the name and address values of memory map macros\n");
    if(override_return_value) {
    fprintf(stdout,"%s","Running in override return value mode, errors are only reported not breaking!\n");
    }

    /*
      This is the name test part of the program, any non complying marcos will cause a compilation error
      on the exact line of the non existing macro name. The test is valuable for finding non compatible
      name changes in the interface
    */
    TEST_NAME(TEST1_MACRO);
    TEST_NAME(TEST2_MACRO);
    TEST_NAME(TEST3_MACRO);
    TEST_NAME(TEST_MACRO_MASK);
    TEST_NAME(DELIBERATE_ERROR_MACRO);

    /*
      This part tests against the original macro values to ensure that we know when memory addresses actually change
      This test is to ensure that we know when a macro value changes. This is not a test to break the interface
      It's simply an indication that an address or mask has changed.
    */
    TEST_VALUE(TEST1_MACRO,0x010000000);
    TEST_VALUE(TEST2_MACRO,0x2000);
    TEST_VALUE(TEST3_MACRO,0x010000000 + 0x2000);
    TEST_VALUE(TEST_MACRO_MASK,0x0010);
    TEST_VALUE(DELIBERATE_ERROR_MACRO,4); /* This is a deliberate error to test the test macro ;) */

    /*
       Reporting test case statistics
     */
    fprintf(stdout,"Tests run: %d\n",test_case);
    fprintf(stdout,"Failures: %d\n",test_result);
    return (override_return_value)?test_result:0;
}



The program is compiled using gcc or equivalent by issuing:

john@BlackWidow gcc macro_dumper.c -o macro_dumper

In your terminal. You run the program by issuing:

john@BlackWidow ~/john/src/c $ ./macro_dumper
This program checks the name and address values of memory map macros
TEST1_MACRO = 10000000
TEST2_MACRO = 2000
TEST3_MACRO = 10002000
TEST_MACRO_MASK = 10
DELIBERATE_ERROR_MACRO = ff
1 - TEST1_MACRO = ok
2 - TEST2_MACRO = ok
3 - TEST3_MACRO = ok
4 - TEST_MACRO_MASK = ok
5 - DELIBERATE_ERROR_MACRO = error
Tests run: 5
Failures: 1
john@BlackWidow ~/john/src/c $ echo $?
0

Above you can see the output from the program. Keep in mind that the program should be compiled at every HW release and run to check against the interface.

The program is constructed in such a way that giving 1 or more arguments of any type will cause the program to report test case failure to the OS meaning that it could be used to break a Jenkins build bot on failures.

Yes it can be expanded. Yes it can be re factored. Yes there's lots of potential for improvement. But for now it's the smallest possible thing that will safe many hours of non working interfaces if something went wrong.

Matplotlib on mint 13

Today, I had a friend who needed to use matplotlib on our company linux boxes. We have some issues with out proxy server that rules python pip out. This mixed with python version and other dependencies lead me to write this small script, as a kind of configure script, for his code project.

The script deals with installing python3 and other prebuilt packages for ubuntu (mint) and then uses python3 build mechanisms & easy_install to add any remaining python egg dependencies.

The script source & listing:
#!/bin/sh
#
# This is the friendly python3 matplotlib installer script
# it checks for all matplotlib dependencies as pr. Sep 9 2014
# Any additions to the installer must be added by hand!
#
# If none of the dependencies are installed, the script will:
# 1. install prebuild ubuntu packages
# 2. Create a local dir for downloaded files
# 3. install dependencies by downloading these from various sources
# 3.a cython
# 3.b numpy, scipy
# 3.c pyttk, six, tkinter, pip,matplotlib
# 4. remove the directory after installation
#
# john

# 1.
sudo aptitude install python3 python3-dev python3-setuptools python3-tk python3-cairo python3-cairo-dev libpng-dev
sudo aptitude install gfortran libopenblas-dev liblapack-dev
sudo easy_install3 -U distribute

# 2.
matlib=/tmp/matlib
[ -d $matlib ] && sudo rm -rf $matlib
mkdir $matlib && cd $matlib

# 3. Install dependencies, these vary and will change for furure releases!
# As the matplotlib we use in the dse require bleeding edge, we'll grap the latest stuff from the bleeding repositories

# 3.a
wget http://cython.org/release/Cython-0.20.2.tar.gz && tar -zxvf Cython-0.20.2.tar.gz && cd Cython-0.20.2/ && sudo python3 setup.py install && cd -

# 3.b
#install bleeding edge eggs from git repositories
for package in numpy scipy; do
    [ -d $package ] && sudo rm -rf $package
    git clone http://github.com/$package/$package.git $package && cd $package && sudo python3 setup.py install && cd -
done

# 3.c
# install bleeding edge eggs using easy_install
# NOTICE: this loop can only be extented if the egg name is the same in both the python code and module name in bash!
for egg in pip six tkinter pyttk matplotlib; do
    sudo easy_install3 $egg
done

pip freeze

cd -

# 4.
sudo rm -rf $matlib

A software company that dosen't suck

Short post, Great article.

Configuring VNC connections to you Box

A quick entry today. I needed to rid the *missing ubuntu session* dialog when connecting via VNC to my box@work.

So, I edited the vnc xstartup file:

#!/bin/sh
unset SESSION_MANAGER
gnome-session --session=gnome-classic &

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &

Finally start the vncserver with something like:

#vncserver -geometry 1280x1024 -depth 24

In depth resource for this issue.

Assogiate

But Mooooooom it wan't it like window$. Me just pointy clicky and then like magic the file i clicked opens in my favorite editor! Sigh... Did you ever have programmers (users) that worked their entire career on a single platform. Programmers that think everything should work like it used to... Programmers unwilling to learn new stuff?

I have a bunch of programmer rednecks who apparently need to point and click in "the configured file manager" to be able to open a source file. And these programmers want the same behavior on Linux as they have in Window$ file Exploder ... Sigh ...

Here's how: http://ubuntugenius.wordpress.com/2009/11/19/create-your-own-file-types-in-ubuntu-with-assogiate/.  In my case I had to create a debian package carrying the settings, enabling these programmers to use synaptic to install the extension.

Programmers? ... Go figure ..

How to Remove the Visual Deals Popup Spam

I normally don't do this, but these deals are so annoying, so here's: How to Remove the Visual Deals Popup Spam, its hidden in your pinterest button!

gdb tips

I usually debug lLnux programs using gdb, I like the neatness of the tui mode, thanks for that feature! Here are some simple commands to get you started using gdb. There are several major doc out there, this is simply meant as a quick start and remembering reference for me, I'll make no attempt to replace any of these.

• Starting your program with arguments: gdb -tui --args love_calc john jane

• Setting a breakpoint in a specific file (in your huge project): b filename.cpp:linenum, i.e. b posix_thread.cpp:25
• Stepping is s or (step) stepping into is stepi and continue is c.
• Examining variables is done with p variable_name, i.e. p argv[1]

The above image shows the result of the commands.

There's a small guide from gnu on tui single key mode & a quick guide if you need more than I have shown here.

Typesafe bit mask operations or Bits, bytes and operators.

In production? You're kidding right? No way some network structures contain c bit-fields. Turns out he wasn't kidding (sigh). Sometimes you just have to wonder how, or who, that stuff get's in there.

There's a neat trick in c++ for creating type safe bitmasks using enums and a template class. You'll have to know your C++ operators and how to override these if you're looking for more than just my simple template class. The source is here. You build it with: g++ bitfiled.cpp -o bitfield -Werror

#include <stdio.h>
#include <stddef.h>
#include <stdint.h>
#include <iostream>
#include <bitset>

template<class T,typename S>
class bitfield {
public:

    inline bitfield();
    inline bitfield(const T &bit);

    inline const size_t size() const;
    inline const S num_bits() const;
    inline const S get(const T &bit) const;
    inline const S &get_bits() const;

    inline void set(const T &bit);
    inline void clear(const T &bit);
    inline void toggle(const T &bit);

    inline S operator^=(const T &bit);
    inline S operator|=(const T &bit);
    inline S operator&(const T &bit);
    inline S operator=(const T &bit);

    inline const char *dump()const;

private:    
    S bits;
    static const S zero = 0;
    static const S bit_base = 8;
};

template<typename T,typename U>
inline void set_bit(U &bits,const T mask)
{
    bits |= mask;
}

template<typename T,typename U>
inline void toggle_bit(U &bits,const T mask) 
{
    bits ^= mask;
}

template<typename T,typename U>
inline uint8_t clear_bit(U &bits,const T mask) 
{
    return bits &= ~mask;
}

template<typename T,typename U>
inline const bool is_bit_set(const U &bits,const T mask)
{
    return bits & mask;
}

template<class T,typename S>
inline bitfield<T,S>::bitfield()
{
    bits = zero;
}

template<class T,typename S>
inline bitfield<T,S>::bitfield(const T &bit)
{
    bits = bit;
}

template<class T,typename S>
inline const S &bitfield<T,S>::get_bits() const
{
    return bits;
}

template<class T,typename S>
inline const size_t bitfield<T,S>::size() const
{
    return sizeof(*this);
}

template<class T,typename S>
inline const S bitfield<T,S>::num_bits() const
{
    return size()*bit_base;
}

template<class T,typename S>
inline void bitfield<T,S>::set(const T &bit) 
{
    ::set_bit(bits,bit);
}

template<class T,typename S>
inline void bitfield<T,S>::clear(const T &bit)
{
    ::clear_bit(bits,bit);
}

template<class T,typename S>
inline const S bitfield<T,S>::get(const T &bit) const
{
    return ::is_bit_set(bits,bit);
}

template<class T,typename S>
inline void bitfield<T,S>::toggle(const T &bit)
{
    ::toggle_bit(bits,bit);
}

template<class T,typename S>
inline const char *bitfield<T,S>::dump() const
{
    std::string out;
    for(unsigned int i=num_bits();0!=i;i--)
    {
out += ((1 << (i-1)) & bits) ? "1" : "0";
    }
    return out.c_str();
}

template<class T,typename S>
inline S bitfield<T,S>::operator^=(const T &bit)
{
    ::toggle_bit(bits,bit);
    return bits;
}

template<class T,typename S>
inline S bitfield<T,S>::operator|=(const T &bit)
{
    ::set_bit(bits,bit);
    return bits;
}

template<class T,typename S>
inline S bitfield<T,S>::operator&(const T &bit)
{
    return ::is_bit_set(bits,bit);
}

template<class T,typename S>
inline S bitfield<T,S>::operator=(const T &bit)
{
    return bits = bit;
}

enum Mask16 {
    ab1=0x0001,
    ab2=0x0002,
    ab3=0x0004,
    ab4=0x0008,
    ab5=0x0010,
    ab6=0x0020,
    ab7=0x0040,
    ab8=0x0080,
    ab9=0x0100,
    ab10=0x0200,
    ab11=0x0400,
    ab12=0x0800,
    ab13=0x1000,
    ab14=0x2000,
    ab15=0x4000,
    ab16=0x8000,
};

enum Mask8 {
    b1 = 0x01,
    b2 = 0x02,
    b3 = 0x04,
    b4 = 0x08,
    b5 = 0x10,
    b6 = 0x20,
    b7 = 0x40,
    b8 = 0x80,
};

int main (int argc, char **argv)
{
    bitfield<Mask8,uint8_t> bf8;
    std::cout << "-------------------------------" << std::endl;
    std::cout << "bf8 size: " << bf8.size() << std::endl;
    std::cout << "Bits constructor: " << std::bitset<8>(bf8.get_bits()) << std::endl;

//    bf8.set(b2);
    bf8 |= b2;
    std::cout << "Bit initialized: " << std::bitset<8>(bf8.get_bits()) << std::endl;

    uint8_t bit_flip = 0;
    for(int i=0;i<8;i++)
    {
bit_flip = (1 << i); 
const char *p = (bit_flip<=0x08) ? "0x0" : "0x";
std::cout << "-------------------------------" << std::endl;
std::cout << "Simulated Mask: " << std::bitset<8>(bit_flip) << std::endl;
std::cout << "Simulated Hex : " << p << std::hex << (int)bit_flip << std::endl;

// bf8.toggle(static_cast<Mask8>(bit_flip));
bf8 ^= static_cast<Mask8>(bit_flip);
// (bf8.get(static_cast<Mask8>(bit_flip))) ? std::cout << "true" << std::endl :std::cout << "false" << std::endl ;
(bf8 & static_cast<Mask8>(bit_flip)) ? std::cout << "true" << std::endl :std::cout << "false" << std::endl ;
std::cout << "bf8.bits " << std::bitset<8>(bf8.get_bits()) << std::endl;
    }

    bitfield<Mask16,uint16_t> bf16;
    std::cout << "-------------------------------" << std::endl;
    std::cout << "bf16 size: " << bf16.size() << std::endl;
    std::cout << "Bits constructor: " << std::bitset<16>(bf16.get_bits()) << std::endl;

    bf16.set(ab9);
    std::cout << "Bit initialized: " << std::bitset<16>(bf16.get_bits()) << std::endl;

    bf16 = ab10;
    std::cout << "Bit initialized: " << std::bitset<16>(bf16.get_bits()) << std::endl;
    std::cout << "num bits: " << std::dec << bf16.num_bits() << std::endl;

    // testing for placement in a telegram!
    struct test_telegram {
uint8_t version;
uint8_t type;
bitfield<Mask8,uint8_t> b;
    }tt = {0};

    std::cout << "-------------------------------" << std::endl;
    std::cout << "tt size: " << sizeof(tt) << std::endl;

    tt.b = b3;
    std::cout << "tt.b: " << std::bitset<8>(tt.b.get_bits()) << std::endl;
    std::cout << "tt.b.dump() : " << tt.b.dump() << std::endl;

    bitfield<Mask8,uint8_t> bf_constructor(b5);
    std::cout << "-------------------------------" << std::endl;
    std::cout << "bf_constructor: " << std::bitset<8>(bf_constructor.get_bits()) << std::endl;
    std::cout << "bf_constructor.dump() : " << bf_constructor.dump() << std::endl;

    // Using the template function to manipulate c style!
    ::set_bit(bf_constructor,b3);
    std::cout << "global function - bf_constructor: " << std::bitset<8>(bf_constructor.get_bits()) << std::endl;

//    ::set_bit(bf_constructor,0x08); // error needs a valid Mask8 type or a cast static_cast<Mask8>(0x08)
//    bf_constructor.set(0x08);       // error needs a valid Mask8 type or a cast static_cast<Mask8>(0x08)
//    bf_constructor.get(0x08);       // error needs a valid Mask8 type or a cast static_cast<Mask8>(0x08)
//    bf_constructor.toggle(0x08);    // error needs a valid Mask8 type or a cast static_cast<Mask8>(0x08)
    return 0;
}

I created 4 template functions, set_bit, clear_bit, toggle_bit and is_bit_set. Their purpose is to both serve as a simple interface, and to keep the types in place. They're used through the class, and they could be declared in the class, but for me that would make the functions loose their purpose. 

Notice there are two enumerations, namely, Mask8 and Mask16. These two types are used in the bitfield to ensure the type safety. The main function is meant as a simple test of the type safety. Play with this stuff all you wan't.

You'll notice that you cannot set, say, an int in either of the mask operators (or functions) all will give you a compile error, the sole intent of the implementation. Where do you use it you ask, I'd be using it every where I find a c/c++ bitfield used inside a structure, to keep the code portable on many platforms and to be able to use the above bitfield class in network telegrams.