Blue Melon gaat samen verder met eMbed Engineering onder de vlag van eMbed Engineering! Met het samenvoegen van beide bedrijven is een stabiele en slagvaardige organisatie ontstaan die flexibel is en garant staat voor kwalitatief hoogwaardige producten. Ons uitgangspunt hierbij is dat we u nu nog beter van dienst kunnen zijn in het ontwikkelen van elektronica. Het samenvoegen van beide bedrijven heeft ook geleidt tot een uitbreiding van het portfolio en een enorme toename van competenties. Competenties die uiteen lopen van hardwareontwikkeling op chipniveau, tot higher level embedded software op basis van embedded Linux met internet connectiviteit. Hierdoor zijn we in staat om met parate kennis de meest uiteenlopende projecten in korte tijd tot een succes maken.

Vanaf begin december is eMbed Engineering te vinden op www.eMbed.nl. Op onze vernieuwde website kunt u naast informatie over eMbed Engineering als organisatie een selectie van ons portfolio vinden. We hopen u met dit bericht volledig te hebben geïnformeerd over de ontwikkelingen bij eMbed Engineering. Als u vragen hebt over de situatie of de ontwikkelingen, dan kunt u ons uiteraard bereiken via onze contactgegevens.

Deze website wordt niet meer bijgewerkt, bekijk de nieuwe website van eMbed Engineering!

From BlueMelon

Automated Test Framework for Embedded Software (ATFES)

'Atfes' stands for "Automatic Test Framework for Embedded Systems". Atfes is a Python tool which can be used to run tests on software programmed in "C".

Introduction

Maintaining software quality presents several problems. If you haven't been using the 'test driven' approach you will probably recognise the following problem:

When the software project is almost completed it can happen that a few nasty bugs have been left. First the developers fixes bug 1, then bug 2 is fixed, etc. How can the programmer be sure that fixing bug 2 does not reintroduce bug 1? Or even worse if the solution introduces yet another bug 3?

On the other hand the programmer has been doing a few tests while fixing bug 1 to check if the fixes solved the bug. If it would be possible to rerun these test the programmer could be certain that the fix for bug 1 is still working after applying other fixes.

In the world of PC software there are several frameworks which can be used to automatically run tests after changes are made to the software source code. Eg. JUnit tests in the Java world. During our work on embedded software we ran into several problems:

1. We did not find any extensive testing framework 2. The existing frameworks (eg. Perl test harnesses) need the sources to be adapted for testing.

Especially embedded software which runs on simple microprocessors with limited resources is sensitive to changes needed for tests. For example debugging messages needs external IO via eg. RS232 and needs access to printf routines.'Printf' can use precious program memory. Even more the debugging messages themselves need to be stored. Last but not least the "C" language is not very expressive. For even the most simple tests it can happen that large amounts of utility code need to be programmed. We have developed an approach which should solve these issues.

• The original source code is annotated with test calls through comments.
• Python is used to described the tests.
• GDB is used as an interface between the tested code (the 'inferior' program) and the Python test

Example

The idea is illustrated with an example. The following program implements a simple function called "swap" which switches the contents of its two arguments:

 int swap(int* a, int* b) {
 	int tmp;
 	tmp = *a;
 	*a = *b;
 	*b = tmp;
 	return 0;
 } 
 
 int main(int argc, char **argv) {
   printf("swap prog \n");
   int a = 20;
   int b = 5;
   // TST.CALL: test1.swap_start(a,b)
   //	self.a = a
   //	self.b = b
 	swap(&a,&b);
 	printf("swap (2,5) = (%d,%d) \n",a,b);
   // TST.CALL: test1.swap_end(a,b)
   //	assert(self.a==b)
   //	assert(self.b==a)  
 	return(0);
 }

The code has special comments which surround the call to the "swap" function. The "swap" program should be compiled as normal (with debugging information). We can use Atfes to generate a test script:

 python swap/test_scripts/env1.py -g -d -e /usr -r swap/report.txt -s prog:swap/swap:test1:swap/test_scripts/test1.py 

The environment "env1.py" script is written by the programmer and acts as a container for the tests. The output of the test is written to a report.txt. All the test calls with id "test1" are added to a script test1.py. The script is the interface to the inferior swap/swap. This definition is added under id "prog" to the environment. The "-g" option instructs Atfes to only write the "test1.py" script. It will not run the test. Atfer execution a script "test1.py" has been created:

 class test1:
 	locations={\
 		"swap_start" : ("/home/dinne/atfes/trunk/examples/swap/swap.c",32),\
 		"swap_end" : ("/home/dinne/atfes/trunk/examples/swap/swap.c",38),}
 
 	@generated()
 	@breakpoint()
 	def swap_start(self,b,a):
 		self.a = a
 		self.b = b
 
 	@generated()
 	@breakpoint()
 	def swap_end(self,b,a):
 		assert(self.a==b)
 		assert(self.b==a)  
 testclass = test1

We can now run the test. Atfes will setup GDB to start the inferior and will insert breakpoints at the appropriate places. When the inferior hits the "swap_start" breakpoint Atfes will pass the contents of the inferior variables "a" and "b" to the test script. In the script you can use them as needed. When the script function exits the inferior is continued. In the second function "swap_end" it is possible to check if the inferior "swap" function has correctly switched the contents of "a" and "b". If this is not the case the assert call will raise an exception. The exception will be written to the report and the test will fail.

The example above illustrates a number of features. Other interesting features which are described into detail in the "reference" chapter are:

• Run multiple inferiors simultaneously. For example it is possible to implement test cases of the consumer-producer problem: The output of one inferior can be captured by the test and fed into the other inferior.
• Is is possible to write the python test in the "C" source comments. Or the code can be written in the test script. It is thus possible to alter the test script without your changes getting overwritten.
• The data of the inferior is easily accessible using normal Python syntax.

Installation

Atfes has been used in the following setup:

1. Install Cygwin 2. Install Python via Cygwin setup 3. Install make, gcc using Cygwin setup 4. Install PyGGy. This is a Python parser generator which can be found in the atfes/thirdparty folder. Install it with python setup.py install 5. Install Atfes: Go into the atfes/trunk folder and run "python setup.py install" 6. Compile examples: Go into atfes/trunk and run "make examples" this will create the example executables 7. Run the tests: cd atfes/trunk; make tests

Reference

Command line

Atfes is a command line utility with the following syntax:

 python env.py [-d] [-g] [-e path] [-r filename] -s id:inferior:test_id:testscript.py [-s more tests]
   env.py : an environment script
   -d : gdb input/output is written to a log
   -g do not execute test, only generate/update test scripts
   -e path: exclude sources on path
   -r filename : test report written to 'filename'
   -s testdefinition : add a test definition. Definition consists of identifier:inferior program:test id:test script filename

eg.:

 python env.py -d -e /usr -r report.txt -s producer:producer.exe:test1:producer.py -s consumer:consumer.exe:test1:consumer.py

Test definitions

Multiple test definitions can be given. Each test definition consists of 4 parts:

1. the definition id. 2. the path to the inferior executable. 3. the test id. 4. the path to the to be generated/updated python test script.

• The definition id is used to allow one test script to access other testscripts.
• The test id is used to filter test calls from the inferior source code.

For example the following two definitions:

 -s prod:comm/producer:producer:comm/scripts/producer.py -s cons:comm/consumer:consumer:comm/scripts/consumer.py 

will start the test with two running inferiors:

• comm/producer is started. The tests filtered with id "producer" will be placed in the script "comm/scripts/producer.py". This test script will be available through the environment through the id "prod".
• comm/consumer is started. The tests filtered with id "consumer" will be placed in the script "comm/scripts/consumer.py". This test script will be available through the environment through the id "cons".

Environment script

The first "env.py" argument should point to a user written environment. The environment acts as a container for all inferior programs. All test scripts have access to the environment. Furthermore the environment allows access from one inferior to other inferiors.

The environment can be any user written class. The class can define a number of hooks:

 class env:
 	def testsAssigned(self):	
 		pass
 	def inferiorsAssigned(self):
 		pass
 	def allInferiorsExited(self):
 		pass

Furthermore Atfes will add several attributes to the environment instance:

• report : This a Report object which can be used to write log messages during the test execution
• testdefinitions : this is a list of tuples.
  • When "testAssigned" is called this list consists of (definition id,inferior executable name, test name, script instance)
  • WHen "inferiorsAssigned" is called this lists consists of (definition id, gdb instance, test name, script instance)
• scripts : a dictionary which contains the definition id as key and the corresponding test instance as value. The dictionary is accesible after testsAssigned is called.

• The "testsAssigned" method is called after reading the test definitions and parsing the generated test scripts.
• The "inferiorsAssigned" method is called when all gdb's have been starten and the inferiors are read to start execution at their main function.
• "allInferiorsExited" is called when all gdb instances (and thus all inferiors) have exited.

Embedding test calls

Test scripts form the interface between the environment class and the inferior program. The test scripts are generated/updated by Atfes. GDB is used to list all sources constituting the inferior program. The sources (except those on the exclude path) are analysed. If certain comments are found action is taken. These comments are of the form:

// TST.CALL: xxxx.yyyy(a,b,c) //[tab]Python code

If a test definition was given on the command line: -s id:inferior:xxxx:script.py

then the a Python method "yyyy" will be created. "xxxx" is used to filter out comments. The following comment lines which start with a [tab] are treated as Python code and are added to the generated method. The arguments a,b,c,... are used to transfer values from inferior variables to the Python code. For example if we have C code:

 int x = 10;
 int y = 20;
 // TST.CALL: sometest.checkvalues(x,y)
 //	z = x + y

and a test definition -s id:inferior:sometest:script.py

this will create a Python script:

 class script:
 	def checkvalues(self,x,y):
 		z = x + y

when "checkvalues" is called "z" will get the value 30.

Note that it is possible to only add TST.CALL comments without the Python method body code. In this case the Python code must be added directly to the generated/updated Python class.

Test scripts

The generated Python test class will look like this:

 class test1:
 	locations={\
 		"method1" : ("/home/source.c",15),\
 		"method2" : ("/home/source.c",38),}
 
 
 	@generated()
 	@breakpoint()
 	def method1(self):
 		self.c = 0
 
 	@generated()
 	@breakpoint()
 	def method2(self,b,a):
 		self.a = a
 		self.b = b
 
 

Every generated method is annotated:

• "@breakpoint" indicates that a breakpoint will be set through gdb in the inferior. The source file and line number is obtained from the locations dictionary at the beginning of the class.

• "@generated" indicates that the body of the method is replaced by the contents of the comment body from which the method was generated. Any manual modification of the Python code is lost.

In stead of "@generated" it is also possible to change this manually to "@readonly()". If a method is annotated with this directive its body will not be updated during script generation.

The "generated" and "readonly" annotations were added to allow the programmer to either specify the test in the comment or to specify it later in the generated Python code. This is especially hand when the Python code is complex and long. In such cases adding the Python code to the original comment would lead to unreadable C source code.

Instance attributes

After instantiation of the Python class a number of properties are added:

• "env" points to the environment instance.
• "gdb" points to the GDB wrapper which controls the inferior.
• "id" contains the definition id which was given on the command line.
• "testround" returns the filter which was used to filter out the generated methods during the script's generation.

Inferior control

In the Python code one can access the inferiors variables through the "vars" attribute:

 self.vars.x = 10

This will access the variable "x" of the inferior and sets it to 10.

 self.vars.x.a = 10

This will access the variable "x" of the inferior and will set the "a" field of the structure to 10.

 self.vars.x.cast(tp,"a")

This will cast a variable "a" to the type 'tp'.

If you want to access the array through the pointer 'ptr'

Example (C code):

 int array[10];
 int* ptr = array;

Example (Python code):

 x = self.gdb.evalExpr("(*ptr@10)")
 x[5] = 1 

 self.vars.z[10] = 2

This will assign the value 2 to array element 10 of the inferior array "z".

GDB wrapper methods

Via the "gdb" property some addition control is possible:

  x = self.gdb.evalExpr("callmethod()")

This will execute the given statements through gdb. It is possible to evaluate exprerssions

  self.gdb.setTestDone()

This will exit the inferior and gdb.

Reporting

Examples

Problem solving

TODO

• Manual
• Report problems
• Hooks
• More examples
  • Synchronous consumer/producer
• Cleaning up debug info
• GDB via JTAG
• Adding/Removing breakpoints