tisdag 7 november 2017

UTF-8 in Python

I have been working a lot with python lately. Especially building tools that convert from Excel to JSON and XML. The data have been translations and similar, so encoding has been a real issue. Generally, it works well in python, but when working in a Windows environment, it is easy to go wrong among the different encoding standards such as "UTF-8" and CP1252 and so on. I won't pretend to be an expert on this at all. I have just tried to get the stuff working without having to understand too much of it. So if you have any useful background information about the snippets below, please go ahead and share a comment.

Today this stole a couple of hours from me again, so I write this down as a memory annotation... once and for all!

When using openpyxl, all strings are converted to "UTF-8" when reading the data from the workbook, so there is no need to do any conversion there or try to find any obscure encoding settings in Excel. This far all strings, Chinese, Swedish or Hungarian, have arrived correct.

To Terminal

Getting it correct when writing them to the terminal was a bit of a trick though. The trick seems to be to always write the text as bytes. To do this to stdout I had to this:



When generating JSON-files it is important to make sure the json generator generates the right content by setting ensure_ascii=False.

Then write the file as a binary file. Whatever I do, things seem to go wrong if I decode the string before writing it.

with open(fileName, mode="wb") as jsonFile:
    jsonFile.write(bytes(json.dumps(dict_to_write, ensure_ascii=False), 'utf-8'))


Again, create a binary file and write an encoded byte array rather than decoding to a string. I have used pybx to create bindings from the XSD files. Pybx builds on the minidom libraries... I think.

from pyxb.namespace.builtin import XMLSchema_instance as xsi

bds = pyxb.utils.domutils.BindingDOMSupport()
dom = self._root.toDOM(bds)
bds.addAttribute(dom.documentElement, xsi.createExpandedName('schemaLocation'), "http://my_target_ns/ns1 ../xsd/my_schema.xsd")
bds.addXMLNSDeclaration(dom.documentElement, xsi)

if self._root is None:
    raise Exception("There is no root element to write to file.")

if self._fileName is None:
    self._fileName = os.path.join(os.getcwd(), 'xml/' + self._root.name + '.xml')

with open(self._fileName, 'wb') as xmlfile:
    xmlfile.write(dom.toprettyxml(indent='  ', encoding='utf-8'))

Good Luck!

måndag 10 augusti 2015

Clock setup for Olimex E407 using STM32CubeMX

Recently I got a reason to dig up my old Olimex E407 board and the RS232 shield I made for it (see earlier posts on this blog). This time though, I'm using Keil in Windows together with a ST Link V2. Getting the clocks and pll's right isn't an easy task, and certainly not a fun one, so I decided to give CubeMX and ST's HAL libraries a try.

The STM32CubeMX program (download it from ST.com) is a fairly new invention, and the libraries leave a lot to whish for still, so you should be careful with it. The libraries are still ful of bugs, but they are also getting a lot better rapidly.

Ok, first of course, download and install STM32CubeMX. If you want to launch it from Keil uVision, add the path to CubeMX to the system path environment variable. The start uVision5 and create a project for an STM407ZG.

Start CubeMX by clicking the "play" button right of the STM32Cube Framework (API).

Now, the Olimex E407 board is equipped with a 12 MHz external crystal clock and a 32.768 kHz external RTC crystal, so we have to enable those pins first.

Switch to the "Clock Configuration" tab and input 12 MHz as HSE input frequency and then make the following selections:
  1. Select HSE as PLL Soure Mux
  2. Set the /M divider to /7
  3. Set the *N multiplier to x196
  4. Set the /P divider to /2
  5. Select PLLSLK as System Clock Mux
  6. Set APB1 Prescaler to /4
  7. Set APB2 Prescaler to /2

Save it and select "Generate Code" from the Project menu.

When done, choose to close the CubeMX project and update the settings in uVision5.

A main.c file is generated for you that uses these new settings , ready to be filled in with your applicaiton source.

Good Luck

fredag 24 april 2015

A Comment on Comments

During recent years I have worked with many very tech savvy people worthy of a lot of respect for their insight in embedded development. However, the discussion regarding comments seems to never die. The more tech oriented the developers, the less they seem to like comments in source and the more they seem to believe self commenting code exist. It doesn't. What exist is self commenting thoughts in your head. I have no chance whatsoever to get a good grasp of your ideas without reading them, or asking you.

I have always taken a more customer focused approach, hoping that I won't have to explain myself too many times when I hand over the result of my efforts. Here are some things I would like to point out to all of you that doubt the usefulness code comments:

  • Code in itself can only describe what it does. Everything else are left out. All the experience you gained through experiments, fixing bugs and working with the environment is lost if you don't document it.
  • Code can't explain what it doesn't do. All the alternatives to the approach you have taken. All the quirkyness of the libraries you are using. That takes a long time to get your head around as a new employee or contractor.
  • Code doesn't explain its purpose. In best case, you can read out something from the function name or similar, but most of the time names like openAuxChannel leave you wandering WHY you would like to open an auxiliary channel.
There are lots of other reasons to write and maintain the comments in the source. But there is also the question of the cost for commenting code. If you comment your code you spend time that not feel very productive. In one sense that's correct, you don't write code, so you're not moving towards the goal of the project. But quality has to be considered too, and maintainability and tracability are such parameters. Without them, the quality of your code is poor. It eventually boils down to the cost of developing a feature vs. the cost of owning it. Over time, the cost of owning poor code is MUCH higher than the gain of developing poor code.

So, in the trade-off of commenting source and writing new features, I would like to promote some techniques that I practice:
  • Use doxygen! You might not like browse the source in rendered HTML or PDF documents, but it gives others a great overview and a starting point that lets them get get up to speed with the source quickly.
  • Write commenting scripts that puts skeletons of comments in your files, classes, members and functions. The cost of developing such a script is a lot less than the effort it saves. It takes away a lot of boring work as well! Often the IDE has some functionality for this, but I often get more fields into my doxygen comments with a script.
  • Don't comment getters and setters. Programmers are not stupid, they will understand anyway. Though, if you have a script putting doxygen comment on them, they will become browsable when the documentation is rendered, and that's very convenient.
  • Make sure you add comments to algorithmic code in the functions. These sections of comments are the most valuable. The not only explain your algorithm, it will affect your design too. If you notice that you have to explain flags and states set in objects or structures in other modules etc. it shows that your approach to the problem is probably not coherent enought, or easy enough. If you can't explain what you have done, you should refactor it.

torsdag 18 oktober 2012

Making Jersey work with Google App Engine

I was looking for a way to get Jersey work in a Google App Engine application that I am working with. Things seems to work just fine with the local test server, but deployed in the production environment it fails miserably.

The answer to the problem wasn't easy to find at all, but after plenty of googling i found a this post that the answer I was looking for.

I will here summarize the steps in a bit shorter post and hope that it will make this solution a little bit easier to find for others.

The problem boils down to the dependency of the asm jar file. The new versions of the GAE libraries use asm-4.0.jar but Jersey still depends on asm-3.1.jar. There are some posts out there that suggest that you can remove the asm-4.0.jar and change the data nucleus version to v1. WARNING!!! Doing this messed my eclipse up totally. I never got it to work properly again and had to reinstall it and all the plug-ins!

What Harry Wye suggests on his blog is to repackage the jar files depending on asm-3.1 using Jarjar Links. This worked fine for me using version 1.3. 1.4 gave me a manifest attribute error. I am not terribly experienced in Java, so I don't know why this happend or what to do about it (if you know... please share it in a comment on this page), but trying 1.3 did the work for me.

To repackage the jersey server jar (which is the only one depending on asm), you first have to create a file (rule.txt) with this content:
rule org.objectweb.asm.**  org.objectweb.asm3.@1

Then run jarjar:

java -jar jarjar-1.3.jar process testrules.txt asm-3.1.jar asm-3.1r.jarjava -jar jarjar-1.3.jar process testrules.txt jersey-server-1.14.jar jersey-server-1.14r.jar

Now you can deploy Jersey with the modified jars along with asm-4.0.jar in your GAE application.

Good Luck

måndag 1 oktober 2012

DIY RS232 shield for the Olimex E407

My first hardware project with the E407 was to create an RS232 shield (they can be bought if you don't want to build it yourself). I used olimed own arduino proto shield. I happened to have a Maxim 232 driver circuit in my drawer, and also the five 1uF electrolytic capacitors needed to create this board.

First of all I discovered that the Olimex proto shield doesn't fit the board very well. I had to grind off one of the corners to make it fit with the power supply capacitors on the E407 board. A regular proto shield won't fit better... it seems like the capacitors aren't too well placed on the E407 board :-/

Since I am not very experienced in programming the STM32 processor I had to verify the construction with an AVR I had laying around and it seemed to work fine. I couldn't make it work with the E407 though.

I found out that the source I had downloaded from ST's site is designed for the Discovery board which has an 8 MHz external clock source. The E407 has a 12MHz clock. I had to generate a new system file with ST's excel tool. The patch can be found here.

I then compiled the example from Elias electronics blog, which seemed to work fine.

Good luck!

tisdag 18 september 2012

Lean electronic prototyping: Prove your assumptions

Conventionally, when developing electronic devices, companies tend to focus on developing a prototype hardware as the first stage. This is a complicated and expensive strategy that consumes large quantities of energy and money in a design up-front.

A complete electronics development studio, open source only!
There are alternatives however. Today there are many open source circuit boards to choose from off the shelf, that suits almost any need for an embedded system. These boards are extremely cheap, and the tools for them are almost always open source, and thus, also free.

The maker movement and DIY (Do It Yourself) trend has brought embedded development to be a serious hobby alternative for anyone... even kids. And there isn't just toy functionality offered on these cheap hobbyist boards, it is real computer power and connectivity there, such as USB, Bluetooth, internet working, motor control. There are even circuit boards meant to be sewn into your clothes with conducting thread! All from the price of 20$ USD or so. In combination with mobile phones, cloud services and integration with other systems, these systems form a very powerful tool for innovation.

One of the most eye catching applications is off course the home 3D printers that has started to pop up in the market. For about 600$ USD, you get your own mechanical factory! The CAD-tools used for these are also often free and/or open source. With the 3D printers you can manufacture mechanical parts for your invention with very small means. This becomes a very important missing piece in the prototyping process that often tends to become the most expensive step when developing the prototype.

With the right prototyping strategy, a company can focus on developing quite advanced Minimum Viable Products that they actually can test in the hands of a real customer. This will give a much more trust worthy feedback, and will help the company to eliminate waste in the development process.

The sad truth is that the most important properties of a system are the least costly to develop. Many electronic devices are completely depending on a good the user experience, which in most cases are implemented in software. I'm not saying that the industrial design of a device is unimportant, on the contrary, but in the development of a minimum viable product, or prototype, you should focus on proving your assumptions regarding the customers needs/desires. To do this, the shortest and cheapest way is often to realize the solution in software. With the use of a lean prototyping strategy, the trade off between software functionality and industrial design can become less dramatic.

So what's the catch? ...

Well, there are off course down sides of these methods. To face the truth, the quality of the open source tools often leave a lot to wish for. Setting up a development environment based on open source CAN be done, but it takes it's fair amount of googling and experimenting to get it running. The documentation is often in a poor state, or none existing. This puts a big burden on the developers involved and the project becomes a lot more dependent on a high level of skills of the team members.

I would like to point out though, that during my time as a devloper and project manager in embedded systems projects, I cannot recall one single occasion when I have received any useful support from a company we have bought "closed source" tools from. So the increase in risk, is probably leveled out by the probability that you would actually get what you pay for, when using proprietary closed source software.

So keep it lean; focus on the customer, eliminate waste, keep the iterations short.

lördag 15 september 2012

Building the GCC tool chain for the STM32F4

So... I have to develop some applications on bare metal arm processors and need to practice a bit on a development board before I get started with the customer. The applications will be written in C, but I also want to investigate how C++ fits a project like this. The program is an audio application which eventually will use some filtering and asynchrounous USB transfer.

I have chosen an Olimex Board (STM32-E407) since it conforms to the Arduino form factor and has a processor close to the model used by my custormer. A JTAG was and a couple of development shields were ordered as well.

Before I can get started I need a toolchain to compile my applications. I could download a prebuilt toolchain from Mentor Graphics (Code Sourcery), but I wanted to see how if I could build one myself.

I have a 32 bit Ubuntu 12.04 host, and I tried to follow two other howto’s but none of them worked, so I had to mix the instructions from both. (http://www.triplespark.net/elec/pdev/arm/stm32.html, http://jeremyherbert.net/get/stm32f4_getting_started )

First install the tools needed for building the toolchain:
$sudo apt-get install flex bison libgmp3-dev libmpfr-dev libncurses5-dev libmpc-dev autoconf texinfo build-essential libftdi-dev libzip-dev

Now clone the source repository:

Note: See the update below for instructions for GDB XML support!
$git clone git://github.com/esden/summon-arm-toolchain
$cd summon-arm-toolchain

Since noone else will use my toolchain I will install the toolchain in my home directory: ~/tools/arm/arm-linaro-eabi-4.6.

Build the tool chain:
$./summon-arm-toolchain PREFIX=~/tools/arm/arm-linaro-eabi-4.6/ OOCD_EN=1 TARGET=arm-none-eabi LIBSTM32_EN=1 USE_LINARO=1 CPUS=2

When done, check the result:
$/bin/arm-none-eabi-gcc --version

It should say something like this:
arm-none-eabi-gcc (Linaro GCC 4.6-2011.10) 4.6.2 20111004 (prerelease)
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO

To be able to work efficiently on the command line I create an environment setup script. With this script I can have several different toolchains for different platforms, and different automation scripts, that doesn't pollute my systems PATH environment variables and bash setup. 

Open up a text editor and create the file env.sh, looking like this:

echo "Olimex STM32-E407 Development Environment"

#Sub shell
#Export the variables to use in the development environment
export PATH=~/tools/arm/arm-linaro-eabi-4.6/bin:$PATH
export PS1="[Olimex E407] \u@\h:\w\$ "

#Start a new shell with the exported variables

Change the mode of the file:
$chmod +x env.sh

Enter your environment:

To exit the environment shell, just type exit and you’ll be back to your normal prompt.

From here I followed the last sections in http://jeremyherbert.net/get/stm32f4_getting_started to build the first test software.

Good luck!

Update Sept. 17 2012
To be able to debug the target I hade to rebuild the toolchain with XML support enabled in the GDB target.

To do this you first need to install expat:
$sudo apt-get install libexpat1 libexpat1-dev

Then find the GDB build target in the automated build script named summon-arm-toolchain and add the parameter "--with-expat \" before the GDBFLAGS parameter (or add it to the GDBFLAGS variable).

Now build the tool chain according to the instructions above.