Obvious assumptions can result in bugs in software: Flask in embedded world

So the other day our team at Nokia was requested to implement web user interface for some embedded device. Usually I write in C++, but I was excited to take this task because we were able to put Python in place. Of course it's also possible to write web applications purely in C++ (e.g. with Cheerp - I encourage you to at least read about this project), but only thin back-end was required, so going with C++ didn't seem to be good choice. After cross-compiling Python to ARM (we anticipated that knowing how to do this would be common knowledge in the Internet, but again there were few resources) it was a time to pick some framework. We decided to use Flask - small cousin of Django, because

• it's smaller than Django and we were already going to eat a lot of disk space with Python itself
• it seemed like a perfect candidate for the task, because it looked so simple

I had my own goal as well :) I didn't know Flask, so there was something new to learn.

Fast forward to a place when I was about to deploy my application on the target device (after long struggle to have everything cross-compiled). Everything was set up, so I eagerly clicked the button in the front-end (react.js-based stuff) to prove that the back-end is working as well. As you can imagine - it didn't work.

http://thenextweb.com/wp-content/blogs.dir/1/files/2015/09/fn63697_01.jpg

Following is what I saw in logs. Some paths were masked, just-in-case. So after looking at this I thought to myself what the... It was working on my x86 machine!

In the rest of this post I'd like to describe a reason of this failure, which I found somewhat funny.

192.168.255.*** - - [01/Jan/2004 02:00:22] "POST /************************* HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1479, in full_dispatch_request
    response = self.process_response(response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1693, in process_response
    self.save_session(ctx.session, response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 837, in save_session
    return self.session_interface.save_session(self, session, response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/sessions.py", line 326, in save_session
    val = self.get_signing_serializer(app).dumps(dict(session))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 569, in dumps
    rv = self.make_signer(salt).sign(payload)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 412, in sign
    timestamp = base64_encode(int_to_bytes(self.get_timestamp()))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 220, in int_to_bytes
    assert num >= 0

Ok, so normally I'm reading tracebacks in a bottom-up fashion. After looking at the last item in the traceback I instantly felt that something really weird is happening. Name of the file wasn't making things less weird as well. Items above the last one are not telling much more, so I had to dive in into source code to look for other hints.

Function int_to_bytes, at the very beginning, asserts that the input number is positive. Quite normal thing to do, right? So I started a debugger to check what value it got instead. I hunched that it would be -1 or something like that. Dark clouds were already in my mind at the time. I was even afraid that the hardware itself does not support some feature. That's not rare in embedded world, after all.

With the debugger I was able to retrieve value passed in. Apparently it was -220917729. Immediately I realized that I'm facing some issues with the time, and Python interpreter itself is not to be blamed. So I jumped into get_timestamp to see how the timestamp is created.

def get_timestamp(self):
    """Returns the current timestamp.  This implementation returns the
    seconds since 1/1/2011.  The function must return an integer.
    """
    return int(time.time() - EPOCH)

I double-checked that time.time() gives positive number. So after factoring out call to time() was obvious that the EPOCH is making the result negative. Then I needed to assemble two facts: EPOCH in this function is set to 1/1/2011 and the target device starts with time set to 1/1/2004. This yielded negative timestamp and a spectacular failure in result.

So I find this one funny, because apparently the authors didn't expect that the system time could be set to 2004-something :) Perhaps they assumed that it's not possible to run with a date earlier than the day when the library was published :)

Nevertheless I was expecting errors to appear in my configuration. You don't deploy web applications on embedded devices that often. Frankly speaking I encountered other funny problems during the day as well. One of them was with nginx, and if the time permits I'll write post about it as well. Stay tuned ;]

Note: I intentionally skipped description of internal Flask organization. In short words Flask uses ItsDangerous package to have signed session storage at the client-side. You can read more about it here.

Video: Check if number is a palindrome in Python

So few months ago I posted this post about finding whether given number is a palindrome in Python using as little characters as possible. Out of the blue the team behind Webucator made great video basing on my blog post. To be honest I didn't anticipate such a thing and I'm very happy that it happened.

The video itself is excellent so I encourage you to watch it.

On their site you can find more videos. I believe it's also worth to mention that they offer pretty comprehensive Python classes.

Python: palindrome number test challenge

Recently, in one of the check.io missions i was playing with a task to find first palindromic prime number (palprime) greater than given argument. The challenge was to make code as small as possible. To check whether a number is a palprime we need to check whether it is both a prime and a palindromic number. In this post I'd like to focus on checking whether a number is palindromic.

Spoiler alert: think twice before reading this post if you're going to play the game.

(source: http://40.media.tumblr.com/42ae198800aa76ad3c06878fd93a2d82/tumblr_nf8x7keJkF1rkt29io1_500.png)

I believe most Python devs out there would quickly come with intuitive solution, presented below.

str(n)==reversed(str(n))

The approach here is pretty straightforward - instead of doing arithmetics (like here) we can simply convert the number to a string and then compare it with reversed version of it. The solution consists of 24 characters. Can we make it smaller?

(source: http://gemssty.com/wp-content/uploads/2013/03/cardboard_phone_2.jpg)

It will be obvious to seasoned Python programmers to use fabulous Python slices, as shown below.

str(n)==str(n)[::-1]

With this simple transformation we saved 4 characters - that's nice. How does it work? Python slices allow specifying start, stop, and step. If we omit start and stop and provide -1 as a step slicing will work in magical way: it will go through the sequence in right-to-left direction. As a side note I can add that slices in Python work in a little weird way - e.g. "abcd"[0:len("abcd"):-1] gives "" in result while "abcd"[len("abcd"):0:-1] renders "dcb". [::-1] specifying start and stop parameters would require us to append firs character to the result: "abcd"[len("abcd"):0:-1]+"a".

So, going back to the topic, are we stuck with this solution? If we can invert the logic (that won't be always the case), then there's other way that I discovered during trials. It required me to adapt rest of the code, but no single character was introduced. Instead, I saved one more.

n-int(str(n)[::-1])

In essence, the idea is that if the number is palindromic then subtracting reversed version of it from the original version should result in 0. Here we are converting to string just to reverse the number and then there's back-to-integer conversion.

So I ended up with 19 characters thinking about what else I can do. I literally ran out of ideas, so I though of googling for operators available in Python language. During my life I managed to ensure myself that even if you think you have mastered your language of choice, you are wrong (C++ helped me to shape this attitude with stuff like template unions, secret shared_ptr c-tors etc. ;)).

In the haystack of results I found a needle. It turns out that in Python 2.x there's a possibility to convert a number (even if it's held in a variable) to a string using backticks (a.k.a backquotes, reverse quotes). Well... what can I say. I never expected existence of such thing, especially in Python.

Fortunately it does the job, so was able to use this feature and cut down length of the test to 14. Awesome!

`n`==`n`[::-1]

Above solution is the shortest I was able to come with. I wouldn't be surprised if there were shorter ones, though. If you know how to make it shorter, please let me know in comments.

Besides making this check so short I was curious what are differences between conversions and reversions in terms of performance. I decided to give it a go on my machine (i5-3320M 2.6GHz).

python -m timeit 'n=2**30; str(n)'
python -m timeit 'n=2**30; repr(n)'
python -m timeit 'n=2**30; `n`'

python -m timeit 's=qwertyuiopasfghjklzxcvbnm; rs=''.join(reversed(s))'
python -m timeit 's=qwertyuiopasfghjklzxcvbnm; rs=s[::-1]'

My conclusions are:

• there's very slight (almost unnoticeable) difference between three methods of converting integer to string
• for some reason Python3.4 is slower than Python2.7
• Pypy cruelly outperforms Python, as usual. If you care about performance you should use pypy.
• there's almost no difference between reverse(n) and [::-1] when it comes to speed

The fact that there are no performance differences between reverse() and [::-1] surprised me a bit. By looking into byte code I can only assume that in my particular case different set of operations yielded the same performance results.

reversed                             [::-1]
-------------------------------------------------------------------
 0 LOAD_CONST    0 ('qwer...')        0 LOAD_CONST    0 ('qwer...')

 3 STORE_NAME    0 (s)                3 STORE_NAME    0 (s)

 6 LOAD_CONST    1 ('')               6 LOAD_NAME     0 (s)

 9 LOAD_ATTR     1 (join)             9 LOAD_CONST    1 (None)

12 LOAD_NAME     2 (reversed)        12 LOAD_CONST    1 (None)

15 LOAD_NAME     0 (s)               15 LOAD_CONST    3 (-1)

18 CALL_FUNCTION 1 (1 pos, 0 kw)     18 BUILD_SLICE   3

21 CALL_FUNCTION 1 (1 pos, 0 kw)     21 BINARY_SUBSCR

24 STORE_NAME    3 (rs)              22 STORE_NAME    1 (rs)

27 LOAD_CONST    2 (None)            25 LOAD_CONST    1 (None)

30 RETURN_VALUE                      28 RETURN_VALUE

In case of conversion from integer to a string I anticipated that str and repr will yield similar results. What I didn't know was how backticks will perform. By looking into byte code I can say that the only difference is that backticks don't call any function, but use UNARY_CONVERT op directly.

str                                  ``
----------------------------------------------------------------------------
 0 LOAD_CONST    0 (1234567890)       0 LOAD_CONST            0 (1234567890)
 3 STORE_NAME    0 (n)                3 STORE_NAME               0 (n)
 6 LOAD_NAME     1 (str)              6 LOAD_NAME                0 (n)
 9 LOAD_NAME     0 (n)                9 UNARY_CONVERT
12 CALL_FUNCTION 1                   10 STORE_NAME               1 (s)
15 STORE_NAME    2 (s)               13 LOAD_CONST               1 (None)
18 LOAD_CONST    1 (None)            16 RETURN_VALUE
21 RETURN_VALUE 

CheckIo.org - A game for software engineers :)

I've just found an excellent way of spending an evening as a programmer - playing a game designed for software engineers :)

Since I discovered that site by an accident I think it is worth to mention it here so maybe you will also get interested. In the game you, in essence, solve various tasks to enable more "levels".

The only one drawback I can see is that only Python is supported (I'm Python enthusiast, but I know a lot of people that are more into Go, JavaScript etc.).

To give you a quick grasp without spoilers let me post here my solution for the very first (and very basic) task - FizzBuzz. This is the simplest one and can be quickly implemented, but I'm kind of a person that like to over-engineer.

return ''.join([['Fizz ', ''][(n%3)!=0], ['Buzz', ''][(n%5)!=0]]).strip() or str(n)

You think this is weird? Check out solution from nickie :)

return ("Fizz "*(1-n%3)+"Buzz "*(1-n%5))[:-1]or str(n)

Happy hacking!!

KnowCamp Wrocław / More functional C++14

Video relation from the first KnowCamp organized by Nokia and held in Wrocław.
My presentation "More functional C++14" is available below (link here). BaSz's presentation "C++14 prostszy niż kiedykolwiek" (eng."C++14 simpler than ever") is here, but only in Polish.

More functional C++14 from szborows

Real Boost.multi_index performance

Update 09.05.2015: thanks to Joaquín it turned out that I had slight bug in my plotting script - x tick labels were wrong. All pictures have been fixed.

Several months ago I prepared and conducted a presentation about multi_index library from Boost at Nokia TechMeetUp. Essentially I showed that it is pretty useful library that should be used when there's a need to represent the same data with multiple "views". For instance, if we have to access the same list of something sorted by different attributes, Boost.MultiIndex seems to be very good choice as the documentation claims strong consistency and, what's also important, better performance. What's also interesting is that Facebook's library - folly - utilizes that in its TimeoutQueue class.

In the middle of the presentation I put some performance graphs which I mindlessly copied them from the documentation. Had I known that they are old, I'd never do that. Fortunately it was pointed out by friend of mine. These graphs in fact illustrate performance gains, but measurements were made with ancient compiler  versions like GCC 3.4.5.

After the presentation I promised myself that I'll investigate this a bit, but as usual time didn't permit. Till now.

I made measurements with newer versions of GCC and Clang. Unfortunately I have no access to Visual Studio. If you can provide me VS stats I'd be pleased, though.

Here are my results for:

CPU: i5-3320M

RAM: 8GB

OS: Debian 7

Compilation flags: -O3

Boost version: 1.57

Images from documentation are placed at the right side.

In the case of ordered index it's visible that either standard containers were improved or compilers learned how to optimize code using them better. Nevertheless, there is visible trend showing that with more than 10⁶ entries, multi_index container could perform better than naive approach.

In case of two indexes - one indexed and the second sequenced situation changes. In the original chart from the documentation we can observe huge improvement in favor to multi_index starting from 10⁵ elements. My results are quite different - there is no performance gain with multi_index. For 10⁶ elements the difference is dramatic comparing the two diagrams: ~130% and ~30%.

Interestingly, on my machine multi_index performed better in case of only 1 sequenced index. This is surprise for me, because in the documentation it is opposite. Well, in this scenario multi_index outperformed naive approach.
Rest of diagrams are presented below. As you can see they illustrate that multi_index documentation is outdated. Sometimes there are some performance gains, but not that huge as presented in diagrams from documentation.

Undoubtedly this deserves deeper investigation. Especially it should be discovered why something that had to perform better stopped to do so.
Unfortunately currently I have no time to dig in into this. (Maybe in next days I'll have some time to check how cache-friendly multi_index library is). Nevertheless, everything that I showed pushes me even harder into opinion that everything should be measured - even 3rd-party things with a lot of promises in the documentation.

image source

Software engineers - the new working class

Is software engineering a prestige occupation? Do programmers have any special abilities that are not possessed by the rest of society? Are there any particular predispositions that make someone a good programmer?

These, among others, questions appeared in my mind during another-in-the-row conversation about nothing with friend of mine. In this post I share my reflections about this topic by putting software engineers in pretty different segment of labor - proletariat (also referred as: working class, blue-collar workers).

Programmers and programmers
Just to make it clear - generally speaking I'm far for generalizing. I just hope that you agree with me that there are a lot sorts of programmers. On the one pole we have very creative and intelligent individuals while the other pole is a place filled with so-called code monkeys. However, there are exceptions. I'm not going to go into details with that. I'm going to focus more on how market demand affected occupation itself rather than saying who is who.

Blue-collar developers
Blue-collar software workers, in my opinion, are (or will soon be) the majority in computer software world. I anticipate that Pareto's rule would work here (80%/20%), but it's extremely hard to prove so I left it as a guess. But who are traditional blue-collar workers in a first place? In order not to make mistake in reasoning, let me stick to the Wiki definition of the proletariat.

The proletariat (/ˌproʊlɪˈtɛəriːət/ from Latin proletarius) is a term used to describe the class of wage-earners (especially industrial workers) in a capitalist society whose only possession of significant material value is their labour-power (their ability to work);[1] a member of such a class is a proletarian.

What does it say to us? Well, certainly one point is clear - the thing that blue-collar worker can offer to her employee is her labor power. Undoubtedly it was the case with industrial workers in the very beginning of the 20th century. It also applies to nowadays jobs, but programmers?

What an average programmer does?
One can simply say that a programmer writes computer software. Of course it's true, but I'd rather say that programmer thinks a lot. Primarily about solving problems. From this sentence it almost spills out that programmers must be very intelligent and knowledgeable creatures. Solving problems require strong analytic skills, doesn't it? This isn't necessarily the truth.

Problems and problems
There are big problems and small problems. Usually big problems are solved by highly intelligent persons with strong education background. This is because, for example, you won't sort 10TB of data with bubble sort algorithm, will you? And efficient sorting know-how is not ubiquitous knowledge. Also, such problems require that the solution smoothly scales. On the other hand majority of programmers deal with small problems involving small problem sets, because they can cope with them. In other words, regardless how stupid it would be, sorting 1kB of data with bubble sort won't hurt. More importantly this problem set will never grow. In such cases our blue-collar programmers feel like ducks in the water.
And perhaps there's nothing wrong with that, but it has interesting implication.

"Make it work"
It is often a good approach to make programs work, then make them work correctly, and eventually improve them so they are fast. The last step can be omitted, though. I believe that this is what we are facing nowadays. Only small part of programmers must care about performance - and it is likely that they are library writers.
So in "user code" it usually doesn't matter who and how will implement some functionality. It doesn't matter what kind of algorithm is under the bonnet. For almost all the time it will be fine.

That leads us to the following question. What such a programmer really needs to know in order to write software?

if, for, class, import
In my humble opinion the basic set of needed tools are: if for conditions, for for looping, class for representing relations, and import for including someone's else work into our project. Of course you should also know what are functions and variables and how to use them.
If you're not low-level programmer you won't need to know anything about operating system, memory management, and so on. Perhaps I missed some other useful knowledge, but this is the basic toolbox.

As you can see it's not that complicated. Thanks to that it is possible for a lot of people to write software. And... if you were conscious in the school you know that the more mass the more gravity force. Being a programmer is not exclusive anymore. This occupation becomes like all other occupations - just ordinary. It doesn't mean that there are no extraordinary people within programmers community, but it shows that programmers are becoming new working class. The only one thing that they offer is usually some kind of labor-power - like the Proletariat.

Number of programmers in the world is continuously increasing. Actually I don't believe we will see any decline in this area in next few years. In Britain the government even introduced programming course in a primary school! It's likely that in next years everyone will be a programmer - like everyone can read.

SDD - StackOverflow driven development
Things are getting even more funny when we think about the internet. Mainly StackOverflow and Google. Believe it or not, but how hard is to program something if you can simply ask Google a question and read an answer on StackOverflow? The other time you will find a blog post saying what to do in detailed steps. You don't even need to know all this stuff. All that is required is that you know how to use search engine and master Ctrl-C-then-Ctrl-V combination.

This topic is deeper. There are many other aspects that could be covered, but it would make this post too long. So... I leave you with these disturbing thoughts :)