Mini REST+JSON benchmark: Python 3.5.1 vs Node.js vs C++

Some time ago at Nokia I voluntarily developed a search engine tailored for internal resources (Windows shares, intranet sites, ldap directories, etc..) - NSearch. Since then few people helped me to improve it so it became unofficial "search that simply works". However, as you can image, this was purely a side-by project so I didn't pay attention to quality of the code much (I'd love to do so, but cruel time didn't permit :/). As a result during passing months a lot of technical debt was borrowed.

Recently we came to a conclusion that it's enough. We agreed that the backend of the service is going to be first in line. Because all of it was about to be rewritten we thought that maybe it's a perfect time to evaluate other technologies. Currently it's using Python2.7 + Django + Gunicorn. We consider going to either Node.js with Express 4, Python3 with aiohttp or C++. Maybe other language would be even a better match? However, we don't program in any other languages on a daily basis...

In this post I'd like to show you results of my very simple evaluation of performance of these three technologies along with some findings.

Technical facts

• everything was tested on machine with Intel E5-2680 v3 CPU and 192 GBs of RAM running on RHEL-7.1 OS,
• applications were run from under Docker containers, so there might be some overhead introduced by libcontainer, libnetwork, etc.
• ab was used for benchmarking with 1M of requests with different concurrency settings
• max timeout was set to 1 second

Attention: I'm not an expert in performance testing. Thus it's possible that I made some mistake. Source code of all inspected programs is available here. I recommend you to have a brief look at it. In case you find anything wrong or suspicious that might have influenced my results please let me know. Thanks.

I prepared two JSON objects used for two following benchmarks. The first was simple {"hello": "world!"} and the second was extracted directly from NSearch and connsisted of about 10k of characters, but I can't include it here. For the C++ I used Restbed framework and jsoncpp library, just to have anything with normal URL path support (otherwise results wouldn't be reliable at all).

First benchmark - conclusions:

• all solutions are asynchronous and event based and they're using event loops. otherwise it wouldn't be the case that 512 concurrent users can be served with max timeout set to 1 second
• starting from 192 concurrent users amount of requests per second starts to decrease slightly 
• Python is more than two times slower than Node.js
• C++ is more than two times faster than Node.js

Quite interesting, isn't it? I was hoping Python 3.5.1 with language async support and it's asyncio module will be faster. I was also anticipating that C++ will be about 30% faster and not 2.x time faster. Again I must admit that the intuition is deceptive.

Second benchmark - conclusions:
(I excluded C++ because of convenience of filling JSON object)

• the gap between Python and Node.js is much smaller when bigger JSON is in question
• apparently starting to handle a request in Python is slow, at least compared to Node.js
• replacing json.dumps with ujson.dumps increases Python performance by about 5%
• Node.js performance drops drastically when bigger JSON is used - from over 5k of requests per second to about 1700!
• Python's drop is not that drastic - from about 1700 to 1200 requests per second. It means that when the handling is ongoing, Python is not slowing down.

Why Python is so slow and Node.js much faster?
Python is interpreted language. This is main reason why it's so slow. Why it's not the case with Node.js? Because Node.js uses V8 - JavaScript interpreter - which has built-in JS JITter. JITter means Just-In-Time compiler which can speed up execution of a program by order of magnitude.

Can Python be faster? Yes!
There's also Python interpreter with built-in JITter - PyPy. Unfortunately it doesn't support 3.5.1 version of Python yet.

CMake disservice: project command resets CMAKE_EXE_LINKER_FLAGS

tl;dr at the bottom
CMake version: 3.3.2 

I think that everyone can agree that in the world of programming there are a lot of frustrating and irritating things. Some of them are less annoying, some are more and they are appearing basically at random resulting in a waste of time. Personally I think the most depressing and worst thing that one can encounter is implicit and silent weird side effects. I was hit by such a side effect today at work and would like to write about is as well as (possibly) help future Googlers.

So let me go straight to the topic. Recently platform software that is used in one of our medium-size C++ projects has changed and started to require some additional shared library. Our project stopped to compile because this new dependency wasn't listed in link libraries. We thought that the fix will be as simple as appending one item to the list, but we were proven wrong. Apparently this new shared library was linked with several other shared objects and this complicated things a little bit.

So the linker could be potentially satisfied just with the dependency itself, but this is not what happens (at least with GNU ld). The linker tries to minimize a chance that some symbols will be unresolved in run-time, so it checks whether all of the symbols are in place, even those coming from the dependent shared object! This effectively means that the linker will go through all of the unresolved symbols in every shared object and check them, regardless whether it's needed or not. I think that this is a nice feature, but there's one caveat - the linker doesn't actually know where to look for sub-dependent shared objects. This must be explicitly specified.

And here comes the unintuitive thing - rpath-link. Let me start with rpath, though. When the program is built and then run, the run-time linker (a.k.a loader) will look for all libraries linked to it and map them to the process memory space. There is specific order where the linker will look for the libraries: it will honor rpath in the first place, then it will consider LD_LIBRARY_PATH environment variable and finally standard system places. The rpath-link is similar to rpath, but... it's related to link-time and not run-time (this is this not intuitive part, or is it?). The linker will utilize rpath-link to look for sub-dependent libraries (Fig. 1).

Figure. 1. Flags for project dependencies and dependencies' dependencies (sub-dependencies)

Okay, but what this all has to do with CMake?

In CMake if you want to provide linker flags you need to utilize CMAKE_*_LINKER_FLAGS family of variables. And it's working perfectly fine unless you need to provide custom toolchain and sysroot (e.g. you are doing crosscompilation). Traditional way to provide custom toolchain with CMake is to pass -DCMAKE_TOOLCHAIN_FILE to cmake command. CMake will then process the toolchain file before CMakeLists.txt allowing to change compilers, sysroots etc. It's worth mentioning here that toolchain file will be processed before project command.

As it turned out, it works flawlessly for all of the variables like CMAKE_CXX_FLAGS etc. but CMAKE_*_LINKER_FLAGS variables. It took us some time to discover that... call to project command actually resets value of the CMAKE_*_LINKER_FLAGS variables. What a pesky side effect! After some digging I've found the culprit in file Modules/CMakeCommonLanguageInclude.cmake:

# executable linker flags
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS_INIT}"
     CACHE STRING "Flags used by the linker.") 

As you can imagine CMAKE_EXE_LINKER_FLAGS_INIT is not set to anything reasonable. So the result of this is that CMake doesn't care what you had in CMAKE_*_LINKER_FLAGS. It simply drops what've been there and replaces it with empty string. In one word - disservice.

This is not end of the story, though. CMake had one more surprise for us - to our misfortune. As some of you know, there's this useful variable_watch command available in recent versions of CMake. It allows to watch variable for all reads and writes. At the beginning we were sure that something within our build system is changing our CMAKE_EXE_LINKER_FLAGS so we used variable_watch to figure out what's going on. However it didn't indicate any modification. In the end it turned out that it does not signal anything that happens under the bonnet (e.g. it ignores what project command does to observed variable). This is a joke!

Building systems like CMake is definitely not an easy task. Especially if you are targeting lots of users and support lot of scenarios ranging from compiling simple one-file project to cross-compiling huge project with huge amount of dependencies. In our case several things contributed to the final effect, but I wouldn't say that custom linker flags in a toolchain files is a corner case. I think it should be definitely fixed some day.

Or maybe this is a feature and I'm the dumb one?

tl;dr:

cmake_minimum_required(VERSION 3.0)
set(CMAKE_EXE_LINKER_FLAGS "-Wl,-rpath-link=/opt/lib64") 
project(GreatProject)
# tada! CMAKE_EXE_LINKER_FLAGS is now empty. 

You know you're no-life when you do things like this during new year's eve :)

Happy 0x7e0 folks! :D

Obvious assumptions can result in bugs in software: Flask in embedded world

So the other day our team at Nokia was requested to implement web user interface for some embedded device. Usually I write in C++, but I was excited to take this task because we were able to put Python in place. Of course it's also possible to write web applications purely in C++ (e.g. with Cheerp - I encourage you to at least read about this project), but only thin back-end was required, so going with C++ didn't seem to be good choice. After cross-compiling Python to ARM (we anticipated that knowing how to do this would be common knowledge in the Internet, but again there were few resources) it was a time to pick some framework. We decided to use Flask - small cousin of Django, because

• it's smaller than Django and we were already going to eat a lot of disk space with Python itself
• it seemed like a perfect candidate for the task, because it looked so simple

I had my own goal as well :) I didn't know Flask, so there was something new to learn.

Fast forward to a place when I was about to deploy my application on the target device (after long struggle to have everything cross-compiled). Everything was set up, so I eagerly clicked the button in the front-end (react.js-based stuff) to prove that the back-end is working as well. As you can imagine - it didn't work.

http://thenextweb.com/wp-content/blogs.dir/1/files/2015/09/fn63697_01.jpg

Following is what I saw in logs. Some paths were masked, just-in-case. So after looking at this I thought to myself what the... It was working on my x86 machine!

In the rest of this post I'd like to describe a reason of this failure, which I found somewhat funny.

192.168.255.*** - - [01/Jan/2004 02:00:22] "POST /************************* HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1479, in full_dispatch_request
    response = self.process_response(response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1693, in process_response
    self.save_session(ctx.session, response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 837, in save_session
    return self.session_interface.save_session(self, session, response)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/sessions.py", line 326, in save_session
    val = self.get_signing_serializer(app).dumps(dict(session))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 569, in dumps
    rv = self.make_signer(salt).sign(payload)
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 412, in sign
    timestamp = base64_encode(int_to_bytes(self.get_timestamp()))
  File "/****...****/Python2.7.5/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg/itsdangerous.py", line 220, in int_to_bytes
    assert num >= 0

Ok, so normally I'm reading tracebacks in a bottom-up fashion. After looking at the last item in the traceback I instantly felt that something really weird is happening. Name of the file wasn't making things less weird as well. Items above the last one are not telling much more, so I had to dive in into source code to look for other hints.

Function int_to_bytes, at the very beginning, asserts that the input number is positive. Quite normal thing to do, right? So I started a debugger to check what value it got instead. I hunched that it would be -1 or something like that. Dark clouds were already in my mind at the time. I was even afraid that the hardware itself does not support some feature. That's not rare in embedded world, after all.

With the debugger I was able to retrieve value passed in. Apparently it was -220917729. Immediately I realized that I'm facing some issues with the time, and Python interpreter itself is not to be blamed. So I jumped into get_timestamp to see how the timestamp is created.

def get_timestamp(self):
    """Returns the current timestamp.  This implementation returns the
    seconds since 1/1/2011.  The function must return an integer.
    """
    return int(time.time() - EPOCH)

I double-checked that time.time() gives positive number. So after factoring out call to time() was obvious that the EPOCH is making the result negative. Then I needed to assemble two facts: EPOCH in this function is set to 1/1/2011 and the target device starts with time set to 1/1/2004. This yielded negative timestamp and a spectacular failure in result.

So I find this one funny, because apparently the authors didn't expect that the system time could be set to 2004-something :) Perhaps they assumed that it's not possible to run with a date earlier than the day when the library was published :)

Nevertheless I was expecting errors to appear in my configuration. You don't deploy web applications on embedded devices that often. Frankly speaking I encountered other funny problems during the day as well. One of them was with nginx, and if the time permits I'll write post about it as well. Stay tuned ;]

Note: I intentionally skipped description of internal Flask organization. In short words Flask uses ItsDangerous package to have signed session storage at the client-side. You can read more about it here.

Video: Check if number is a palindrome in Python

So few months ago I posted this post about finding whether given number is a palindrome in Python using as little characters as possible. Out of the blue the team behind Webucator made great video basing on my blog post. To be honest I didn't anticipate such a thing and I'm very happy that it happened.

The video itself is excellent so I encourage you to watch it.

On their site you can find more videos. I believe it's also worth to mention that they offer pretty comprehensive Python classes.

Python: palindrome number test challenge

Recently, in one of the check.io missions i was playing with a task to find first palindromic prime number (palprime) greater than given argument. The challenge was to make code as small as possible. To check whether a number is a palprime we need to check whether it is both a prime and a palindromic number. In this post I'd like to focus on checking whether a number is palindromic.

Spoiler alert: think twice before reading this post if you're going to play the game.

(source: http://40.media.tumblr.com/42ae198800aa76ad3c06878fd93a2d82/tumblr_nf8x7keJkF1rkt29io1_500.png)

I believe most Python devs out there would quickly come with intuitive solution, presented below.

str(n)==reversed(str(n))

The approach here is pretty straightforward - instead of doing arithmetics (like here) we can simply convert the number to a string and then compare it with reversed version of it. The solution consists of 24 characters. Can we make it smaller?

(source: http://gemssty.com/wp-content/uploads/2013/03/cardboard_phone_2.jpg)

It will be obvious to seasoned Python programmers to use fabulous Python slices, as shown below.

str(n)==str(n)[::-1]

With this simple transformation we saved 4 characters - that's nice. How does it work? Python slices allow specifying start, stop, and step. If we omit start and stop and provide -1 as a step slicing will work in magical way: it will go through the sequence in right-to-left direction. As a side note I can add that slices in Python work in a little weird way - e.g. "abcd"[0:len("abcd"):-1] gives "" in result while "abcd"[len("abcd"):0:-1] renders "dcb". [::-1] specifying start and stop parameters would require us to append firs character to the result: "abcd"[len("abcd"):0:-1]+"a".

So, going back to the topic, are we stuck with this solution? If we can invert the logic (that won't be always the case), then there's other way that I discovered during trials. It required me to adapt rest of the code, but no single character was introduced. Instead, I saved one more.

n-int(str(n)[::-1])

In essence, the idea is that if the number is palindromic then subtracting reversed version of it from the original version should result in 0. Here we are converting to string just to reverse the number and then there's back-to-integer conversion.

So I ended up with 19 characters thinking about what else I can do. I literally ran out of ideas, so I though of googling for operators available in Python language. During my life I managed to ensure myself that even if you think you have mastered your language of choice, you are wrong (C++ helped me to shape this attitude with stuff like template unions, secret shared_ptr c-tors etc. ;)).

In the haystack of results I found a needle. It turns out that in Python 2.x there's a possibility to convert a number (even if it's held in a variable) to a string using backticks (a.k.a backquotes, reverse quotes). Well... what can I say. I never expected existence of such thing, especially in Python.

Fortunately it does the job, so was able to use this feature and cut down length of the test to 14. Awesome!

`n`==`n`[::-1]

Above solution is the shortest I was able to come with. I wouldn't be surprised if there were shorter ones, though. If you know how to make it shorter, please let me know in comments.

Besides making this check so short I was curious what are differences between conversions and reversions in terms of performance. I decided to give it a go on my machine (i5-3320M 2.6GHz).

python -m timeit 'n=2**30; str(n)'
python -m timeit 'n=2**30; repr(n)'
python -m timeit 'n=2**30; `n`'

python -m timeit 's=qwertyuiopasfghjklzxcvbnm; rs=''.join(reversed(s))'
python -m timeit 's=qwertyuiopasfghjklzxcvbnm; rs=s[::-1]'

My conclusions are:

• there's very slight (almost unnoticeable) difference between three methods of converting integer to string
• for some reason Python3.4 is slower than Python2.7
• Pypy cruelly outperforms Python, as usual. If you care about performance you should use pypy.
• there's almost no difference between reverse(n) and [::-1] when it comes to speed

The fact that there are no performance differences between reverse() and [::-1] surprised me a bit. By looking into byte code I can only assume that in my particular case different set of operations yielded the same performance results.

reversed                             [::-1]
-------------------------------------------------------------------
 0 LOAD_CONST    0 ('qwer...')        0 LOAD_CONST    0 ('qwer...')

 3 STORE_NAME    0 (s)                3 STORE_NAME    0 (s)

 6 LOAD_CONST    1 ('')               6 LOAD_NAME     0 (s)

 9 LOAD_ATTR     1 (join)             9 LOAD_CONST    1 (None)

12 LOAD_NAME     2 (reversed)        12 LOAD_CONST    1 (None)

15 LOAD_NAME     0 (s)               15 LOAD_CONST    3 (-1)

18 CALL_FUNCTION 1 (1 pos, 0 kw)     18 BUILD_SLICE   3

21 CALL_FUNCTION 1 (1 pos, 0 kw)     21 BINARY_SUBSCR

24 STORE_NAME    3 (rs)              22 STORE_NAME    1 (rs)

27 LOAD_CONST    2 (None)            25 LOAD_CONST    1 (None)

30 RETURN_VALUE                      28 RETURN_VALUE

In case of conversion from integer to a string I anticipated that str and repr will yield similar results. What I didn't know was how backticks will perform. By looking into byte code I can say that the only difference is that backticks don't call any function, but use UNARY_CONVERT op directly.

str                                  ``
----------------------------------------------------------------------------
 0 LOAD_CONST    0 (1234567890)       0 LOAD_CONST            0 (1234567890)
 3 STORE_NAME    0 (n)                3 STORE_NAME               0 (n)
 6 LOAD_NAME     1 (str)              6 LOAD_NAME                0 (n)
 9 LOAD_NAME     0 (n)                9 UNARY_CONVERT
12 CALL_FUNCTION 1                   10 STORE_NAME               1 (s)
15 STORE_NAME    2 (s)               13 LOAD_CONST               1 (None)
18 LOAD_CONST    1 (None)            16 RETURN_VALUE
21 RETURN_VALUE 

CheckIo.org - A game for software engineers :)

I've just found an excellent way of spending an evening as a programmer - playing a game designed for software engineers :)

Since I discovered that site by an accident I think it is worth to mention it here so maybe you will also get interested. In the game you, in essence, solve various tasks to enable more "levels".

The only one drawback I can see is that only Python is supported (I'm Python enthusiast, but I know a lot of people that are more into Go, JavaScript etc.).

To give you a quick grasp without spoilers let me post here my solution for the very first (and very basic) task - FizzBuzz. This is the simplest one and can be quickly implemented, but I'm kind of a person that like to over-engineer.

return ''.join([['Fizz ', ''][(n%3)!=0], ['Buzz', ''][(n%5)!=0]]).strip() or str(n)

You think this is weird? Check out solution from nickie :)

return ("Fizz "*(1-n%3)+"Buzz "*(1-n%5))[:-1]or str(n)

Happy hacking!!