wtorek, 8 marca 2016

Mini REST+JSON benchmark: Python 3.5.1 vs Node.js vs C++

Some time ago at Nokia I voluntarily developed a search engine tailored for internal resources (Windows shares, intranet sites, ldap directories, etc..) - NSearch. Since then few people helped me to improve it so it became unofficial "search that simply works". However, as you can image, this was purely a side-by project so I didn't pay attention to quality of the code much (I'd love to do so, but cruel time didn't permit :/). As a result during passing months a lot of technical debt was borrowed.

Recently we came to a conclusion that it's enough. We agreed that the backend of the service is going to be first in line. Because all of it was about to be rewritten we thought that maybe it's a perfect time to evaluate other technologies. Currently it's using Python2.7 + Django + Gunicorn. We consider going to either Node.js with Express 4, Python3 with aiohttp or C++. Maybe other language would be even a better match? However, we don't program in any other languages on a daily basis...

In this post I'd like to show you results of my very simple evaluation of performance of these three technologies along with some findings.

Technical facts

  • everything was tested on machine with Intel E5-2680 v3 CPU and 192 GBs of RAM running on RHEL-7.1 OS,
  • applications were run from under Docker containers, so there might be some overhead introduced by libcontainer, libnetwork, etc.
  • ab was used for benchmarking with 1M of requests with different concurrency settings
  • max timeout was set to 1 second
Attention: I'm not an expert in performance testing. Thus it's possible that I made some mistake. Source code of all inspected programs is available here. I recommend you to have a brief look at it. In case you find anything wrong or suspicious that might have influenced my results please let me know. Thanks.

I prepared two JSON objects used for two following benchmarks. The first was simple {"hello": "world!"} and the second was extracted directly from NSearch and connsisted of about 10k of characters, but I can't include it here. For the C++ I used Restbed framework and jsoncpp library, just to have anything with normal URL path support (otherwise results wouldn't be reliable at all).

First benchmark - conclusions:

  • all solutions are asynchronous and event based and they're using event loops. otherwise it wouldn't be the case that 512 concurrent users can be served with max timeout set to 1 second
  • starting from 192 concurrent users amount of requests per second starts to decrease slightly 
  • Python is more than two times slower than Node.js
  • C++ is more than two times faster than Node.js
Quite interesting, isn't it? I was hoping Python 3.5.1 with language async support and it's asyncio module will be faster. I was also anticipating that C++ will be about 30% faster and not 2.x time faster. Again I must admit that the intuition is deceptive.

Second benchmark - conclusions:
(I excluded C++ because of convenience of filling JSON object)

  • the gap between Python and Node.js is much smaller when bigger JSON is in question
  • apparently starting to handle a request in Python is slow, at least compared to Node.js
  • replacing json.dumps with ujson.dumps increases Python performance by about 5%
  • Node.js performance drops drastically when bigger JSON is used - from over 5k of requests per second to about 1700!
  • Python's drop is not that drastic - from about 1700 to 1200 requests per second. It means that when the handling is ongoing, Python is not slowing down.

Why Python is so slow and Node.js much faster?
Python is interpreted language. This is main reason why it's so slow. Why it's not the case with Node.js? Because Node.js uses V8 - JavaScript interpreter - which has built-in JS JITter. JITter means Just-In-Time compiler which can speed up execution of a program by order of magnitude.

Can Python be faster? Yes!
There's also Python interpreter with built-in JITter - PyPy. Unfortunately it doesn't support 3.5.1 version of Python yet.

6 komentarzy:

  1. Nice article. I've reproduced your experiments with node and aiohttp. On my Local machine node is about 2~3 times faster than the asyncio version.
    Have you done any experiments on aiohttp + api-hour? Since you have a processor with a lot of cores I think you could benefit from spawning multiple workers. Following this tutorial: http://pythonhosted.org/api_hour/tutorials/all_in_one.html#benchmark-the-difference-with-several-workers. With only 4 workers I was able to overcome node by 10%.

    1. Thanks!
      I didn't know about existence of API-hour. Good to know that such a thing exists.
      If time permits I will test it on my machine. I was also thinking about testing node with JITter turned off to see how big difference it makes. What do you think? Maybe you also have other ideas what to test?

    2. One more idea: Docker w/o virtual interfaces and rapidjson instead of jsoncpp :)

  2. Good article, interesting to see metrics of how a real application measures up in those languages.

    1. Thanks. Any idea how to simulate real application ;)? I was thinking about this, but it all boils down to business logic that's application is actually solving.
      However, maybe there are some "benchmark apps"?

  3. It would be fun to see how python fares with uvloop. I've heard it gives better performance than nodejs.