Post for googlers that stumble on the same issue - it seems that "overconfiguration" is not a great idea for Filebeat and Logstash.
I've decided to explicitly set ssl.verification_mode to none in my Filebeat config and then I got following Filebeat and Logstash errors:
filebeat_1 | 2017/01/03 07:43:49.136717 single.go:140: ERR Connecting error publishing events (retrying): EOF
filebeat_1 | 2017/01/03 07:43:50.152824 single.go:140: ERR Connecting error publishing events (retrying): EOF
filebeat_1 | 2017/01/03 07:43:52.157279 single.go:140: ERR Connecting error publishing events (retrying): EOF
filebeat_1 | 2017/01/03 07:43:56.173144 single.go:140: ERR Connecting error publishing events (retrying): EOF
filebeat_1 | 2017/01/03 07:44:04.189167 single.go:140: ERR Connecting error publishing events (retrying): EOF
logstash_1 | 07:42:35.714 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
logstash_1 | 07:43:49.135 [nioEventLoopGroup-4-1] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
logstash_1 | 07:43:49.139 [nioEventLoopGroup-4-1] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
logstash_1 | 07:43:50.150 [nioEventLoopGroup-4-2] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
logstash_1 | 07:43:50.154 [nioEventLoopGroup-4-2] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
logstash_1 | 07:43:52.156 [nioEventLoopGroup-4-3] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
logstash_1 | 07:43:52.157 [nioEventLoopGroup-4-3] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
logstash_1 | 07:43:56.170 [nioEventLoopGroup-4-4] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 3
logstash_1 | 07:43:56.175 [nioEventLoopGroup-4-4] ERROR org.logstash.beats.BeatsHandler - Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 1
It seems it's better to stay quiet with Filebeat :) Hopefully this helped to resolve your issue.
wtorek, 3 stycznia 2017
wtorek, 13 grudnia 2016
std::queue's big default footprint in assembly code
Recently I've been quite busy and now I'm kind of scrounging back into C++ world. Friend of mine told me about IncludeOS project and I thought that it may be pretty good exercise to put my hands on my keyboard and help in this wonderful project.
To be honest, the learning curve is quite steep (or I'm getting too old to learn so fast) and I'm still distracted by a lot of other things, so no big deliverables so far... but by just watching discussion on Gitter and integrating it with what I know I spotted probably obvious, but a little bit surprising thing about std::queue.
std::queue is not a container. Wait, what?, you ask. It's a container adapter. It doesn't have implementation. Instead, it takes other implementation, uses it as underlying container and just provides some convenient interface for end-user. By the way it isn't the only one. There are others like std::stack and std::priority_queue to name a few.
One of the dimension in which C++ shines are options for customizing stuff. We can customize things like memory allocators. In container adapters we can customize this underlying container if we decide that the one chosen by library writers isn't good match for us.
By default, perhaps because std::queue requires fast access at the beginning and end, it's underlying container is std::deque. std::deque provides O(1) complexity for pushing/popping at both ends. Perfect match, isn't it?
Well, yes if you care about performance at the cost of increased binary size. As it turns out by simply changing std::deque to std::vector:
std::queue<int> qd;
std::queue<int, std::vector<int>> qv;
Generated assembly code for x86-64 clang 3.8 (-O3 -std=c++14) is 502 and 144 respectively.
I know that in most context binary size is secondary consideration, but still I believe it's an interesting fact that the difference is so big. In other words there must be a lot of things going on under the bonnet of std::deque. I don't recommend changing deque to vector in production - it can seriously damage your performance.
You can play around with the code here: https://godbolt.org/g/XaLhS7 (code based on Voultapher example).
To be honest, the learning curve is quite steep (or I'm getting too old to learn so fast) and I'm still distracted by a lot of other things, so no big deliverables so far... but by just watching discussion on Gitter and integrating it with what I know I spotted probably obvious, but a little bit surprising thing about std::queue.
std::queue is not a container. Wait, what?, you ask. It's a container adapter. It doesn't have implementation. Instead, it takes other implementation, uses it as underlying container and just provides some convenient interface for end-user. By the way it isn't the only one. There are others like std::stack and std::priority_queue to name a few.
One of the dimension in which C++ shines are options for customizing stuff. We can customize things like memory allocators. In container adapters we can customize this underlying container if we decide that the one chosen by library writers isn't good match for us.
By default, perhaps because std::queue requires fast access at the beginning and end, it's underlying container is std::deque. std::deque provides O(1) complexity for pushing/popping at both ends. Perfect match, isn't it?
Well, yes if you care about performance at the cost of increased binary size. As it turns out by simply changing std::deque to std::vector:
std::queue<int> qd;
std::queue<int, std::vector<int>> qv;
Generated assembly code for x86-64 clang 3.8 (-O3 -std=c++14) is 502 and 144 respectively.
I know that in most context binary size is secondary consideration, but still I believe it's an interesting fact that the difference is so big. In other words there must be a lot of things going on under the bonnet of std::deque. I don't recommend changing deque to vector in production - it can seriously damage your performance.
You can play around with the code here: https://godbolt.org/g/XaLhS7 (code based on Voultapher example).
środa, 7 września 2016
elasticdiff
Recently I needed to compare two ElasticSearch indices. To be honest I was pretty sure that I'll find something in the Internet. It was a surprise that no such a tool do exist. I thought that this is good time to pay off portion of my debt to open-source :)
Enter ElasticDiff
elasticdiff is a simple project hosted on GitHub which allows you to compare two ES indices without pain. It is early development, so don't expect much, but for simple indices it works pretty well.
Usage is quite trivial:
python3 elasticdiff.py -t docType -i key http://1.2.3.4/index1 http://2.3.4.5/index1
Output was designed to imitate diff command:
Hopefully someone will find this tool useful.
More information available at GitHub page. I'm planning releasing this on PyPi when it gets more mature. I will update this post when this happens.
elasticdiff is a simple project hosted on GitHub which allows you to compare two ES indices without pain. It is early development, so don't expect much, but for simple indices it works pretty well.
Usage is quite trivial:
python3 elasticdiff.py -t docType -i key http://1.2.3.4/index1 http://2.3.4.5/index1
Output was designed to imitate diff command:
only in left: a5
only in left: a1
only in right: a4
only in right: a2
entries for key a3 differ
---
+++
@@ -1,4 +1,4 @@
{
"key": "a3",
- "value": "DEF"
+ "value": "XYZ"
}
Summary:
2 entries only in left index
2 entries only in right index
1 common entries
0 of them are the same
1 of them differHopefully someone will find this tool useful.
More information available at GitHub page. I'm planning releasing this on PyPi when it gets more mature. I will update this post when this happens.
poniedziałek, 25 lipca 2016
Elasticsearch cluster as Eucalyptus on RHEL 7.2, using Ansible
Just thought that this can be useful for someone else.
https://github.com/szborows/es-cluster-rhel7-ansible-eucalyptus
https://github.com/szborows/es-cluster-rhel7-ansible-eucalyptus
niedziela, 8 maja 2016
me @ ACCU 2016 - what every C++ programmer should know about modern compilers
In Poland we have this proverb "pierwsze koty za płoty"*. Basically we say it when we do something for the first time in our lives. After finding a translation, I was quite amused - "the first pancake is always spoiled". Recently I had my debut abroad on ACCU 2016 conference. You decide if it was spoiled pancake or not :)
My short 15min talk was about modern compilers. I start with basic introduction, which may be a bit surprising, then I move to C++ standard perspective and undefined behaviors, then there's a part about optimizations and finally I'm covering the ecosystem that has grown around compilers in recent years (sanitizers, 3rd tools based on compilers, etc.). Enjoy!
Slides: with comments / without comments
GCC 6.1 released [link]
If GCC was released two weeks earlier I would definitely include one slide about how verbose and helpful compilers are nowadays.
Several years ago they were unhelpful and very quiet - even if something was going terribly wrong (and the programmer would discover this in run-time) they were happily compiling assuming that the programmer knows what she's doing.
And now.. what a change. The compilers became so friendly that GCC 6.1 can even let you know that you have unmerged git content in the file!
* direct translation doesn't work :) - "first cats over the fences".
My short 15min talk was about modern compilers. I start with basic introduction, which may be a bit surprising, then I move to C++ standard perspective and undefined behaviors, then there's a part about optimizations and finally I'm covering the ecosystem that has grown around compilers in recent years (sanitizers, 3rd tools based on compilers, etc.). Enjoy!
Slides: with comments / without comments
GCC 6.1 released [link]
If GCC was released two weeks earlier I would definitely include one slide about how verbose and helpful compilers are nowadays.
Several years ago they were unhelpful and very quiet - even if something was going terribly wrong (and the programmer would discover this in run-time) they were happily compiling assuming that the programmer knows what she's doing.
And now.. what a change. The compilers became so friendly that GCC 6.1 can even let you know that you have unmerged git content in the file!
* direct translation doesn't work :) - "first cats over the fences".
czwartek, 17 marca 2016
Really simple JSON server
As some of you may already know there's this json-server project which aims to provide an easy way to create fake servers returning JSON responses over HTTP. This is very great project, but if you need something simpler that does not necessarily follow all of the REST rules, then you're out of luck.
For instance, if you use json-server with PUT/POST requests then underlying database will change.
really-simple-json-server, on the other hand, is an effort to create really simple JSON server. To actually show you how simple it is, let's have a look at example routes (example.json).
{
"/config": {"avatar_upstream_server": "172.17.42.42"},
"/u/1/friends": ["slavko", "asia", "agata", "zbyszek", "lucyna"],
"/u/2/friends": ["adam", "grzegorz"],
"/u/3/friends": [],
"/u/4/friends": ["slavko"]
}
Then, assuming all dependencies (see below) are installed it's all about starting a server:
$ ./server.py --port 1234 example.json
And we can start querying the server!
$ curl http://localhost:1234/config
{"avatar_upstream_server": "172.17.42.42"}
The project uses Python 3.5.1 along with aiohttp package. It is shipped with Docker image, so it's pretty easy to start hacking.
$ docker build -t szborows/python351_aiohttp .
$ docker run -it -v $PWD:/app:ro szborows/python351_aiohttp /bin/bash -c "/app/server.py --port 1234 /app/example.json"
For instance, if you use json-server with PUT/POST requests then underlying database will change.
really-simple-json-server, on the other hand, is an effort to create really simple JSON server. To actually show you how simple it is, let's have a look at example routes (example.json).
{
"/config": {"avatar_upstream_server": "172.17.42.42"},
"/u/1/friends": ["slavko", "asia", "agata", "zbyszek", "lucyna"],
"/u/2/friends": ["adam", "grzegorz"],
"/u/3/friends": [],
"/u/4/friends": ["slavko"]
}
Then, assuming all dependencies (see below) are installed it's all about starting a server:
$ ./server.py --port 1234 example.json
And we can start querying the server!
$ curl http://localhost:1234/config
{"avatar_upstream_server": "172.17.42.42"}
The project uses Python 3.5.1 along with aiohttp package. It is shipped with Docker image, so it's pretty easy to start hacking.
$ docker build -t szborows/python351_aiohttp .
$ docker run -it -v $PWD:/app:ro szborows/python351_aiohttp /bin/bash -c "/app/server.py --port 1234 /app/example.json"
środa, 16 marca 2016
ElasticSearch AWS cloud plugin problem connecting to Riak S3 endpoint without wildcard certs
curl -XPUT 'http://es.address.here:9200/_snapshot/s3_dev_backup?verify=false' -d '{
"type": "s3",
"settings": {
"access_key": "*****-***-**********",
"secret_key": "****************************************",
"bucket": "elastic",
"endpoint": "s3_cloud_front.address.here"
}
}'
As usual, in corporate reality, it didn't work. Instead, we got following exception.
Error injecting constructor, com.amazonaws.AmazonClientException: Unable to execute HTTP request: hostname in certificate didn't match: <elastic.s3_cloud_front.address.here> != <s3_cloud_front.address.here> OR <s3_cloud_front.address.here>
at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
while locating org.elasticsearch.repositories.s3.S3Repository
while locating org.elasticsearch.repositories.Repository
So apparently endpoint certificate (self signed one, btw) wasn't prepared in such a way that both e.g. endpoint.address and subdomain.endpoint.address would match. In other words it wasn't wildcard certificate bound to the endpoint domain name.
We decided to debug with StackOverflow (it's common technique nowadays, isn't it :D?). After trying numerous solutions like disabling certificate check using java flags or even using so-called java agents to hijack default hostname verifier we ran out of ideas how to elegantly solve the problem. And here comes our ultimate hack - hijacking Apache httpclient library.
The idea behind the hack is simple - re-compile httpclient library with one small modification - put premature return statement at the very top of hostname verification function. The function, where the change should be made is as follows.
org.apache.http.conn.ssl.SSLSocketFactory.verifyHostname(SSLSocketFactory.java:561)
One thing to remember is that the return statement must be wrapped with some silly if statement (e.g. if (1 == 1)), so the compiler won't complain about unreachable code below. It's funny that java compiler doesn't eliminate such trivial things, but in this particular scenario it was a feature rather than a bug. If the verifyHostname method wasn't throwing an exception then the modification would be even simpler - just an return statement.
The change is trivial, so I'm not including it here. The last step is to replace httpclient jar in AWS cloud plugin with raped one and we're done. No more complains about SSL hostnames.
Subskrybuj:
Posty (Atom)