rsyslog going down frequently

classic Classic list List threaded Threaded
6 messages Options
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

rsyslog going down frequently

Luv
I am sending data to elasticsearch via rsyslog.

I have been successful in that. But from yesterday, rsyslog is going down. There was no problem in elasticsearch server, but rsyslog went down a couple of times,

alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep "rsyslog"
[3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or sacrifice child
[3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB, anon-rss:1064096kB, file-rss:0kB
[3829358.495953] init: rsyslog main process (8340) killed by KILL signal
[3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or sacrifice child
[3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB, anon-rss:747232kB, file-rss:0kB
[3829822.518360] init: rsyslog main process (10452) killed by KILL signal
[3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or sacrifice child
[3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB, anon-rss:646832kB, file-rss:0kB
[3830138.748289] init: rsyslog main process (11990) killed by KILL signal
[3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or sacrifice child
[3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB, anon-rss:1062824kB, file-rss:0kB
[3830609.161384] init: rsyslog main process (13230) killed by KILL signal
[3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or sacrifice child
[3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB, anon-rss:497632kB, file-rss:0kB
[3830848.700492] init: rsyslog main process (15113) killed by KILL signal
[3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or sacrifice child
[3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB, anon-rss:1064088kB, file-rss:0kB
[3831401.784038] init: rsyslog main process (16116) killed by KILL signal


It is telling me that it is going out of memory. I used rsyslog because it is light and fast, but even this is getting a large OOM score which I did not expect.
What is the possible reason ? How can I avoid it ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslog going down frequently

rsyslog-users mailing list
impstats will show yo how many queues you have, and you can figure it
out how much memory they are taking.

Each queue you use, increases the memory footprint.


On 29/06/17 11:12, Luv via rsyslog wrote:

> I am sending data to elasticsearch via rsyslog.
>
> I have been successful in that. But from yesterday, rsyslog is going down.
> There was no problem in elasticsearch server, but rsyslog went down a couple
> of times,
>
> alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep
> "rsyslog"
> [3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or
> sacrifice child
> [3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB,
> anon-rss:1064096kB, file-rss:0kB
> [3829358.495953] init: rsyslog main process (8340) killed by KILL signal
> [3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or
> sacrifice child
> [3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB,
> anon-rss:747232kB, file-rss:0kB
> [3829822.518360] init: rsyslog main process (10452) killed by KILL signal
> [3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or
> sacrifice child
> [3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB,
> anon-rss:646832kB, file-rss:0kB
> [3830138.748289] init: rsyslog main process (11990) killed by KILL signal
> [3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or
> sacrifice child
> [3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB,
> anon-rss:1062824kB, file-rss:0kB
> [3830609.161384] init: rsyslog main process (13230) killed by KILL signal
> [3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or
> sacrifice child
> [3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB,
> anon-rss:497632kB, file-rss:0kB
> [3830848.700492] init: rsyslog main process (15113) killed by KILL signal
> [3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or
> sacrifice child
> [3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB,
> anon-rss:1064088kB, file-rss:0kB
> [3831401.784038] init: rsyslog main process (16116) killed by KILL signal
>
>
> It is telling me that it is going out of memory. I used rsyslog because it
> is light and fast, but even this is getting a large OOM score which I did
> not expect.
> What is the possible reason ? How can I avoid it ?
>
>
>
> --
> View this message in context: http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslog going down frequently

Rainer Gerhards
2017-06-29 11:25 GMT+02:00 mostolog--- via rsyslog <[hidden email]>:
> impstats will show yo how many queues you have, and you can figure it out
> how much memory they are taking.
>
> Each queue you use, increases the memory footprint.

+1

From past experience I would bet that Elastic is too slow, causing
queue buildup. Large queues are permitted by conf and so you end up
with what you say. As mostolog says very correctly, impstats will show
you this.

Rainer

>
>
>
> On 29/06/17 11:12, Luv via rsyslog wrote:
>>
>> I am sending data to elasticsearch via rsyslog.
>>
>> I have been successful in that. But from yesterday, rsyslog is going down.
>> There was no problem in elasticsearch server, but rsyslog went down a
>> couple
>> of times,
>>
>> alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep
>> "rsyslog"
>> [3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or
>> sacrifice child
>> [3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB,
>> anon-rss:1064096kB, file-rss:0kB
>> [3829358.495953] init: rsyslog main process (8340) killed by KILL signal
>> [3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or
>> sacrifice child
>> [3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB,
>> anon-rss:747232kB, file-rss:0kB
>> [3829822.518360] init: rsyslog main process (10452) killed by KILL signal
>> [3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or
>> sacrifice child
>> [3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB,
>> anon-rss:646832kB, file-rss:0kB
>> [3830138.748289] init: rsyslog main process (11990) killed by KILL signal
>> [3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or
>> sacrifice child
>> [3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB,
>> anon-rss:1062824kB, file-rss:0kB
>> [3830609.161384] init: rsyslog main process (13230) killed by KILL signal
>> [3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or
>> sacrifice child
>> [3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB,
>> anon-rss:497632kB, file-rss:0kB
>> [3830848.700492] init: rsyslog main process (15113) killed by KILL signal
>> [3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or
>> sacrifice child
>> [3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB,
>> anon-rss:1064088kB, file-rss:0kB
>> [3831401.784038] init: rsyslog main process (16116) killed by KILL signal
>>
>>
>> It is telling me that it is going out of memory. I used rsyslog because it
>> is light and fast, but even this is getting a large OOM score which I did
>> not expect.
>> What is the possible reason ? How can I avoid it ?
>>
>>
>>
>> --
>> View this message in context:
>> http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html
>> Sent from the rsyslog-users mailing list archive at Nabble.com.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslog going down frequently

Luv
This post was updated on .
Yeah thats fine.

But I did not make any use of queues. This is one of the actions I was usng,

action(type="omelasticsearch"
  template="all-json-nginx"  # use the template defined earlier
  searchIndex="django-logs-alpha"
  searchType="django"
  server="xx.xx.xx.xx"
  serverport="9200"
  uid="elastic"
  pwd="xxxxx"
  bulkmode="on"  # use the bulk API
  action.resumeretrycount="-1"  # retry indefinitely if Elasticsearch is unreachable
)


So, you see no queues implemented.

I did not implement queues as I was just testing this config.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslog going down frequently

Rainer Gerhards
2017-06-29 12:31 GMT+02:00 Luv via rsyslog <[hidden email]>:

> Yeah thats fine.
>
> But I did not make any use of queues. This is one of the actions I was usng,
>
> action(type="omelasticsearch"
>   template="all-json-nginx"  # use the template defined earlier
>   searchIndex="django-logs-alpha"
>   searchType="django"
>   server="xx.xx.xx.xx"
>   serverport="9200"
>   uid="elastic"
>   pwd="xxxxx"
>   bulkmode="on"  # use the bulk API
>   action.resumeretrycount="-1"  # retry indefinitely if Elasticsearch is
> unreachable
> )
>
>
> So, you see no queues implemented.
>


there still is the main queue...

We also see other things in pstats, so if you want to get this solved,
I think providing the stats is the way to go.

Rainer

>
>
> --
> View this message in context: http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527p7592530.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslog going down frequently

David Lang
> there still is the main queue...
>
> We also see other things in pstats, so if you want to get this solved,
> I think providing the stats is the way to go.

we will also need to see the full config, not just snippets of it.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...