rsyslogd Consuming CPU

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

rsyslogd Consuming CPU

Heaton, Tobias
I'm noticing rsyslog is consuming 200% of my CPU this morning. I restarted rsyslog and rebooted the system - CPU usage persists.

My rsyslog installation is writing to multiple MySQL databases that was working fine up until this morning. Now the MySQL writes are slowing to a crawl as the CPU is consumed.

I don't see any errors when restarting, no issues with the configuration and no recent updates to rsyslog (current version 8.2.7).

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslogd Consuming CPU

Rainer Gerhards
2017-06-28 15:12 GMT+02:00 Heaton, Tobias <[hidden email]>:
> I'm noticing rsyslog is consuming 200% of my CPU this morning. I restarted rsyslog and rebooted the system - CPU usage persists.
>
> My rsyslog installation is writing to multiple MySQL databases that was working fine up until this morning. Now the MySQL writes are slowing to a crawl as the CPU is consumed.
>
> I don't see any errors when restarting, no issues with the configuration and no recent updates to rsyslog (current version 8.2.7).


Is it 8.2.7 or 8.27? If the former, it's heavily outdated.

If the latter, a debug log would probably make sense.

Rainer
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rsyslogd Consuming CPU

Heaton, Tobias
Hi Rainer,

It is version 8.27 - sorry for the typo. I leverage yum repos to keep rsyslog up-to-date.

Running in debug mode I think I found the source of the issue. I have a configuration that detects error 1146 (missing database table) and if one is detected, it captures the missing table name and fires a Perl script to create the table in the database.

For whatever reason that configuration stopped working to create the tables, causing a serious backlog of 1146 errors which caused the spike in CPU. I created the tables manually and CPU is back to normal now. I'll have to look a little deeper as to why that configuration file failed to create the necessary tables dynamically.

Thanks,
Tobias

-----Original Message-----
From: rsyslog [mailto:[hidden email]] On Behalf Of Rainer Gerhards
Sent: Wednesday, June 28, 2017 9:15 AM
To: rsyslog-users <[hidden email]>
Subject: Re: [rsyslog] rsyslogd Consuming CPU

2017-06-28 15:12 GMT+02:00 Heaton, Tobias <[hidden email]>:
> I'm noticing rsyslog is consuming 200% of my CPU this morning. I restarted rsyslog and rebooted the system - CPU usage persists.
>
> My rsyslog installation is writing to multiple MySQL databases that was working fine up until this morning. Now the MySQL writes are slowing to a crawl as the CPU is consumed.
>
> I don't see any errors when restarting, no issues with the configuration and no recent updates to rsyslog (current version 8.2.7).


Is it 8.2.7 or 8.27? If the former, it's heavily outdated.

If the latter, a debug log would probably make sense.

Rainer
>
> _______________________________________________
> rsyslog mailing list
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=kg5dn0DDn2GK70iIAp_XmroX0ihVFfTXcDtZvweGIVI&s=eT_Q8iirkdoXAN5JGy6VEgDkMWH5Wd1q7lOqaBa76v0&e= 
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=kg5dn0DDn2GK70iIAp_XmroX0ihVFfTXcDtZvweGIVI&s=YVh9b4NXpcWnU8d8QFa-TCQ4A7ndtYa0tUvcGRFh6LA&e= 
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=kg5dn0DDn2GK70iIAp_XmroX0ihVFfTXcDtZvweGIVI&s=eT_Q8iirkdoXAN5JGy6VEgDkMWH5Wd1q7lOqaBa76v0&e= 
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwICAg&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=kg5dn0DDn2GK70iIAp_XmroX0ihVFfTXcDtZvweGIVI&s=YVh9b4NXpcWnU8d8QFa-TCQ4A7ndtYa0tUvcGRFh6LA&e= 
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...