Why are action queues needed ?

classic Classic list List threaded Threaded
8 messages Options
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Why are action queues needed ?

Luv
I want to know that why action queues are needed when a Main queue is already there ? What extra benefits it provide, why don't we increase size of main queue instead of making action queues ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

David Lang
On Thu, 15 Jun 2017, Luv via rsyslog wrote:

> I want to know that why action queues are needed when a Main queue is already
> there ? What extra benefits it provide, why don't we increase size of main
> queue instead of making action queues ?

action queues let you decouple actions that are likely to block from others.

If you only use the main queue, then when any action blocks (say ElasticSearch
goes down), all message processing stops. If you put an action queue on the ES
action (or a ruleset containing the ES action), then when ES goes down, only log
delivery to ES stops (or the things in the ruleset stop), and other log
processing continues (for example, writing out to local files so that you can
see the log message saying that deliver to ES is failing)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

Luv
I have some more questions.

1 - Suppose I have 2 actions. One outputting to elasticsearch, and other outputting to postgresql. I make action queue for only elasticsearch, not for postgres. Which queue, main OR action will be used if

i) elastic goes down
ii) postgres goes down

2 - http://blog.gerhards.net/2012/07/rsyslog-action-queue.html This blog here tells us how to create action queues. So, we have to write those lines for every action ? This will make the code repititve, Is there any short method for that ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

Rainer Gerhards
This gives a simple to grasp analogy of how queues work:

http://www.rsyslog.com/doc/queues_analogy.html

Rainer

Am 15.06.2017 15:21 schrieb "Luv via rsyslog" <[hidden email]>:

> I have some more questions.
>
> 1 - Suppose I have 2 actions. One outputting to elasticsearch, and other
> outputting to postgresql. I make action queue for only elasticsearch, not
> for postgres. Which queue, main OR action will be used if
>
> i) elastic goes down
> ii) postgres goes down
>
> 2 - http://blog.gerhards.net/2012/07/rsyslog-action-queue.html This blog
> here tells us how to create action queues. So, we have to write those lines
> for every action ? This will make the code repititve, Is there any short
> method for that ?
>
>
>
> --
> View this message in context: http://rsyslog-users.1305293.
> n2.nabble.com/Why-are-action-queues-needed-tp7592475p7592477.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

David Lang
In reply to this post by Luv
On Thu, 15 Jun 2017, Luv via rsyslog wrote:

> I have some more questions.
>
> 1 - Suppose I have 2 actions. One outputting to elasticsearch, and other
> outputting to postgresql. I make action queue for only elasticsearch, not
> for postgres. Which queue, main OR action will be used if
>
> i) elastic goes down

Since it has a queue, that queue will be used until it fills up, then the main
queue will be used.

> ii) postgres goes down

since there is no action queue for this, the main queue will be used.

note that you could create a ruleset with a queue and then put both of these in
the ruleset, that way you only have one copy of the log message (not one per
action queue) when you have a network issue.

> 2 - http://blog.gerhards.net/2012/07/rsyslog-action-queue.html This blog
> here tells us how to create action queues. So, we have to write those lines
> for every action ? This will make the code repititve, Is there any short
> method for that ?

use the newer action() syntax, it makes it much clearer what is happening. Yes,
you do need to repeat the queue settings for every action that you want to have
an independent queue, but that's by design.

you should consider making a rulest and using a queue on it instead of a lot of
separate action queues, separate action queues will each make a copy of the log
message, and that can eat up memory fast.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

Luv
This post was updated on .
Okay, cool !

That road lane analogy makes things much much clearer.

So, it means if direct queue action goes down, then all other actions will be stopped and logs will be stored in the main queue, did I get it right ?

And discard function works only in direct queues.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

rsyslog-users mailing list
In reply to this post by Luv
> 2 - http://blog.gerhards.net/2012/07/rsyslog-action-queue.html This blog
> here tells us how to create action queues. So, we have to write those lines
> for every action ? This will make the code repititve, Is there any short
> method for that ?

Two things I would add:

- Ansible is your best friend when managing a complex rsyslog configurations.  Ansible can turn a few 20-30 line templates, and inventory of your inputs and outputs in to a sprawling rsyslog config with all the rulesets and queue configurations necessary

- If you have multiple configuration files and find yourself repeating configurations between them, look in to using $IncludeConfig to manage those common elements

Andrew Griffin
Apple
ETS / Integration Services
1 Infinite Loop, 175-DR
Cupertino, CA 95014, USA
Office 408-783-8348
iPhone 916-897-4335
[hidden email]

This email and any attachments may be privileged and may contain confidential information intended only for the recipient(s) named above. Any other distribution, forwarding, copying or disclosure of this message is strictly prohibited. If you have received this email in error, please notify me immediately by telephone or return email, and delete this message from your system.



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why are action queues needed ?

David Lang
In reply to this post by Luv
On Thu, 15 Jun 2017, Luv via rsyslog wrote:

> Okay, cool !
>
> That road lane analogy makes things much much clearer.
>
> So, it means if direct queue action goes down, then all other actions will
> be stopped and logs will be stored in the main queue, did I get it right ?

yes.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...