action queues vs message modifier modules vs dequeuebatchsize

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

action queues vs message modifier modules vs dequeuebatchsize

Radu Gheorghe-2
Hello,

I have three questions about action queues:
1) if I have a message modifier action (e.g. mmnormalize), should I add an
action queue for better performance?
2) what's the flow of messages if I have both mmnormalize and, say,
omelasticsearch? Messages flow from the main queue to mmnormalize's queue
(if any) and then back to the main queue to be processed by omelasticsearch?
3) how does queue.dequeuebatchsize work if I have an action queue. For
example:

* I have a main queue with dequeuebatchsize=100
* I have an omelasticsearch action with no queue (direct) and
dequeuebatchsize=1000 and BulkMode="on"

How many messages are sent to ES in a single bulk? 100? 1000? 1?

I can do a test to answer this last question, but I'd like to understand
how multiple actions and direct queues work. For example, I didn't notice
any difference in performance (during a benchmark) with and without an
action queue with mmnormalize. But maybe the bottleneck was somewhere else.

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

Rainer Gerhards
2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
> Hello,
>
> I have three questions about action queues:
> 1) if I have a message modifier action (e.g. mmnormalize), should I add an
> action queue for better performance?

no, simply because that wouldn't work as expected. Remember that a
queue runs async, so the modification would not get back to the
original message object.

> 2) what's the flow of messages if I have both mmnormalize and, say,
> omelasticsearch? Messages flow from the main queue to mmnormalize's queue
> (if any) and then back to the main queue to be processed by omelasticsearch?

I guess I already answered this ;) With a queue, message objects are
duplicated and never be injected back. that's the prime reason you
want a queue.

> 3) how does queue.dequeuebatchsize work if I have an action queue. For
> example:
>
> * I have a main queue with dequeuebatchsize=100
> * I have an omelasticsearch action with no queue (direct) and
> dequeuebatchsize=1000 and BulkMode="on"

dequeuebatchsize has no effect in direct mode.

>
> How many messages are sent to ES in a single bulk? 100? 1000? 1?

at most 100 (fewer if the batch doesn't get full).

> I can do a test to answer this last question, but I'd like to understand
> how multiple actions and direct queues work. For example, I didn't notice
> any difference in performance (during a benchmark) with and without an
> action queue with mmnormalize. But maybe the bottleneck was somewhere else.

queues always *cost* performance. If you need more concurrency, you
should adjust the main or ruleset queue accordingly.

HTH
Rainer

>
> Best regards,
> Radu
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

Radu Gheorghe-2
Thanks Rainer! So let me see if I get this straight. Say I want to
mmnormalize some logs and then omelasticsearch them. I would (please
correct me where I'm wrong):

- define a beefy main queue for buffering and everything
- add a mmnormalize action that will change events from the main queue. Can
I use queue.workerthreads to parallelize this? What about processing in
batches - does queue.dequeuebatchsize on the main queue govern that or it's
impossible?
- add a omelasticsearch action with no action queue (unless I want to copy
events from the main queue and process them async - say write them to ES
but also to a file). The ES bulk size would be queue.dequeuebatchsize of
the main queue and I can use multiple threads via queue.workerthreads

Am I getting it right?

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards <[hidden email]>
wrote:

> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
> > Hello,
> >
> > I have three questions about action queues:
> > 1) if I have a message modifier action (e.g. mmnormalize), should I add
> an
> > action queue for better performance?
>
> no, simply because that wouldn't work as expected. Remember that a
> queue runs async, so the modification would not get back to the
> original message object.
>
> > 2) what's the flow of messages if I have both mmnormalize and, say,
> > omelasticsearch? Messages flow from the main queue to mmnormalize's queue
> > (if any) and then back to the main queue to be processed by
> omelasticsearch?
>
> I guess I already answered this ;) With a queue, message objects are
> duplicated and never be injected back. that's the prime reason you
> want a queue.
>
> > 3) how does queue.dequeuebatchsize work if I have an action queue. For
> > example:
> >
> > * I have a main queue with dequeuebatchsize=100
> > * I have an omelasticsearch action with no queue (direct) and
> > dequeuebatchsize=1000 and BulkMode="on"
>
> dequeuebatchsize has no effect in direct mode.
>
> >
> > How many messages are sent to ES in a single bulk? 100? 1000? 1?
>
> at most 100 (fewer if the batch doesn't get full).
>
> > I can do a test to answer this last question, but I'd like to understand
> > how multiple actions and direct queues work. For example, I didn't notice
> > any difference in performance (during a benchmark) with and without an
> > action queue with mmnormalize. But maybe the bottleneck was somewhere
> else.
>
> queues always *cost* performance. If you need more concurrency, you
> should adjust the main or ruleset queue accordingly.
>
> HTH
> Rainer
> >
> > Best regards,
> > Radu
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

Rainer Gerhards
2015-04-08 12:43 GMT+02:00 Radu Gheorghe <[hidden email]>:

> Thanks Rainer! So let me see if I get this straight. Say I want to
> mmnormalize some logs and then omelasticsearch them. I would (please
> correct me where I'm wrong):
>
> - define a beefy main queue for buffering and everything
> - add a mmnormalize action that will change events from the main queue. Can
> I use queue.workerthreads to parallelize this? What about processing in
> batches - does queue.dequeuebatchsize on the main queue govern that or it's
> impossible?
> - add a omelasticsearch action with no action queue (unless I want to copy
> events from the main queue and process them async - say write them to ES
> but also to a file). The ES bulk size would be queue.dequeuebatchsize of
> the main queue and I can use multiple threads via queue.workerthreads
>
> Am I getting it right?

sounds good, assuming this is your whole config (sounds so). Regarding
the queue settings: they affect every action that's not runing on a
separate queue. So if you define workers and batch sizes, they also
apply to mmnormalize. Note that you define an *upper limit* on worker
threads. Rsyslog will only spawn them if required (what's required is
defined via other queue settings).

Rainer

>
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards <[hidden email]>
> wrote:
>
>> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
>> > Hello,
>> >
>> > I have three questions about action queues:
>> > 1) if I have a message modifier action (e.g. mmnormalize), should I add
>> an
>> > action queue for better performance?
>>
>> no, simply because that wouldn't work as expected. Remember that a
>> queue runs async, so the modification would not get back to the
>> original message object.
>>
>> > 2) what's the flow of messages if I have both mmnormalize and, say,
>> > omelasticsearch? Messages flow from the main queue to mmnormalize's queue
>> > (if any) and then back to the main queue to be processed by
>> omelasticsearch?
>>
>> I guess I already answered this ;) With a queue, message objects are
>> duplicated and never be injected back. that's the prime reason you
>> want a queue.
>>
>> > 3) how does queue.dequeuebatchsize work if I have an action queue. For
>> > example:
>> >
>> > * I have a main queue with dequeuebatchsize=100
>> > * I have an omelasticsearch action with no queue (direct) and
>> > dequeuebatchsize=1000 and BulkMode="on"
>>
>> dequeuebatchsize has no effect in direct mode.
>>
>> >
>> > How many messages are sent to ES in a single bulk? 100? 1000? 1?
>>
>> at most 100 (fewer if the batch doesn't get full).
>>
>> > I can do a test to answer this last question, but I'd like to understand
>> > how multiple actions and direct queues work. For example, I didn't notice
>> > any difference in performance (during a benchmark) with and without an
>> > action queue with mmnormalize. But maybe the bottleneck was somewhere
>> else.
>>
>> queues always *cost* performance. If you need more concurrency, you
>> should adjust the main or ruleset queue accordingly.
>>
>> HTH
>> Rainer
>> >
>> > Best regards,
>> > Radu
>> > --
>> > Performance Monitoring * Log Analytics * Search Analytics
>> > Solr & Elasticsearch Support * http://sematext.com/
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

Radu Gheorghe-2
Thanks for your answers, Rainer. That clarifies a lot!

Regarding upper limits, I'm aware that both workers and batch sizes defined
are maximums.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Apr 8, 2015 at 1:51 PM, Rainer Gerhards <[hidden email]>
wrote:

> 2015-04-08 12:43 GMT+02:00 Radu Gheorghe <[hidden email]>:
> > Thanks Rainer! So let me see if I get this straight. Say I want to
> > mmnormalize some logs and then omelasticsearch them. I would (please
> > correct me where I'm wrong):
> >
> > - define a beefy main queue for buffering and everything
> > - add a mmnormalize action that will change events from the main queue.
> Can
> > I use queue.workerthreads to parallelize this? What about processing in
> > batches - does queue.dequeuebatchsize on the main queue govern that or
> it's
> > impossible?
> > - add a omelasticsearch action with no action queue (unless I want to
> copy
> > events from the main queue and process them async - say write them to ES
> > but also to a file). The ES bulk size would be queue.dequeuebatchsize of
> > the main queue and I can use multiple threads via queue.workerthreads
> >
> > Am I getting it right?
>
> sounds good, assuming this is your whole config (sounds so). Regarding
> the queue settings: they affect every action that's not runing on a
> separate queue. So if you define workers and batch sizes, they also
> apply to mmnormalize. Note that you define an *upper limit* on worker
> threads. Rsyslog will only spawn them if required (what's required is
> defined via other queue settings).
>
> Rainer
> >
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> > On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards <
> [hidden email]>
> > wrote:
> >
> >> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
> >> > Hello,
> >> >
> >> > I have three questions about action queues:
> >> > 1) if I have a message modifier action (e.g. mmnormalize), should I
> add
> >> an
> >> > action queue for better performance?
> >>
> >> no, simply because that wouldn't work as expected. Remember that a
> >> queue runs async, so the modification would not get back to the
> >> original message object.
> >>
> >> > 2) what's the flow of messages if I have both mmnormalize and, say,
> >> > omelasticsearch? Messages flow from the main queue to mmnormalize's
> queue
> >> > (if any) and then back to the main queue to be processed by
> >> omelasticsearch?
> >>
> >> I guess I already answered this ;) With a queue, message objects are
> >> duplicated and never be injected back. that's the prime reason you
> >> want a queue.
> >>
> >> > 3) how does queue.dequeuebatchsize work if I have an action queue. For
> >> > example:
> >> >
> >> > * I have a main queue with dequeuebatchsize=100
> >> > * I have an omelasticsearch action with no queue (direct) and
> >> > dequeuebatchsize=1000 and BulkMode="on"
> >>
> >> dequeuebatchsize has no effect in direct mode.
> >>
> >> >
> >> > How many messages are sent to ES in a single bulk? 100? 1000? 1?
> >>
> >> at most 100 (fewer if the batch doesn't get full).
> >>
> >> > I can do a test to answer this last question, but I'd like to
> understand
> >> > how multiple actions and direct queues work. For example, I didn't
> notice
> >> > any difference in performance (during a benchmark) with and without an
> >> > action queue with mmnormalize. But maybe the bottleneck was somewhere
> >> else.
> >>
> >> queues always *cost* performance. If you need more concurrency, you
> >> should adjust the main or ruleset queue accordingly.
> >>
> >> HTH
> >> Rainer
> >> >
> >> > Best regards,
> >> > Radu
> >> > --
> >> > Performance Monitoring * Log Analytics * Search Analytics
> >> > Solr & Elasticsearch Support * http://sematext.com/
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

chenlin rao
In reply to this post by Rainer Gerhards
rainer:
     What's main queue dequeuebatchsize meaning? I should define the
options in `action()`, `ruleset()` or `$MainMsgQueuedequeuebatchsize`?

     I'm not so clear about ruleset queue and action queue. Especially one
time, I defined `queue.workerthreads="5"` both in my two different
rulesets, then found reciever got two copy...

    And I didn't find too much example about those things, hope a better
document... Thanks very much!!

2015-04-08 17:28 GMT+08:00 Rainer Gerhards <[hidden email]>:

> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
> > Hello,
> >
> > I have three questions about action queues:
> > 1) if I have a message modifier action (e.g. mmnormalize), should I add
> an
> > action queue for better performance?
>
> no, simply because that wouldn't work as expected. Remember that a
> queue runs async, so the modification would not get back to the
> original message object.
>
> > 2) what's the flow of messages if I have both mmnormalize and, say,
> > omelasticsearch? Messages flow from the main queue to mmnormalize's queue
> > (if any) and then back to the main queue to be processed by
> omelasticsearch?
>
> I guess I already answered this ;) With a queue, message objects are
> duplicated and never be injected back. that's the prime reason you
> want a queue.
>
> > 3) how does queue.dequeuebatchsize work if I have an action queue. For
> > example:
> >
> > * I have a main queue with dequeuebatchsize=100
> > * I have an omelasticsearch action with no queue (direct) and
> > dequeuebatchsize=1000 and BulkMode="on"
>
> dequeuebatchsize has no effect in direct mode.
>
> >
> > How many messages are sent to ES in a single bulk? 100? 1000? 1?
>
> at most 100 (fewer if the batch doesn't get full).
>
> > I can do a test to answer this last question, but I'd like to understand
> > how multiple actions and direct queues work. For example, I didn't notice
> > any difference in performance (during a benchmark) with and without an
> > action queue with mmnormalize. But maybe the bottleneck was somewhere
> else.
>
> queues always *cost* performance. If you need more concurrency, you
> should adjust the main or ruleset queue accordingly.
>
> HTH
> Rainer
> >
> > Best regards,
> > Radu
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

David Lang
In reply to this post by Radu Gheorghe-2
On Wed, 8 Apr 2015, Radu Gheorghe wrote:

> Thanks Rainer! So let me see if I get this straight. Say I want to
> mmnormalize some logs and then omelasticsearch them. I would (please
> correct me where I'm wrong):
>
> - define a beefy main queue for buffering and everything
> - add a mmnormalize action that will change events from the main queue. Can
> I use queue.workerthreads to parallelize this? What about processing in
> batches - does queue.dequeuebatchsize on the main queue govern that or it's
> impossible?
> - add a omelasticsearch action with no action queue (unless I want to copy
> events from the main queue and process them async - say write them to ES
> but also to a file). The ES bulk size would be queue.dequeuebatchsize of
> the main queue and I can use multiple threads via queue.workerthreads
>
> Am I getting it right?

I'm pretty sure that you understand this, but for others who see this thread:

The reason to have a separate queue for an action or ruleset is so that if that
blocks, it doesn't immediatly block all other processing of logs on the main
queue. The other things happening can be writing to local files, or delivering
to other remote destinations.

giving everything it's own queue and creating multiple worker threads will
actually significantly HURT performance, because one of the big bottlenecks at
high performance is the locking of the queues as messages get added and removed
from a queue. When everything is a queue and you allow many worker threads, you
can get a batch of messages arriving (enough to trigger additional worker thread
creation), but then the worker threads get in each other's way as they are all
trying to lock the main queue at the same time to move a batch of messages to
the other queues.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: action queues vs message modifier modules vs dequeuebatchsize

David Lang
In reply to this post by chenlin rao
On Wed, 8 Apr 2015, chenlin rao wrote:

> rainer:
>     What's main queue dequeuebatchsize meaning? I should define the
> options in `action()`, `ruleset()` or `$MainMsgQueuedequeuebatchsize`?

legacy format would be $MainMsgQueuedequeuebatchsize

new format would be as part of the main_queue() definition

http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html

David Lang

>     I'm not so clear about ruleset queue and action queue. Especially one
> time, I defined `queue.workerthreads="5"` both in my two different
> rulesets, then found reciever got two copy...
>
>    And I didn't find too much example about those things, hope a better
> document... Thanks very much!!
>
> 2015-04-08 17:28 GMT+08:00 Rainer Gerhards <[hidden email]>:
>
>> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[hidden email]>:
>>> Hello,
>>>
>>> I have three questions about action queues:
>>> 1) if I have a message modifier action (e.g. mmnormalize), should I add
>> an
>>> action queue for better performance?
>>
>> no, simply because that wouldn't work as expected. Remember that a
>> queue runs async, so the modification would not get back to the
>> original message object.
>>
>>> 2) what's the flow of messages if I have both mmnormalize and, say,
>>> omelasticsearch? Messages flow from the main queue to mmnormalize's queue
>>> (if any) and then back to the main queue to be processed by
>> omelasticsearch?
>>
>> I guess I already answered this ;) With a queue, message objects are
>> duplicated and never be injected back. that's the prime reason you
>> want a queue.
>>
>>> 3) how does queue.dequeuebatchsize work if I have an action queue. For
>>> example:
>>>
>>> * I have a main queue with dequeuebatchsize=100
>>> * I have an omelasticsearch action with no queue (direct) and
>>> dequeuebatchsize=1000 and BulkMode="on"
>>
>> dequeuebatchsize has no effect in direct mode.
>>
>>>
>>> How many messages are sent to ES in a single bulk? 100? 1000? 1?
>>
>> at most 100 (fewer if the batch doesn't get full).
>>
>>> I can do a test to answer this last question, but I'd like to understand
>>> how multiple actions and direct queues work. For example, I didn't notice
>>> any difference in performance (during a benchmark) with and without an
>>> action queue with mmnormalize. But maybe the bottleneck was somewhere
>> else.
>>
>> queues always *cost* performance. If you need more concurrency, you
>> should adjust the main or ruleset queue accordingly.
>>
>> HTH
>> Rainer
>>>
>>> Best regards,
>>> Radu
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.