Working of Preprocessors in rsyslog ?

classic Classic list List threaded Threaded
7 messages Options
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Working of Preprocessors in rsyslog ?

Luv
Is there any kind of documentation post where working and use of preprocessor(before main queue) is described ? I could not find any.

What is its uses ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

David Lang
On Tue, 20 Jun 2017, Luv via rsyslog wrote:

> Is there any kind of documentation post where working and use of
> preprocessor(before main queue) is described ? I could not find any.
>
> What is its uses ?

I'm not sure exactly what you are referring to.

Input modules (im) get logs from someplace (varies by the module) and may use
parser modules (pm) to extract meaningful logs from that input. The Input
Modules then put the message on the main queue where a worker thread picks up
the message and runs the ruleset against it. This is a series of filters and
actions. The filters decide if the actions are taken for this particular
message. An Action can be message modification modules (mm) that change the
message object (usually to create additional variables), or it can be variable
set commands, or it can be internal functions like stats gathering, or it can be
an invocation of functions from an output module (om). Actions frequently take
templates to form their output. These templates can be things that are
interpreted at runtime, or they can be the result of string modules (sm) that
use C code to create the string instead of interpreting the template definition.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

Luv
These preprocessors David, I am having trouble inserting an image here, but visit this page.

http://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html#turning-lanes-and-rsyslog-queues

The preprocessors before main queue and after input.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

David Lang
Rainer will have to comment on this. But I don't think it is referring to a
specific piece of code the way you think it does.

David Lang

On Wed, 21 Jun 2017, Luv via rsyslog wrote:

> Date: Wed, 21 Jun 2017 00:17:38 -0700 (MST)
> From: Luv via rsyslog <[hidden email]>
> To: [hidden email]
> Cc: Luv <[hidden email]>
> Subject: Re: [rsyslog] Working of Preprocessors in rsyslog ?
>
> These preprocessors David, I am having trouble inserting an image here, but
> visit this page.
>
> http://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html#turning-lanes-and-rsyslog-queues
>
> The preprocessors before main queue and after input.
>
>
>
> --
> View this message in context: http://rsyslog-users.1305293.n2.nabble.com/Working-of-Preprocessors-in-rsyslog-tp7592500p7592503.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

Luv
The parser modules, which detect whether to keep a message OR discard it, where are they implemented ?
in these preprocessors(before the main queue) OR in the parsing and filtering modules(after the main queue)  ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

David Lang
On Wed, 21 Jun 2017, Luv via rsyslog wrote:
>
> Date: Wed, 21 Jun 2017 02:56:02 -0700 (MST)
> From: Luv via rsyslog <[hidden email]>> The parser modules, which detect whether to keep a message OR discard it,
> where are they implemented ?
> in these preprocessors(before the main queue) OR in the parsing and
> filtering modules(after the main queue)  ?

parser modules do not decide to keep or discard a message. They are run against
the incoming message to populate the standard properties (hostname, timestamp,
facility/severity) and they either return a success (this parser was able to
decipher the message) or failure (this parser was not able to decipher the
message, run the next parser in the stack)

the rfc3164 parser always returns success, it's the fallback and has all sorts
of heuristics to try and extract _some_ meaning out of any garbage that is sent.

The message is then put on the main queue. There is no ability to throw messages
away before they are put on the main queue [1]

All filtering is done by the work thread reading data from the queue and
processing it.

[1] well, there is the message repeated functionality, but that only works for
some input types
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working of Preprocessors in rsyslog ?

Rainer Gerhards
In reply to this post by David Lang
Its an abstract description, but actual module structure parser modules
work at this logical layer.

Rainer

Sent from phone, thus brief.

Am 21.06.2017 12:49 nachm. schrieb "David Lang" <[hidden email]>:

> Rainer will have to comment on this. But I don't think it is referring to
> a specific piece of code the way you think it does.
>
> David Lang
>
> On Wed, 21 Jun 2017, Luv via rsyslog wrote:
>
> Date: Wed, 21 Jun 2017 00:17:38 -0700 (MST)
>> From: Luv via rsyslog <[hidden email]>
>> To: [hidden email]
>> Cc: Luv <[hidden email]>
>> Subject: Re: [rsyslog] Working of Preprocessors in rsyslog ?
>>
>> These preprocessors David, I am having trouble inserting an image here,
>> but
>> visit this page.
>>
>> http://www.rsyslog.com/doc/v8-stable/whitepapers/queues_anal
>> ogy.html#turning-lanes-and-rsyslog-queues
>>
>> The preprocessors before main queue and after input.
>>
>>
>>
>> --
>> View this message in context: http://rsyslog-users.1305293.n
>> 2.nabble.com/Working-of-Preprocessors-in-rsyslog-tp7592500p7592503.html
>> Sent from the rsyslog-users mailing list archive at Nabble.com.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...