About Message Modification Modules

classic Classic list List threaded Threaded
6 messages Options
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

About Message Modification Modules

Luv
This post was updated on .
I have a question about the implementation of the message modification modules.

If I have the following config,

if $programname == 'nginx' then {

$ActionQueueType LinkedList
$ActionQueueDequeueBatchSize 100
$ActionQueueSize 10000
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueFileName elastic_queue_filelog


action(type="mmnormalize"
  rulebase="/opt/rsyslog/apache.rb"
)

if $parsesuccess == "OK" then {
action(type="omelasticsearch"
  template="all-json-nginx"  # use the template defined earlier
  searchIndex="nginx-logs"
  searchType="nginx"
  server="127.0.0.1"
  serverport="9200"
  bulkmode="on"  # use the bulk API
  action.resumeretrycount="-1"  # retry indefinitely if Logsene/Elasticsearch is unreachable
)
} else {
action(type="omelasticsearch"
  template="all-json-nginx"  # use the template defined earlier
  searchIndex="nginx-logs-2"
  searchType="nginx"
  server="127.0.0.1"
  serverport="9200"
  bulkmode="on"  # use the bulk API
  action.resumeretrycount="-1"  
}


The Action queue which I have created over there before the mmnormalize type action, for which action it will work for ?

The mmnormalize is not an output module. It is a parsing module, so for what action this action queue will be available ,

if $parsesuccess == True

if $parsesuccess == False

?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: About Parsing Modules

David Lang
On Mon, 19 Jun 2017, Luv via rsyslog wrote:

> Subject: [rsyslog] About Parsing Modules
>
> I have a question about the implementation of the message modification
> modules.

parsing modules and message modification modules are very different things.
Please make the subject match what you are asking about.

> The Action queue which I have created over there before the mmnormalize type
> action, for which action it will work for ?

the next legacy style statement in your file (none are listed here)

> The mmnormalize is not an output module. It is a parsing module, so for what
> action this action queue will be available ,

no, it is a message modification module (that's what the mm in the name means),
a parsing module is something very different. A parser module takes the raw
message as it comes in on the wire and parses it (populating the standard
properties), a message modification module operates much later and modifies the
message object (usually creating $! variables)

you never want to put a queue on a mm module action, doing so makes a copy of
the message, has the mm module change the message, and then throws away the
modified message because there's nothing else to do with it.

> if $parsesuccess == True
>
> if $parsesuccess == False

I have not had any success using this variable. It may have gotten fixed in a
recent version (I remember some discussion about this)

David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: About Parsing Modules

Luv
Well, the $parsesuccess does work for me. I have tried it.

Is there any article OR blog post by your team on a working example of mmfields module ?

I see that apart from performance issues, the only difference between mmfields and mmnormalize is that mmfields work according to the separators and mmnormalize work according to the given rule.

How can we give name to fields in mmfields like we can give in mmnormalize ?
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: About Parsing Modules

David Lang
On Mon, 19 Jun 2017, Luv via rsyslog wrote:

> Well, the $parsesuccess does work for me. I have tried it.
>
> Is there any article OR blog post by your team on a working example of
> mmfields module ?

have you checked the documenstion?
k
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: About Parsing Modules

Luv
Well, I did.

But I could not get that why do we need mmfields when we have mmnormalize ? You see, mmnormalize have clean and nice liblognorm rules, it can parse any log message given the correct rules, and we can name the fields as we want but the separator in mmfields is not enough to parse all the messages, and we could not name the fields.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: About Parsing Modules

David Lang
On Mon, 19 Jun 2017, Luv via rsyslog wrote:

> Well, I did.
>
> But I could not get that why do we need mmfields when we have mmnormalize ?
> You see, mmnormalize have clean and nice liblognorm rules, it can parse any
> log message given the correct rules, and we can name the fields as we want
> but the separator in mmfields is not enough to parse all the messages, and
> we could not name the fields.

That's a very different question. The ultimate answer is 'because someone wrote
it and contributed it', it may have been that they thought mmnormalize was too
complicated, it may have been that mmfields did something that was hard to do in
mmnormalize at the time it was written.

Open Source Software is written because someone thinks their idea is better than
what exists. Sometimes they are correct, sometimes they are not.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...