Split messages options

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Split messages options

Scot Kreienkamp
Hi everyone,

I have a java program that is using log4j to write into rsyslog.  The messages are so large they are getting split, which causes the sorting rule (if $syslogtag == 'RMS-Tomcat:' then) to write the first part to the correct file, but the second part goes into messages because it has no header info.  I'm at a loss as to the best way to deal with this.  The log4j implementation is fairly old and doesn't seem to have any controls that we can use to help with this problem, and getting them to update it is a year-long chore at best.  I already have $MaxMessageSize 64k set before the module load lines in my config, so I'm fairly certain that the entries I'm receiving are larger than 64k.  Not certain if I could go larger, old mail messages seemed to indicate 64k was the ceiling and higher values would be ignored.  Is raising that value the best way to deal with this problem, and do I need the old directive with the global directive present?


Thanks for any help.



Relevent snippets of config:

$umask 0000
$FileCreateMode 0644
$DirCreateMode 0755
$RepeatedMsgReduction off
$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
$RepeatedMsgReduction off
$WorkDirectory /var/spool/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$MaxOpenFiles 4096
module(load="imfile") #needs to be done just once
module(load="imudp") # needs to be done just once for listener only
module(load="imptcp") # needs to be done just once for listener only

global(workDirectory="/var/spool/rsyslog" preserveFQDN="on" maxMessageSize="64k")

main_queue(
        queue.size="50000"           # or this many messages
        queue.discardmark="40000"
        queue.DiscardSeverity="0"
        queue.dequeueBatchSize="1024"
        queue.spoolDirectory="/var/spool/rsyslog"  # where to write on disk
        queue.fileName="rsyslogmainqueue"
        queue.maxDiskSpace="1g"        # it will stop at this much disk space
        queue.saveOnShutdown="on"      # save memory queue contents to disk when rsyslog is exiting
        queue.type="LinkedList"
)


#Dynafile template for Ecomm logs
template (name="DailyPerHostLogEcomm" type="string" string="/var/log/tomcat6/Ecomm-%$YEAR%-%$MONTH%-%$DAY%")

#activate listener on localhost address
input(type="imudp" port="514" address="127.0.0.1")
input(type="imptcp" port="514" address="127.0.0.1")

if $syslogtag == 'RMS-Ecomm:' then {
        action(
                name="omfile-Ecomm.log"
                type="omfile"
                dynafilecachesize="5"
                DynaFile="DailyPerHostLogEcomm"
                template="msgonly-no1sp"
                ioBufferSize="64k"
                flushOnTXEnd="off"
                asyncWriting="on"
                dirCreateMode="0755"
        )
        stop
}

Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162  | * 734-384-6403 | | * 7349151444 | *  [hidden email]<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> | twitter.com/lazboy<https://twitter.com/lazboy> | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>

[cid:lzbVertical_hres.jpg]



This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

lzbVertical_hres.jpg (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

David Lang
you don't show us your imfile config, check to see if the java program is
outputting multi-line messages (and if so, are you handling them correctly)

with current rsyslo versions, I have set the maxmessagesize larger than 64k

unfortunantly, rsyslog processes each message it sees independently, so there's
no good way to reference the prior 'message' when messages get split. You need
to work to avoid having them split before rsyslog processes them.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

Scot Kreienkamp
Hi David,

I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.



Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
-----Original Message-----
From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
Sent: Monday, July 10, 2017 3:59 PM
To: rsyslog-users <[hidden email]>
Subject: Re: [rsyslog] Split messages options

you don't show us your imfile config, check to see if the java program is
outputting multi-line messages (and if so, are you handling them correctly)

with current rsyslo versions, I have set the maxmessagesize larger than 64k

unfortunantly, rsyslog processes each message it sees independently, so there's
no good way to reference the prior 'message' when messages get split. You need
to work to avoid having them split before rsyslog processes them.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

David Lang
ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
size, so it splits things before they get to rsyslog, and your maxmessagesize
isn't going to help.

normally I am not in favor of writing to disk and then reading them, but if you
are stuck with an old log4j, that may be your best option.

older log4j implementations are UDP only with a 1000 byte max size

slightly newer implementations support TCP with larger message sizes (but I've
seen some that still limit you to 2k)

current implementations are far more flexible.

David Lang

> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

Scot Kreienkamp
That makes sense, I wasn't aware of the limitation in log4j.  And yes, it is a very old implementation and getting them to update would likely take years.  :-)

In my case I'm transporting the logfile from the prod servers to a common collector server internally that the devs can have access to; all I need to do is reconstitute the file exactly as is on the other side.  I'll change syslogappender to a non-default port and use a custom template to forward it on so I can force the correct tag on all forwarded messages, that way the splitting won't matter and the logfile will be written on the destination exactly as the source.  That will solve my problem for now.

Thanks for the bit about log4j.  I'll have to do some more research there.


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
-----Original Message-----
From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
Sent: Monday, July 10, 2017 4:38 PM
To: rsyslog-users <[hidden email]>
Subject: Re: [rsyslog] Split messages options

ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
size, so it splits things before they get to rsyslog, and your maxmessagesize
isn't going to help.

normally I am not in favor of writing to disk and then reading them, but if you
are stuck with an old log4j, that may be your best option.

older log4j implementations are UDP only with a 1000 byte max size

slightly newer implementations support TCP with larger message sizes (but I've
seen some that still limit you to 2k)

current implementations are far more flexible.

David Lang

> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

David Lang
note that there is some potential for messages to get out of order (over the
network with UDP, and inside rsyslog if you use disk queues or multiple threads)

UDP messages can be dropped if the network is busy as well (by any
router/firewall or receiving host)

but the vast majority of the time, everything will be in order.

David Lang

On Mon, 10 Jul 2017, Scot Kreienkamp wrote:

> Date: Mon, 10 Jul 2017 20:49:42 +0000
> From: Scot Kreienkamp <[hidden email]>
> Reply-To: rsyslog-users <[hidden email]>
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> That makes sense, I wasn't aware of the limitation in log4j.  And yes, it is a very old implementation and getting them to update would likely take years.  :-)
>
> In my case I'm transporting the logfile from the prod servers to a common collector server internally that the devs can have access to; all I need to do is reconstitute the file exactly as is on the other side.  I'll change syslogappender to a non-default port and use a custom template to forward it on so I can force the correct tag on all forwarded messages, that way the splitting won't matter and the logfile will be written on the destination exactly as the source.  That will solve my problem for now.
>
> Thanks for the bit about log4j.  I'll have to do some more research there.
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
> -----Original Message-----
> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
> Sent: Monday, July 10, 2017 4:38 PM
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
> size, so it splits things before they get to rsyslog, and your maxmessagesize
> isn't going to help.
>
> normally I am not in favor of writing to disk and then reading them, but if you
> are stuck with an old log4j, that may be your best option.
>
> older log4j implementations are UDP only with a 1000 byte max size
>
> slightly newer implementations support TCP with larger message sizes (but I've
> seen some that still limit you to 2k)
>
> current implementations are far more flexible.
>
> David Lang
>
>> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

Scot Kreienkamp
Yep, understood on the ordering.  Log4j is sending to rsyslog@localhost via UDP, rsyslog is relaying via TCP.  For my usage, vast majority of the time is just fine.

I would have assumed that rsyslog would attempt processing of messages from any queue in the order received though...  No?


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
-----Original Message-----
From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
Sent: Monday, July 10, 2017 4:56 PM
To: rsyslog-users <[hidden email]>
Subject: Re: [rsyslog] Split messages options

note that there is some potential for messages to get out of order (over the
network with UDP, and inside rsyslog if you use disk queues or multiple threads)

UDP messages can be dropped if the network is busy as well (by any
router/firewall or receiving host)

but the vast majority of the time, everything will be in order.

David Lang

On Mon, 10 Jul 2017, Scot Kreienkamp wrote:

> Date: Mon, 10 Jul 2017 20:49:42 +0000
> From: Scot Kreienkamp <[hidden email]>
> Reply-To: rsyslog-users <[hidden email]>
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> That makes sense, I wasn't aware of the limitation in log4j.  And yes, it is a very old implementation and getting them to update would likely take years.  :-)
>
> In my case I'm transporting the logfile from the prod servers to a common collector server internally that the devs can have access to; all I need to do is reconstitute the file exactly as is on the other side.  I'll change syslogappender to a non-default port and use a custom template to forward it on so I can force the correct tag on all forwarded messages, that way the splitting won't matter and the logfile will be written on the destination exactly as the source.  That will solve my problem for now.
>
> Thanks for the bit about log4j.  I'll have to do some more research there.
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
> -----Original Message-----
> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
> Sent: Monday, July 10, 2017 4:38 PM
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
> size, so it splits things before they get to rsyslog, and your maxmessagesize
> isn't going to help.
>
> normally I am not in favor of writing to disk and then reading them, but if you
> are stuck with an old log4j, that may be your best option.
>
> older log4j implementations are UDP only with a 1000 byte max size
>
> slightly newer implementations support TCP with larger message sizes (but I've
> seen some that still limit you to 2k)
>
> current implementations are far more flexible.
>
> David Lang
>
>> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

David Lang
If you have multiple threads working to process messages, thread 1 will grab
messages 1-10 and start processing them, thread 2 will grab messages 11-20 and
start processing them in parallel, so ordering will be lost.

avoid using multiple threads when processing them, and you avoid that problem.

If logs get written to a disk queue, when new messages arrive they are processed
first, and messages from the queue get read and processed interspersed with the
new messages. The only way to avoid this problem is to not use a disk queue.

David Lang

On Mon, 10 Jul 2017, Scot Kreienkamp wrote:

> Yep, understood on the ordering.  Log4j is sending to rsyslog@localhost via UDP, rsyslog is relaying via TCP.  For my usage, vast majority of the time is just fine.
>
> I would have assumed that rsyslog would attempt processing of messages from any queue in the order received though...  No?
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
> -----Original Message-----
> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
> Sent: Monday, July 10, 2017 4:56 PM
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> note that there is some potential for messages to get out of order (over the
> network with UDP, and inside rsyslog if you use disk queues or multiple threads)
>
> UDP messages can be dropped if the network is busy as well (by any
> router/firewall or receiving host)
>
> but the vast majority of the time, everything will be in order.
>
> David Lang
>
> On Mon, 10 Jul 2017, Scot Kreienkamp wrote:
>
>> Date: Mon, 10 Jul 2017 20:49:42 +0000
>> From: Scot Kreienkamp <[hidden email]>
>> Reply-To: rsyslog-users <[hidden email]>
>> To: rsyslog-users <[hidden email]>
>> Subject: Re: [rsyslog] Split messages options
>>
>> That makes sense, I wasn't aware of the limitation in log4j.  And yes, it is a very old implementation and getting them to update would likely take years.  :-)
>>
>> In my case I'm transporting the logfile from the prod servers to a common collector server internally that the devs can have access to; all I need to do is reconstitute the file exactly as is on the other side.  I'll change syslogappender to a non-default port and use a custom template to forward it on so I can force the correct tag on all forwarded messages, that way the splitting won't matter and the logfile will be written on the destination exactly as the source.  That will solve my problem for now.
>>
>> Thanks for the bit about log4j.  I'll have to do some more research there.
>>
>>
>> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
>> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
>> -----Original Message-----
>> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
>> Sent: Monday, July 10, 2017 4:38 PM
>> To: rsyslog-users <[hidden email]>
>> Subject: Re: [rsyslog] Split messages options
>>
>> ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
>> size, so it splits things before they get to rsyslog, and your maxmessagesize
>> isn't going to help.
>>
>> normally I am not in favor of writing to disk and then reading them, but if you
>> are stuck with an old log4j, that may be your best option.
>>
>> older log4j implementations are UDP only with a 1000 byte max size
>>
>> slightly newer implementations support TCP with larger message sizes (but I've
>> seen some that still limit you to 2k)
>>
>> current implementations are far more flexible.
>>
>> David Lang
>>
>>> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>> This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Split messages options

Scot Kreienkamp
From my config text, I'm not writing a disk queue so that won't be a problem, and my network queues I'm assuming are single threaded by default so that shouldn't be a problem either.

ruleset(name="RMS-Ecomm-1514-1531"){
        action(
                name="omfile-Ecomm.log"
                type="omfile"
                dynafilecachesize="5"
                DynaFile="DailyPerHostLogEcomm"
                template="msgonly-no1sp"
                ioBufferSize="64k"
                flushOnTXEnd="off"
                asyncWriting="on"
                dirCreateMode="0755"
        )
        action(
                name="fwd-monvsyslog-1531"
                type="omfwd"
                Target="monvsyslog.na.lzb.hq"
                Port="1531"
                Protocol="tcp"
                template="RMS-ForwardEcomm"
        )
        stop
}

input(type="imudp" port="1514" address="127.0.0.1" ruleset="RMS-Ecomm-1514-1531")
input(type="imptcp" port="1514" address="127.0.0.1" ruleset="RMS-Ecomm-1514-1531")





Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
-----Original Message-----
From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
Sent: Monday, July 10, 2017 5:12 PM
To: rsyslog-users <[hidden email]>
Subject: Re: [rsyslog] Split messages options

If you have multiple threads working to process messages, thread 1 will grab
messages 1-10 and start processing them, thread 2 will grab messages 11-20 and
start processing them in parallel, so ordering will be lost.

avoid using multiple threads when processing them, and you avoid that problem.

If logs get written to a disk queue, when new messages arrive they are processed
first, and messages from the queue get read and processed interspersed with the
new messages. The only way to avoid this problem is to not use a disk queue.

David Lang

On Mon, 10 Jul 2017, Scot Kreienkamp wrote:

> Yep, understood on the ordering.  Log4j is sending to rsyslog@localhost via UDP, rsyslog is relaying via TCP.  For my usage, vast majority of the time is just fine.
>
> I would have assumed that rsyslog would attempt processing of messages from any queue in the order received though...  No?
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
> -----Original Message-----
> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
> Sent: Monday, July 10, 2017 4:56 PM
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Split messages options
>
> note that there is some potential for messages to get out of order (over the
> network with UDP, and inside rsyslog if you use disk queues or multiple threads)
>
> UDP messages can be dropped if the network is busy as well (by any
> router/firewall or receiving host)
>
> but the vast majority of the time, everything will be in order.
>
> David Lang
>
> On Mon, 10 Jul 2017, Scot Kreienkamp wrote:
>
>> Date: Mon, 10 Jul 2017 20:49:42 +0000
>> From: Scot Kreienkamp <[hidden email]>
>> Reply-To: rsyslog-users <[hidden email]>
>> To: rsyslog-users <[hidden email]>
>> Subject: Re: [rsyslog] Split messages options
>>
>> That makes sense, I wasn't aware of the limitation in log4j.  And yes, it is a very old implementation and getting them to update would likely take years.  :-)
>>
>> In my case I'm transporting the logfile from the prod servers to a common collector server internally that the devs can have access to; all I need to do is reconstitute the file exactly as is on the other side.  I'll change syslogappender to a non-default port and use a custom template to forward it on so I can force the correct tag on all forwarded messages, that way the splitting won't matter and the logfile will be written on the destination exactly as the source.  That will solve my problem for now.
>>
>> Thanks for the bit about log4j.  I'll have to do some more research there.
>>
>>
>> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
>> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]
>> -----Original Message-----
>> From: rsyslog [mailto:[hidden email]] On Behalf Of David Lang
>> Sent: Monday, July 10, 2017 4:38 PM
>> To: rsyslog-users <[hidden email]>
>> Subject: Re: [rsyslog] Split messages options
>>
>> ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
>> size, so it splits things before they get to rsyslog, and your maxmessagesize
>> isn't going to help.
>>
>> normally I am not in favor of writing to disk and then reading them, but if you
>> are stuck with an old log4j, that may be your best option.
>>
>> older log4j implementations are UDP only with a 1000 byte max size
>>
>> slightly newer implementations support TCP with larger message sizes (but I've
>> seen some that still limit you to 2k)
>>
>> current implementations are far more flexible.
>>
>> David Lang
>>
>>> I'm not bringing them in with imfile, log4j  is using the builtin syslog appender to submit it directly to syslog over UDP 514, the default UDP listener port.  I was assuming rsyslog is doing the splitting of the messages due to size.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>> This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...