Long line, can't understand forward truncation reason

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Long line, can't understand forward truncation reason

Scot Kreienkamp
Hi everyone,

I have a program that is sending long lines into the logs.... they are arriving at the local server and into the logs correctly but are being truncated when being received on a remote server, and I can't understand why.

In crontab I have a program being run like so:
*/5 * * * * java  foo |& nc  -w 240 localhost 1516

Basically it's running the java program and redirecting all IO to netcat, which is sending it to port 1516 on localhost via the default TCP.  That is working fine, and the local logfile has all the content in it.  The problem comes when I'm trying to forward to a remote host.  The local file has a line that goes on for pages, however the remote end writes the line into the file but truncates that line after approximately 950 characters and then does really weird things with the rest of that incoming line.  Sometimes it shows up under other servers (the destination separates by hostname and tag), sometimes it creates new folders with partial names of servers, etc.


Here's my rule that encompasses the local file write and the forward:

template(name="RMS-ForwardDMA" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    constant(value="retv6030.na.lzb.hq")
    constant(value=" ")
    constant(value="RMS-DMA:")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
}
ruleset(name="RMS-DMA-1516-1531"
        queue.filename="Net-1516-1531"
        queue.dequeuebatchsize="1024"
        queue.discardmark="45000"
        queue.DiscardSeverity="0"
        queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on"
        queue.size="50000"
        queue.spoolDirectory="/var/spool/rsyslog"
        queue.type="LinkedList"
){
        action(
                name="omfile-RMS-ForwardDMA"
                type="omfile"
                dynafilecachesize="5"
                DynaFile="DailyPerHostLogDMA"
                template="msgonly-no1sp"
                ioBufferSize="64k"
                flushOnTXEnd="off"
                asyncWriting="on"
                dirCreateMode="0755"
        )
        action(
                name="fwd-monvsyslog-1531-RMS-DMA"
                queue.dequeuebatchsize="1024"
                queue.discardmark="45000"
                queue.DiscardSeverity="0"
                queue.maxDiskSpace="1g"
                queue.saveOnShutdown="on"
                queue.size="50000"
                queue.spoolDirectory="/var/spool/rsyslog"
                queue.type="LinkedList"
                queue.filename="fwd-monvsyslog-1531-RMS-DMA"
                type="omfwd"
                TCP_Framing="octet-counted"
                protocol="tcp"
                Target="monvsyslog.na.lzb.hq"
                Port="1531"
                template="RMS-ForwardDMA"
        )
        stop
}
#setup local port listeners for ecomm
input(type="imudp" port="1516" address="127.0.0.1" ruleset="RMS-DMA-1516-1531")
input(type="imptcp" port="1516" address="127.0.0.1" ruleset="RMS-DMA-1516-1531")



And here's my receiving line:

        if $syslogtag startswith "RMS-DMA:"  then {
                action(name="Net-1531-DMA" type="omfile" dynafilecachesize="50" DynaFile="DailyPerHostLogDMA" ioBufferSize="128k" flushOnTXEnd="off" asyncWriting="on" dirCreateMode="0755" template="msgonly")
                stop
        }


Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162  | * 734-384-6403 | | * 7349151444 | *  [hidden email]<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> | twitter.com/lazboy<https://twitter.com/lazboy> | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>

[cid:lzbVertical_hres.jpg]



This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

lzbVertical_hres.jpg (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Long line, can't understand forward truncation reason

David Lang
On Tue, 4 Dec 2018, Scot Kreienkamp wrote:

> Hi everyone,
>
> I have a program that is sending long lines into the logs.... they are
> arriving at the local server and into the logs correctly but are being
> truncated when being received on a remote server, and I can't understand why.

what is your maxmessagesize on the two systems? that sounds like the cause of
your problem.

when you exceed maxmessagesize, that starts the next message, and since there
isn't the syslog header at that point, it parses incorrectly, causing the type
of problem that you are having.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: Long line, can't understand forward truncation reason

Rainer Gerhards
Is this an old version? Current ones should report oversize messages and do
not break but truncate (except if you Vogue differently iirc).

Rainer

Sent from phone, thus brief.

Am Mi., 5. Dez. 2018, 01:10 hat David Lang <[hidden email]> geschrieben:

> On Tue, 4 Dec 2018, Scot Kreienkamp wrote:
>
> > Hi everyone,
> >
> > I have a program that is sending long lines into the logs.... they are
> > arriving at the local server and into the logs correctly but are being
> > truncated when being received on a remote server, and I can't understand
> why.
>
> what is your maxmessagesize on the two systems? that sounds like the cause
> of
> your problem.
>
> when you exceed maxmessagesize, that starts the next message, and since
> there
> isn't the syslog header at that point, it parses incorrectly, causing the
> type
> of problem that you are having.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: Long line, can't understand forward truncation reason

Scot Kreienkamp
In reply to this post by David Lang
>
> > Hi everyone,
> >
> > I have a program that is sending long lines into the logs.... they are
> > arriving at the local server and into the logs correctly but are being
> > truncated when being received on a remote server, and I can't understand
> why.
>
> what is your maxmessagesize on the two systems? that sounds like the cause of
> your problem.
>
> when you exceed maxmessagesize, that starts the next message, and since
> there
> isn't the syslog header at that point, it parses incorrectly, causing the type
> of problem that you are having.
>
I had thought the same, but I have:


$MaxMessageSize 8192k
global(workDirectory="/var/spool/rsyslog" preserveFQDN="on" maxMessageSize="8192k")

at the top of my receiver config, and the sender has

$MaxMessageSize 7168k
global(workDirectory="/var/spool/rsyslog" preserveFQDN="on" maxMessageSize="7168k")


But what gets me is the small amount of message that's received... that seems smaller than even the default.

Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|

Re: Long line, can't understand forward truncation reason

Scot Kreienkamp
In reply to this post by Rainer Gerhards
We're running 8.38 currently, waiting for the next patch cycle for 8.39.  Not seeing any kind of truncation messages.


Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  Mobile: 7349151444 | Email: [hidden email]

> -----Original Message-----
> From: rsyslog [mailto:[hidden email]] On Behalf Of Rainer
> Gerhards
> Sent: Wednesday, December 5, 2018 2:51 AM
> To: rsyslog-users <[hidden email]>
> Subject: Re: [rsyslog] Long line, can't understand forward truncation reason
>
> Is this an old version? Current ones should report oversize messages and do
> not break but truncate (except if you Vogue differently iirc).
>
> Rainer
>
> Sent from phone, thus brief.
>
> Am Mi., 5. Dez. 2018, 01:10 hat David Lang <[hidden email]> geschrieben:
>
> > On Tue, 4 Dec 2018, Scot Kreienkamp wrote:
> >
> > > Hi everyone,
> > >
> > > I have a program that is sending long lines into the logs.... they are
> > > arriving at the local server and into the logs correctly but are being
> > > truncated when being received on a remote server, and I can't understand
> > why.
> >
> > what is your maxmessagesize on the two systems? that sounds like the cause
> > of
> > your problem.
> >
> > when you exceed maxmessagesize, that starts the next message, and since
> > there
> > isn't the syslog header at that point, it parses incorrectly, causing the
> > type
> > of problem that you are having.
> >
> > David Lang
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.