file offset as imfile metadata

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

file offset as imfile metadata

Derek DiFilippo
Hi,

This is my first email to the rsyslog list. I've been getting to know
rsyslog over the last few months and it's been a pleasure to work with,
thank you.

******

I would like file offset metadata to reliably order messages on a receiving
machine.

I'm hoping for something like the following in imfile.c. This diff wouldn't
compile, I know -- it's just here to make the idea concrete:
diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c
index 758d8273..b5b2ce64 100644
--- a/plugins/imfile/imfile.c
+++ b/plugins/imfile/imfile.c
@@ -547,6 +547,7 @@ static rsRetVal enqLine(lstn_t *const __restrict__
pLstn,
        MsgSetRuleset(pMsg, pLstn->pRuleset);
        if(pLstn->addMetadata)
                msgAddMetadata(pMsg, (uchar*)"filename",
pLstn->pszFileName);
+               msgAddMetadata(pMsg, (uchar*)"fileoffset",
pLstn->pStrm->iCurrOffs);
        ratelimitAddMsg(pLstn->ratelimiter, &pLstn->multiSub, pMsg);

to enable a template like:
$template
fwdtemplate,"%hostname%^%$!metadata!filename%^$!metadata!fileoffset%^%msg%"

Is this change something I could submit as a pull request or one that a
regularly contributing developer could easily implement?

******

Here's the use case. As I mentioned, I want the file offset metadata to
reliably order messages on the receiving machine.

I have three machines running rsyslog.

Messages are written to files on machine A and we process those messages on
machine C after passing through a queuing rsyslog on machine B.

A: imfile (matching by glob) --> omfwd (adds filename metadata via
template) --> to B
B: imtcp --> omfwd (with disk-backed linkedlist action queue) --> to C
C: imtcp --> omprog (running a custom message parser) --> to our application

Occasionally the files on machine A are raw XML and I'm seeing the file
reconstructed 'perfectly' on machine C, same number of lines, same number
of bytes, but out of order and therefore not valid XML.

I am happily using rsyslog for other traditional syslog message processing
and could keep that system in place if I had access to the fileoffset in
addition to the file name.

Thank you for your time,
-Derek.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: file offset as imfile metadata

David Lang
Please post this as an issue on github

https://github.com/rsyslog/rsyslog

David Lang

On Fri, 23 Jun 2017, Derek DiFilippo wrote:

> Date: Fri, 23 Jun 2017 20:02:27 -0700
> From: Derek DiFilippo <[hidden email]>
> Reply-To: rsyslog-users <[hidden email]>
> To: [hidden email]
> Subject: [rsyslog] file offset as imfile metadata
>
> Hi,
>
> This is my first email to the rsyslog list. I've been getting to know
> rsyslog over the last few months and it's been a pleasure to work with,
> thank you.
>
> ******
>
> I would like file offset metadata to reliably order messages on a receiving
> machine.
>
> I'm hoping for something like the following in imfile.c. This diff wouldn't
> compile, I know -- it's just here to make the idea concrete:
> diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c
> index 758d8273..b5b2ce64 100644
> --- a/plugins/imfile/imfile.c
> +++ b/plugins/imfile/imfile.c
> @@ -547,6 +547,7 @@ static rsRetVal enqLine(lstn_t *const __restrict__
> pLstn,
>        MsgSetRuleset(pMsg, pLstn->pRuleset);
>        if(pLstn->addMetadata)
>                msgAddMetadata(pMsg, (uchar*)"filename",
> pLstn->pszFileName);
> +               msgAddMetadata(pMsg, (uchar*)"fileoffset",
> pLstn->pStrm->iCurrOffs);
>        ratelimitAddMsg(pLstn->ratelimiter, &pLstn->multiSub, pMsg);
>
> to enable a template like:
> $template
> fwdtemplate,"%hostname%^%$!metadata!filename%^$!metadata!fileoffset%^%msg%"
>
> Is this change something I could submit as a pull request or one that a
> regularly contributing developer could easily implement?
>
> ******
>
> Here's the use case. As I mentioned, I want the file offset metadata to
> reliably order messages on the receiving machine.
>
> I have three machines running rsyslog.
>
> Messages are written to files on machine A and we process those messages on
> machine C after passing through a queuing rsyslog on machine B.
>
> A: imfile (matching by glob) --> omfwd (adds filename metadata via
> template) --> to B
> B: imtcp --> omfwd (with disk-backed linkedlist action queue) --> to C
> C: imtcp --> omprog (running a custom message parser) --> to our application
>
> Occasionally the files on machine A are raw XML and I'm seeing the file
> reconstructed 'perfectly' on machine C, same number of lines, same number
> of bytes, but out of order and therefore not valid XML.
>
> I am happily using rsyslog for other traditional syslog message processing
> and could keep that system in place if I had access to the fileoffset in
> addition to the file name.
>
> Thank you for your time,
> -Derek.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: file offset as imfile metadata

Rainer Gerhards
In reply to this post by Derek DiFilippo
A PR would be very welcome by me.

Rainer

Sent from phone, thus brief.

Am 24.06.2017 06:02 schrieb "Derek DiFilippo" <[hidden email]>:

> Hi,
>
> This is my first email to the rsyslog list. I've been getting to know
> rsyslog over the last few months and it's been a pleasure to work with,
> thank you.
>
> ******
>
> I would like file offset metadata to reliably order messages on a receiving
> machine.
>
> I'm hoping for something like the following in imfile.c. This diff wouldn't
> compile, I know -- it's just here to make the idea concrete:
> diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c
> index 758d8273..b5b2ce64 100644
> --- a/plugins/imfile/imfile.c
> +++ b/plugins/imfile/imfile.c
> @@ -547,6 +547,7 @@ static rsRetVal enqLine(lstn_t *const __restrict__
> pLstn,
>         MsgSetRuleset(pMsg, pLstn->pRuleset);
>         if(pLstn->addMetadata)
>                 msgAddMetadata(pMsg, (uchar*)"filename",
> pLstn->pszFileName);
> +               msgAddMetadata(pMsg, (uchar*)"fileoffset",
> pLstn->pStrm->iCurrOffs);
>         ratelimitAddMsg(pLstn->ratelimiter, &pLstn->multiSub, pMsg);
>
> to enable a template like:
> $template
> fwdtemplate,"%hostname%^%$!metadata!filename%^$!metadata!
> fileoffset%^%msg%"
>
> Is this change something I could submit as a pull request or one that a
> regularly contributing developer could easily implement?
>
> ******
>
> Here's the use case. As I mentioned, I want the file offset metadata to
> reliably order messages on the receiving machine.
>
> I have three machines running rsyslog.
>
> Messages are written to files on machine A and we process those messages on
> machine C after passing through a queuing rsyslog on machine B.
>
> A: imfile (matching by glob) --> omfwd (adds filename metadata via
> template) --> to B
> B: imtcp --> omfwd (with disk-backed linkedlist action queue) --> to C
> C: imtcp --> omprog (running a custom message parser) --> to our
> application
>
> Occasionally the files on machine A are raw XML and I'm seeing the file
> reconstructed 'perfectly' on machine C, same number of lines, same number
> of bytes, but out of order and therefore not valid XML.
>
> I am happily using rsyslog for other traditional syslog message processing
> and could keep that system in place if I had access to the fileoffset in
> addition to the file name.
>
> Thank you for your time,
> -Derek.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Loading...