liblognorm rule for nginx logs

classic Classic list List threaded Threaded
10 messages Options
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

liblognorm rule for nginx logs

Luv
I am sending logs to elasticsearch via rsyslog. For the parsing of those logs, I am using liblognorm rule.

I want to create fields of nginx logs,

here is a log entry,
 
    127.0.0.1 - kibanaadmin [13/Jun/2017:14:18:17 +0530] "GET /ui/favicons/favicon-32x32.png HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"


Here is the pattern file,

    version=2

    rule=:%clientip:ipv4% - %user:word% [%timestamp:char-to:]%] %auth:word% "%verb:alpha% %request:word%" %response:number% %bytes:number% "%referrer:word"%" "%agent:char-to:{"extradata":"("}"

The reason for parsefailure is I believe due to the date-time format.

Can somebody help in creating a rule for parsing nginx logs ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

David Lang
On Tue, 13 Jun 2017, Luv via rsyslog wrote:

> I am sending logs to elasticsearch via rsyslog. For the parsing of those
> logs, I am using liblognorm rule.
>
> I want to create fields of nginx logs,
>
> here is a log entry,
>
>    127.0.0.1 - kibanaadmin [13/Jun/2017:14:18:17 +0530] "GET
> /ui/favicons/favicon-32x32.png HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11;
> Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
>
>
> Here is the pattern file,
>
>    version=2
>
>    rule=:%clientip:ipv4% - %user:word% [%timestamp:char-to:]%] %auth:word%
> "%verb:alpha% %request:word%" %response:number% %bytes:number%
> "%referrer:word"%" "%agent:char-to:{"extradata":"("}"
>
> The reason for parsefailure is I believe due to the date-time format.

no, you get past that, your problem is in auth:word, there isn't an auth word in
this line, it goes directly to the "verb

> Can somebody help in creating a rule for parsing nginx logs ?

does nginx have an option to output in json?

what is the log format as defined in the nginx config?

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Luv
Hey david, thanks for replying.

I am using the default log format provided by nginx, here it is,

log_format compression '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Bob Gregory
In reply to this post by Luv
Hi Luv,

we use the following rules :

rule=http:%remote_addr:word% %ident:word% %auth:word%
[%timestamp:char-to:]%] "%method:word% %request:word%
HTTP/%httpversion:float%" %status:number% %bytes_sent:number%
"%referrer:char-to:"%" "%agent:char-to:"%"%blob:rest%

rule=http:%remote_addr:word% %ident:word% %auth:word%
[%timestamp:char-to:]%] "%method:word% %request:word%
HTTP/%httpversion:float%" %status:number% %bytes_sent:number%
"%referrer:char-to:"%" "%agent:char-to:"%"

rule=http: %remote_addr:word% %ident:word% %auth:word%
[%timestamp:char-to:]%] "%method:word% %request:word%
HTTP/%httpversion:float%" %status:number% %bytes_sent:number%
"%referrer:char-to:"%" "%agent:char-to:"%"%blob:rest%

rule=http: %remote_addr:word% %ident:word% %auth:word%
[%timestamp:char-to:]%] "%method:word% %request:word%
HTTP/%httpversion:float%" %status:number% %bytes_sent:number%
"%referrer:char-to:"%" "%agent:char-to:"%"


Our nginx access log rules look like this:

    log_format main '$remote_addr - $remote_user [$time_local] "$request "'
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';


On Tue, 13 Jun 2017 at 10:49 Luv via rsyslog <[hidden email]>
wrote:

> I am sending logs to elasticsearch via rsyslog. For the parsing of those
> logs, I am using liblognorm rule.
>
> I want to create fields of nginx logs,
>
> here is a log entry,
>
>     127.0.0.1 - kibanaadmin [13/Jun/2017:14:18:17 +0530] "GET
> /ui/favicons/favicon-32x32.png HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11;
> Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
>
>
> Here is the pattern file,
>
>     version=2
>
>     rule=:%clientip:ipv4% - %user:word% [%timestamp:char-to:]%] %auth:word%
> "%verb:alpha% %request:word%" %response:number% %bytes:number%
> "%referrer:word"%" "%agent:char-to:{"extradata":"("}"
>
> The reason for parsefailure is I believe due to the date-time format.
>
> Can somebody help in creating a rule for parsing nginx logs ?
>
>
>
>
> --
> View this message in context:
> http://rsyslog-users.1305293.n2.nabble.com/liblognorm-rule-for-nginx-logs-tp7592454.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Luv
In reply to this post by Luv
Moreover, I have removed that auth part now, still it is not working.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Luv
In reply to this post by Bob Gregory
Hi bob, that was very helpful and this time, the logs were parsed.

But I am facing a new problem,



I think kibana is facing difficulty in getting a timefield for this. Did you face it also ? Can you direct as to why this problem has come and also steps to solve this ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Bob Gregory
Hi Luv,

How are you sending the logs across to Elasticsearch? Without knowing a lot
more about your setup, I can't help with that one. Assuming that your index
really is called `aaaaaaaaa` what are the fields defined in there? How did
you configure the index?

 -- B

On Tue, 13 Jun 2017 at 11:33 Luv via rsyslog <[hidden email]>
wrote:

> Hi bob, that was very helpful and this time, the logs were parsed.
>
> But I am facing a new problem,
>
> <
> http://rsyslog-users.1305293.n2.nabble.com/file/n7592459/Screenshot_from_2017-06-13_16-00-57.png
> >
>
> I think kibana is facing difficulty in getting a timefield for this. Did
> you
> face it also ? Can you direct as to why this problem has come and also
> steps
> to solve this ?
>
>
>
> --
> View this message in context:
> http://rsyslog-users.1305293.n2.nabble.com/liblognorm-rule-for-nginx-logs-tp7592454p7592459.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Luv
Here bob, this is my configuration for sending the logs to elasticsearch.

template(name="all-json-nginx"
    type="list"){
 property(name="$!all-json")
 }


if $programname == 'nginx' then {

action(type="mmnormalize"
  rulebase="/opt/rsyslog/apache.rb"   # file where rules are kept for parsing
)

if $parsesuccess == "OK" then {
action(type="omelasticsearch"
  template="all-json-nginx"  # use the template defined earlier
  searchIndex="aaaaaaaaaaaa"
  searchType="nginx"
  server="127.0.0.1"
  serverport="9200"
  bulkmode="on"  # use the bulk API
  action.resumeretrycount="-1"  # retry indefinitely if Logsene/Elasticsearch is unreachable
) } else action(type="omelasticsearch"
  template="all-json-apache"  # use the template defined earlier
  searchIndex="nginx-logs-2"
  searchType="nginx"
  server="127.0.0.1"
  serverport="9200"
  bulkmode="on"  # use the bulk API
  action.resumeretrycount="-1"  # retry indefinitely if Logsene/Elasticsearch is unreachable
)

} else {
 action( name="all-logs"
       type="omelasticsearch"
       template="JSONDefault"
       server="127.0.0.1"
       serverport="9200"
       searchIndex="test1"
       bulkmode="on"
       action.resumeretrycount="-1"
)
}




And here are the fields in index "aaaaaaaaaaaa",

      {
        "_index" : "aaaaaaaaaaaa",
        "_type" : "nginx",
        "_id" : "AVyhAYODBtmjLGraDsh5",
        "_score" : 1.0,
        "_source" : {
          "blob" : " \"-\"",
          "agent" : "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0",
          "referrer" : "http://127.0.0.1:8012/app/kibana",
          "bytes_sent" : "980",
          "status" : "200",
          "httpversion" : "1.1",
          "request" : "/elasticsearch/aaaaaaaaaaaa/_mapping/field/*?_=1497349849968&ignore_unavailable=false&allow_no_indices=false&include_defaults=true",
          "method" : "GET",
          "timestamp" : "13/Jun/2017:16:00:49 +0530",
          "auth" : "kibanaadmin",
          "ident" : "-",
          "remote_addr" : "127.0.0.1",
          "event.tags" : [
            "http"
          ]
        }


The timestamp format is causing this problem I think. Can you suggest something ?
Luv
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

Luv
Hey Bob,

the problem was solved.

I replaced $time_local with $time_iso8601 in the Nginx Log Format.

Thanks for your so valuable support.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: liblognorm rule for nginx logs

matthew.gaetano
FYI - The default timestamps in rsyslog are not supported in elasticsearch. You either need to convert them in rsyslog (liblognorm in your case) or via your elasticsearch mappings configuration. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html 
~Regards

Matthew Gaetano
Loading...