Feature Request: Advanced Column Configuration

WARNING: You currently have Javascript disabled!
This website will not function correctly without Javascript enabled.

On logs that have several header lines prior to the actual columnar data i usually have to manually pare the file down for the column data to work properly. The other issue is multiple separators. Variable separators (for space formatted logs) the option to treat multiple consecutive separators as a single separator would enable logs that have variable separators between columns of data of one or more separators to more cleaning parse otherwise its just better to load the file without column structuring on these types of files in my experience.

Oct 29, 2011 • #1

I've added the option for treating multiple consecutive separators as one column to our feature request list.

As for the header lines before the columnar data, could you provide an example log file that has this issue?

Thanks!

Nov 3, 2011 • #2

This is only an example but occurs throughout much of I.T. to where a log file attempts to explain the subsequent lines of data

Code
Copy
Select All

TypeDateTimeSourceCategoryEventUserComputer
Error11/3/20119:31:08 PMInvScanNone14N/ASRV001
Information11/3/20118:49:25 PMSymantec AntiVirusNone7N/ASRV001
Information11/3/20118:48:00 PMWinMagicSDNone0N/ASRV001
Information11/3/20118:46:36 PMSceCliNone1704N/ASRV001
Information11/3/20118:46:20 PMSymantec AntiVirusNone14N/ASRV001
Information11/3/20118:45:43 PMbtwdinsNone0N/ASRV001
Information11/3/20118:45:25 PMSecurityCenterNone1807N/ASRV001
Information11/3/20118:45:22 PMVMware NAT ServiceNone1000N/ASRV001
Information11/3/20118:45:21 PMVMware NAT ServiceNone1000N/ASRV001
Information11/3/20118:45:13 PMMSSQL$SQLEXPRESSServer 3408N/ASRV001

When the column feature attempts to structure this file into columns it gets caught up on the initial lines being set into columns and the subsequent lines not being seperated correctly.

Nov 4, 2011 (modified Nov 4, 2011) • #3

Awesome, thanks! I'll check it out and report back.

Nov 4, 2011 • #4

I just noticed that this forum software doesnt allow tabs to be inserted into the post text. Im not sure how to properly represent the file. I'll grab another one and see if i can just attach it.. If you want to replicate what i did, I just exported the windows event log.

Nov 5, 2011 • #5

It seems to work ok for me. I exported the System Event Log as a Tab Separated text file. Opened it in LogFusion and added 6 tab separated columns. See the attached screenshot for the output.

Perhaps I'm testing it differently than you are? If you could attach a log that's broken, along with a screenshot of the column configuration in LogFusion, that would be fantastic.

Thanks!

• Attachment: System Event Log with Columns.jpg [426,043 bytes]

System Event Log with Columns.jpg

Nov 8, 2011 (modified Nov 8, 2011) • #6

Could you also provide a sample log for the variable separators, and let me know how you currently configure the columns, and how you would like to be able to configure them? There are a few different ways we could handle the variable separators, so we just want to make sure we're understanding your request fully.

Thanks!

Nov 8, 2011 • #7

Quote:

Below is what i call a dirty log where the separator ISN'T the space in between the different columns of data. Rather its parsed via regex where the DATE is filtered into a column follow by a single space and then a single digit followed by a single space followed by a right justified (Variable number of white-space) number of variable length followed by a right justified number of variable length again followed by variable length text. This is a nightmare to attempt any kind of separator defined column separation.

Code
Copy
Select All

Time    Sev Major Minor Args
Dec 31 19:00:00 2  19 1800 logline detailed text here
Dec 31 19:00:00 2   5     0 logline detailed text here
Dec 31 19:00:00 1  99 0 logline detailed text here
Dec 31 19:00:00 1  99 0 logline detailed text here
Dec 31 19:00:00 2  19 1800 logline detailed text here
Dec 31 19:00:00 2   5   100 logline detailed text here
Dec 31 19:00:01 5 10000 101 logline detailed text here
Dec 31 19:00:02 3 10000 101 logline detailed text here
Dec 31 19:00:02 5 10000 101 logline detailed text here
Dec 31 19:00:02 6 10000 101 logline detailed text here
Dec 31 19:00:02 6 10000 101 logline detailed text here
Dec 31 19:00:02 3 10000 101 logline detailed text here

• Attachment: example_log_lines.png [60,577 bytes]

example_log_lines.png

Nov 9, 2011 (modified Nov 9, 2011) • #8

Thanks David! We'll see what we can do to make this easier to setup

Nov 11, 2011 • #9

In Beta 3 we've added an option to strip out multiple consecutive separators and treat them as one.

As for parsing columns using RegEx, we won't be able to include it in this version, but we'll work on it for LogFusion 2.1.

Thanks!

Nov 15, 2011 • #10

Great this makes the columns very readable. I have a few more features beyond the basics but this makes the feature very workable.

When possible can we still have "Remove header line(s) from beginning of log" as an option on the column configuration page?

Nov 16, 2011 • #11

Glad to hear it! I had missed the request about removing the column headers from the log but I've added it to our list now. We'll see what we can do

Nov 16, 2011 • #12

Was this helpful?

(-)