Microsoft Log Parser Toolkit: A complete toolkit for Microsoft's undocumented log analysis tool [NOOK Book]

Overview

HIGHLIGHT
Written by Microsoft's Log Parser developer, this is the first book available on Microsoft's popular yet undocumented log parser tool. The book and accompanying Web site contain hundreds of customized, working scripts and templates that system administrators will find invaluable for analyzing the log files from Windows Server, Snort IDS, ISA Server, IIS Server, Exchange Server, and other products.

...
See more details below
Microsoft Log Parser Toolkit: A complete toolkit for Microsoft's undocumented log analysis tool

Available on NOOK devices and apps  
  • NOOK Devices
  • Samsung Galaxy Tab 4 NOOK
  • NOOK HD/HD+ Tablet
  • NOOK
  • NOOK Color
  • NOOK Tablet
  • Tablet/Phone
  • NOOK for Windows 8 Tablet
  • NOOK for iOS
  • NOOK for Android
  • NOOK Kids for iPad
  • PC/Mac
  • NOOK for Windows 8
  • NOOK for PC
  • NOOK for Mac
  • NOOK for Web

Want a NOOK? Explore Now

NOOK Book (eBook)
$41.95
BN.com price

Overview

HIGHLIGHT
Written by Microsoft's Log Parser developer, this is the first book available on Microsoft's popular yet undocumented log parser tool. The book and accompanying Web site contain hundreds of customized, working scripts and templates that system administrators will find invaluable for analyzing the log files from Windows Server, Snort IDS, ISA Server, IIS Server, Exchange Server, and other products.

System administrators running Windows, Unix, and Linux networks manage anywhere from 1 to thousands of operating systems (Windows, Unix, etc.), Applications (Exchange, Snort, IIS, etc.), and hardware devices (firewalls, routers, etc.) that generate incredibly long and detailed log files of all activity on the particular application or device. This book will teach administrators how to use Microsoft's Log Parser to data mine all of the information available within these countless logs. The book teaches readers how all queries within Log Parser work (for example: a Log Parser query to an Exchange log may provide information on the origin of spam, viruses, etc.). Also, Log Parser is completely scriptable and customizable so the book and accompanying Web site will provide the reader with hundreds of original, working scripts that will automate these tasks and provide formatted charts and reports detailing the results of the queries.

* Written by Microsoft's sole developer of Log Parser, this is the first book available on the powerful yet completely undocumented product that ships with Microsoft's IIS, Windows Advanced Server 2003, and is available as a free download from the
Microsoft Web site.

* The book and accompanying Web site contain dozens of original, working Log Parser scripts and templates for Windows Server, ISA Server, Snort IDS, Exchange Server, IIS, and more!

* This book and accompanying scripts will save system administrators countless hours by scripting and automating the most common to the most complex log analysis tasks.

Written by Microsoft's Log Parser developer, this is the first book available on Microsoft's popular yet undocumented log parser tool. The book and companion Web site contain hundreds of customized, working scripts and templates that system administrators will find invaluable for analyzing the log files from Windows Server, Snort IDS, ISA Server, IIS Server, Exchange Server, and other products.

Read More Show Less

Product Details

  • ISBN-13: 9780080489391
  • Publisher: Elsevier Science
  • Publication date: 2/10/2005
  • Sold by: Barnes & Noble
  • Format: eBook
  • Edition number: 1
  • Pages: 350
  • Sales rank: 1,050,026
  • File size: 26 MB
  • Note: This product may take a few minutes to download.

Read an Excerpt

Microsoft Log Parser Toolkit


By Gabriele Giuseppini Mark Burnett

Syngress

Copyright © 2004 Syngress Publishing, Inc.
All right reserved.

ISBN: 978-0-08-048939-1


Chapter One

Introducing Log Parser

Scripts and Samples in this Chapter:

* Retrieving fields from the Event Log * Searching for Information * Ordering the Output * Top-Ten Lists * Parsing Text Files * Querying System Information * Parsing NetMon Files * Implementing Custom Input Formats * Using the DATAGRID Output Format * Creating Custom Text Files * Creating Charts * Uploading Query Results to a SQL Database * Converting Log File Formats

In This Toolbox

Log files—records of events that have occurred in a system—often contain information deemed crucial to a business. However, most times log files can be so large that it becomes almost impossible to extract useful data or find meaningful patterns in the amount of information that they provide. The Log Parser tool allows users to treat log files and other information as SQL tables, the rows of which can be queried, processed, and formatted in different ways. The power of the SQL language, together with the variety of data that can be processed by Log Parser, make it possible to extract specific information, calculate statistics, and generate reports from large amounts of data.

A Brief Background

The Log Parser tool was born around 2000 as a utility to test the logging mechanisms of Microsoft's Internet Information Services (IIS). The first inception of the tool allowed users to retrieve and display all the fields from a single log file in any of the three text-logging formats supported by IIS.

As tests became more complicated, the need arose for more specialized tasks, including the possibility to filter log entries matching specific criteria and to sort the resulting entries according to values of specific fields. To provide a succinct and well-established way to specify these operations from a command-line shell, the tool underwent its first major makeover and began to support a very limited dialect of the SQL language. Log Parser 1.0 was born, and its use began spreading among internal Microsoft users and product support analysts.

After some time, the SQL language dialect processor was completely redesigned, enriched with functions, aggregate functions, and the GROUP BY clause, and with improved performance characteristics; at the same time, the tool underwent a second major architectural makeover, which separated the log file parsers and the console output formatter from the core SQL engine, making it possible for generic "input formats" and "output formats" to easily plug in with the new architecture. Just for fun, the first non-IIS input formats made their appearance, including the Event Log (EVT) input format and the File System (FS) input format, together with the first nonconsole output format, the SQL output format. It didn't take long for IIS program managers to notice that the tool's capabilities could be leveraged by end users as well, greatly simplifying most of the tasks related to processing of log files. In fact, in November 2001 Log Parser 2.0 made its public debut as a freely downloadable tool from the pages of www.microsoft.com.

The user response was so favorable that the new 2.1 version of the tool, which included many new input and output formats, was included in the IIS 6.0 Resource Kit Tools, published in April 2002.

The tool has been continuously improved since then, thanks especially to the feedback and suggestions of its many users; new input and output formats have been added, and the core SQL engine has been improved with new functions and better-performing algorithms.

This 2.2 version marks the latest release of the Log Parser tool, designed and engineered with the vision of helping users achieve their data-processing goals in a simple, fast, and powerful way.

Building Queries

With the Log Parser tool, users can run queries written in a dialect of the Structured Query Language (SQL) on data coming from different sources. The data processed by the queries is provided by input formats; these are data processors that enumerate information from different sources and provide the data as input records on which the queries operate. Once a query is executed, it generates output records, which are handed to output formats—data processors that consume output records, format data, and send them to different targets.

Retrieving Fields from the Event Log

For our first example of a Log Parser command, we'll start with a simple, basic task: retrieving some fields from the Event Log.

To run this command, start a command-line shell, navigate to the folder where the LogParser.exe command-line executable has been copied or make sure that the folder is in the system path, and type the following command:

C:\>LogParser -i:EVT -o:NAT "SELECT TimeGenerated, SourceName FROM System"

The output will look like this:

TimeGenerated SourceName

18:48:04 EventLog 2004-04-18 18:48:04 EventLog 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager 2004-04-18 18:48:27 Service Control Manager Press a key ...

Let's analyze this command in detail. The first argument is -i:EVT, which tells Log Parser that we want to use the EVT input format—that is, the input format that enumerates the events in the Windows Event Log. In fact, the -i parameter is used to indicate which input format we want our query to operate on.

The second argument is -o:NAT, which tells Log Parser that we want to use the NAT output format—that is, the output format that displays the query results in a readable, nicely formatted way. In fact, the -o parameter is used to indicate which output format we want our query results processed by.

The third argument is our SQL query. In this example, the query is a very simple one: The SELECT clause specifies which of the input record fields we want to be handed to the output format, and the FROM clause specifies where we want to retrieve the input records from.

In our example, we have specified that we want to retrieve two fields: the TimeGenerated field and the SourceName field. These are just two fields out of the 15 fields exported by the EVT input format. We can see what fields are exported by the EVT input format by typing the following help command:

C:\>LogParser -h -i:EVT

The output of this help command displays some useful information on the EVT input format, which includes a list of the exported fields:

Fields:

EventLog (S) RecordNumber (I) TimeGenerated (T) TimeWritten (T) EventID (I) EventType (I) EventTypeName (S) EventCategory (I) EventCategoryName (S) SourceName (S) Strings (S) ComputerName (S) SID (S) Message (S) Data (S)

The FROM clause that we have used specifies System as the data source that we want the EVT input format to retrieve records from. This means that we want to retrieve records from the System Event Log; if we wanted to retrieve records from the Security Event Log, we could have specified Security in the FROM clause. If we wanted to retrieve events from both the System and Security Event Logs, we could have specified both in our FROM clause, using a comma character as a separator, this way:

C:\>LogParser -i:EVT -o:NAT "SELECT TimeGenerated, SourceName FROM System, Security"

When we execute commands using the NAT output format, we can see that the format displays 10 lines and then prompts the user to press a key before displaying the next 10 lines. This behavior can be overridden by specifying a value for the rtp parameter of the NAT output format.

Most of the Log Parser input and output formats accept parameters that are used to fine-tune their behaviors to match our needs. The rtp parameter of the NAT output format specifies the number of rows to print before prompting the user to press a key. If we want to change the value of this parameter to 5, for example, we would type the following command:

C:\>LogParser -i:EVT -o:NAT -rtp:5 "SELECT TimeGenerated, SourceName FROM System"

Specifying -1 for this parameter disables the prompt altogether and causes the NAT output format to display all the output records without interruption.

We can see a description of all the parameters supported by an input format by typing the input format help command that we have used before:

C:\>LogParser -h -i:EVT

We now have a command that displays selected fields from the Event Log to the console output. What if we want to write this output to a file instead?

Well, forget command-line redirection. In the Log Parser SQL-centric world, this can be accomplished using the INTO clause of the SQL language:

C:\>LogParser -i:EVT -o:NAT "SELECT TimeGenerated, SourceName INTO myoutput. Txt FROM System"

This command causes the NAT output format to write its output to the myoutput. Txt file instead of the console output. When an INTO clause is used, the NAT output format disables the user prompt between output records and writes all the output records to the file without interruption. When a query does not include an INTO clause, the NAT output formats assumes a default INTO clause with a value of STDOUT—that is, the console output.

The commands that we have used in these examples can be simplified by leveraging a couple of shortcuts. When the FROM clause in a query specifies a well-known value, Log Parser is able to automatically recognize the input format that should handle the FROM clause. This is true for the System or Security values we have used so far in our examples: Log Parser recognizes that System or Security are commonly used names of Event Logs and automatically selects the EVT input format for us.

The same applies for output formats: If a query does not include an INTO clause, Log Parser automatically selects the NAT output format. This means that we could rewrite our example command as follows:

logparser.exe file: "SELECT TimeGenerated, SourceName FROM System"

Thus we obtain the same effect as the fully specified command in our first example.

Retrieving Fields from IIS Logs

The simple command that we have used to display Event Log fields can be easily modified to display fields from an IIS Web log file. To do so, we need to first change the input format to the input format responsible for parsing the IIS logs in their currently configured format. Assuming that our IIS Web server is configured to log using the W3C Extended Log File Format, we need to pick the IISW3C input format.

Then we need to decide which of the IISW3C fields we want to retrieve. Again, we can see what fields are exported by this input format using the following help command:

Let's assume that we are interested in the date, time, and cs-uri-stem fields.

Finally, we need to change our FROM clause to specify the IIS log files that we are interested in parsing. We can do so in two ways: by specifying the paths to the log files or by specifying the ID number of a virtual IIS site whose logs we want to parse. Assuming that we want to parse the log files of the site with ID 1 (the default Web site), our command looks like the following:

C:\>LogParser -i:IISW3C -o:NAT "SELECT date, time, cs-uri-stem FROM <1>"

The output will look like this:

date time cs-uri-stem

2003-11-18 00:28:33 /Default.htm 2003-11-18 00:28:33 /style.css 2003-11-18 00:28:33 /images/address.gif 2003-11-18 00:28:33 /cgi-bin/counts.exe 2003-11-18 00:28:42 /gorice/rulesinfo.nsf 2003-11-18 00:28:42 /gorice/rulesinfo.nsf 2003-11-18 00:28:42 /gorice/rulesinfo.nsf 2003-11-18 00:28:42 /gorice/rulesinfo.nsf 2003-11-18 00:28:42 /gorice/rulesinfo.nsf 2003-11-18 00:28:42 /gorice/rulesinfo.nsf Press a key ...

(Continues...)



Excerpted from Microsoft Log Parser Toolkit by Gabriele Giuseppini Mark Burnett Copyright © 2004 by Syngress Publishing, Inc.. Excerpted by permission of Syngress. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Read More Show Less

Table of Contents

Parsing Logfile Data ; Assuring IIS Performance and Stability ; Tracking Down Intruders ; Getting the Most from Exchange Server Logs ; Finding Knowledge in the Windows EventLog ; Tracking ISA Server Logs ; Enhancing Log Parser with Custom Input Processors ; Formatting, Reporting, and Charting ; The Log Parser Security Audit ; Keeping an Eye on Critical Servers ; Working with Tricky Data ; Converting, Rotating, and Archiving Logs ; Log Parser: The All-Purpose Hacker's Tool ; Studying Firewall, IDS, and Router Logs ; Appendix A: Log Parser SQL Grammar Reference ; Appendix B: Log Parser Function Reference ; Appendix C: Log Parser Input Format Reference ; Appendix D: Log Parser Output Format Reference
Read More Show Less

Customer Reviews

Average Rating 5
( 3 )
Rating Distribution

5 Star

(3)

4 Star

(0)

3 Star

(0)

2 Star

(0)

1 Star

(0)

Your Rating:

Your Name: Create a Pen Name or

Barnes & Noble.com Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & Noble.com that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & Noble.com does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at BN.com or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation

Reminder:

  • - By submitting a review, you grant to Barnes & Noble.com and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Noble.com Terms of Use.
  • - Barnes & Noble.com reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & Noble.com also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on BN.com. It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

 
Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously
Sort by: Showing all of 3 Customer Reviews
  • Anonymous

    Posted April 15, 2005

    Stop looking - start parsing!

    Log Parser 2.2 is the lastest version of Microsoft's FREE ( I know, I can't believe it either!) tool for extracting useful information from log files, text files, Active directory, registries, spreadsheets, syslogs, you name it. This program is one of those things that keep making you wonder why everyone isn't using it. Albeit a little cryptic for the beginner, this book provides some invaluable examples and idea's to beat every server and network device you have into submission. Simply paging through this book and looking at the section headings has given me so many ideas that I designed a custom solution to monitor and manage every system on my ASP's network. In addition, the forum and web site information are first rate, and timely responses are par for the course. Gabriele has molded this program from the red-headed-step-child of Microsoft, into something they should be proud of! I have been a network engineer for almost 10 years, and this program is the first thing that has excited me since I cheated on my wife! From my workstation, I can get almost any kind of information, from any server, into any format using a few short command line statments. Just read the back cover and you'll see what I mean. Thanks Gabriele, your book got me a raise, and some extra time to play video games! (HINT: intergrate with Sharepoint services, and you'll be managments hero!)

    Was this review helpful? Yes  No   Report this review
  • Anonymous

    Posted February 22, 2005

    A must have for the Network Administrator as well as the Security Professional

    This is a complete reference for utilizing the Microsoft® Log Parser Tool in real world scenarios. The authors do an outstanding job of bringing you from the basics of Log Parser through advanced techniques and tricks. I have thoroughly enjoyed reading it end to end, and have begun utilizing Log Parser in my daily log assessment routines. The Tips, Swiss Army Knifes, and Master Craftsmen sidebars prove extremely creative and helpful.

    Was this review helpful? Yes  No   Report this review
  • Anonymous

    Posted February 18, 2005

    Certain to become an admin's best freind.

    Log Parser by it self is a wonderful tool, this book lets you get up and running with it in no time. Additionally it gives a great insight on logs. This book is essential for any admin that wants to keep with ¿the going on¿ on his network without running agents everywhere. Log Parser toolkit lets you make logs readable (what a novel concept) without the need for heavy programming, all the scripts are included and really easy to customize. If you keep any type of log this will make your life easier and can save you major head hakes. My only recommendation is when you get one for your self, don¿t forget to get one for your IIS admin and your security guy, or be ready to share it. Enjoy it¿

    Was this review helpful? Yes  No   Report this review
Sort by: Showing all of 3 Customer Reviews

If you find inappropriate content, please report it to Barnes & Noble
Why is this product inappropriate?
Comments (optional)