/usr/share/doc/httpry/doc/perl-tools is in httpry 0.1.7-3.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | A number of Perl scripts are included with the project to provide a basic log
parsing framework. There is a central Perl script (parse_log.pl) that handles
all of the core log parsing duties, and a set of Perl plugins (plugins/*.pm)
with associated config files (plugins/*.cfg) that perform assorted processing
duties. It is straightforward to create or modify the plugins to customize
processing.
*** IMPORTANT! ***
The log parser will attempt to use all plugins that it finds in the default
plugin directory. To override this behavior, use the -p switch to specify
plugins by path and name and/or use -d to specify an alternate directory to
use. If the plugin requires a configuration file, it must be located in the
same directory as the plugin and named '<plugin name>.cfg'. Additionally, any
input log file needs to be a valid httpry log with the field specifier at the
start of the file.
These non-core Perl modules must be installed to run all scripts:
DBI
Included Perl scripts:
---------------------
./scripts/parse_log.pl
Core script for plugins. Processes log files and delivers data to each
initialized plugin. Plugins are utilized by having a .pm extension
and being placed in the plugin directory (default is ./plugins). If
a plugins directory is specified (-d), that is used instead. If
no directory is specified, the script first searches the current
directory for a plugin directory followed by the base directory of
the script. The first directory found is used. To load only specifc
plugins, use the -p switch to provide a comma-delimited list of
plugins (-d is not applied here, so specify them by their full path).
Most included plugins require a config file, which must be named
'<plugin name>.cfg' and placed in the same directory as the plugin.
Included plugins for parse_log.pl:
---------------------------------
./scripts/plugins/common_log.pm
Outputs the log data in the standard common logfile format. This
allows webserver log analysis tools to be applied to the captured
data.
./scripts/plugins/content_analysis.pm
Breaks the input file into flows, which are time delimited blocks of
log lines by client IP. It then searches for specified keywords and
scores the flows accordingly. Outputs a summary file listing all
scored flows, as well as flow data for each scored flow. Requires an
input file containing search terms.
./scripts/plugins/db_dump.pm
Dumps log file data into a previously initialized database. Useful
for more extensive data mining or adding a more user-friendly web
interface. Use the included db_dump.mysql to create the required
database tables.
./scripts/plugins/find_proxies.pm
Performs some basic tests looking for proxy usage. The tests are
currently rather rudimentary and generate quite a lot of noise.
./scripts/plugins/hostnames.pm
Outputs a list of unique hostnames with counts found in the log data.
./scripts/plugins/log_summary.pm
Provides summary information regarding the input files including
visited hosts, top talkers and file extensions, along with other
assorted data. Useful for a high-level overview of the input files.
./scripts/plugins/sample_plugin.pm
An example plugin with no functionality designed to demonstrate how
plugins are structured and to provide a start for new plugins.
./scripts/plugins/search_terms.pm
Outputs a raw dump of all entered search terms for a number of the
major search services. It also calculates the average length of
extracted search queries.
./scripts/plugins/tokenize.pm
Outputs a set of tokens for each source IP address, broken into
separate output files (a token here is defined as a sequence of
alphanumeric characters delimited by non-alphanumeric characters).
./scripts/plugins/xml_format.pm
Outputs the input log data in XML format.
|