There's a little SE statistician inside every and each of us

Posted: Sat Nov 06, 2004 1:23 pm

Cheers,

As per real-life discussions for some week ago here at Ravens' Forums, I decided to have a closer look at the possible means of keeping oneself up-to-date with Site Specific Search Engine Activities from statistics/monitoring point of view. As a result of it, I'm coming up with a minor app in few days that allows keeping track of ("9+1") SE's throughout the Site predictable Life Span. ["(9) pcs in main engines' AT-category" + "(1) in branch others AT-category"]

This straightforward PHP-Nuke dedicated Tool, actually Combo, will allow tracking down the received hits from the (9+1) major "SE's"/"web Crawler's", whereas the identification of those parties/agents is based solely on IP detection (upon the most up-to-date SE specific IP information.) The PHP-Nuke Module is based on heavy modification/explosion of PHP-Nuke "Advanced Statistics" MODULE, which was chosen as a modification/application basis due to extremely clear an well accomplished core struct - (1) pcs of 30kBytes file(s) only.

The package will incorporate a MySQL Dump Generation Tool being generated and tested for these purposes exclusively - so, not only will the ToolKit allow User to integrate the simple Add-On core into User's PHP-Nuke site, but it will also allow the User to carry out MySQL DumpGen Sessions for those purposes, whenever he/she wishes to update the MySQL DB Tables (SE specific/dedicated) regarding the available refreshed/updated Search Engine specific information.

The actual results on received/detected hits will be then summarized/categorized as follows;

* yearly basis per SE [single view][+dynamically genearated PNG graphs]
* monthly basis per SE [single view]["all in parallel view"][+dynamically genearated PNG graphs]
* daily basis per SE [single view]["all in parallel view"][+dynamically genearated PNG graphs]
* hourly basis per SE [single view]["all in parallel view"][+dynamically genearated PNG graphs]

So, the SE Activity is sliceable/trackable from Top-Down perspective as follows:

--> Choose Year --> (inspect Monthly Summary) <---> "navigate in between" (SingleViews/MultiViews)<--->
--> Choose Month --> (inspect Daily Summary) <---> "navigate in between" (SingleViews/MultiViews)<--->
--> Choose Day --> (inspect Hourly Summary)

The above core approach should provide sufficient basis for reviewing/examining tendencies over the "yearly/monthly/weekly/daily" periods - i.e. progress vs. regression & Site Specific SE Big Picture from historical Viewpoint.

Currently, the DumpGen Tool can handle both full octet IP Ranges and single IPs - in case of detected IP Ranges the lowest 8-bit address space (octets) are exploded into a set of single IPs, which are then further being processsed into required MySQL Database format.

What possible could this Package offer vs. "wheels already invented"?

Well, nothing glamorous for sure - there are many superb tools around, and by no means is intended to compete against those masterpieces - the key point is that it'll simply bring in an potential alternative to perform the SE Activity surveillance "from inside the Nuke" with reduced/limitted options to configuralibility - i.e. easy to use, but still performing. Some of the positive outcomes may be associated as follows;

* End-User executable, eased up SE specific IP DB updates with MySQL DumpGen
* Yearly-Monthly-Daily-Hourly Stats[*|MultiViews|SingleViews]
* pre-configure ParallelVies - "key sorted" vs "SE sorted" view in parallel
categorizable/trackable SE Activity SingleView/MultiView listings
<-> Clear categorized Table Presentation
<-> Supportive Dynamic PNG Stats Graphs
* SE's interrelational activity comparisons enpowered

Intend to release also the most up-to-date SE Specific IPs as separate montly releases (MySQL Dumps), for those who do not wish to get involved with the MySQL DumpGen phase themselves.

::: Package SE Specifics/Details :::

Number of unique IPs to be tracked per "SE"/"WebCrawler" (current status : Nov-2nd-2004)

[0] * Alta Vista unique tracked IPs : (3957)
[1] * Ask Jeeves/Teoma unique tracked IPs : (366)
[2] * Excite unique tracked IPs : (543)
[3] * FAST/AllTheWeb unique tracked IPs : (1328)
[4] * Google unique tracked IPs : (5327)
[5] * InfoSeek unique tracked IPs : (1629)
[6] * Inktomi unique tracked IPs : (5481)
[7] * Lycos unique tracked IPs : (2074)
[8] * MSN unique tracked IPs : (894)
[9] * Northern Light unique tracked IPs : (524) <--- (I selected this as a class "other" option at my end...)

Here's a couple of screenshots, yet evolving, from preliminary version:

http://www.ec-clan.org/EXAMPLE_SCREENSHOTS/eC-SE-AT-INTRO_today_hourly.gif
http://www.ec-clan.org/EXAMPLE_SCREENSHOTS/eC-SE-AT-INTRO_month_daily_top_view.gif
http://www.ec-clan.org/EXAMPLE_SCREENSHOTS/eC-SE-AT-INTRO_year_montly_top_view.gif

...and just from the curiosity point of view - a SHELL window view at intermediate *Google* MySQL Dump generation stage:

http://www.ec-clan.org/EXAMPLE_SCREENSHOTS/ec_se_at_mysql_dump_gen.gif

Talk to you more soon in few days.

Thanks,

-beetraham