Ravens PHP Scripts: Forums
 

 

View next topic
View previous topic
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - Feedback
Author Message
Dawg
RavenNuke(tm) Development Team


Joined: Nov 07, 2003
Posts: 910

PostPosted: Wed Dec 05, 2007 11:00 pm Reply with quote

My suggestion is to have IP to country have a seperate prefix in the config so that sites could share the IP to C tables. Then to take that one set further and also have a prefix for the NS tables.

Here is what I have come up againest.....I wanted to share IP to C tables amoung several sites. So I went in and made all the edits...and got it working....then they released new tables ....had to update that....then I had to update NS to 14....now I am back to square 1. I am not going to do this everytime NS is updated or the IP to C tables are updated.

It would be nice if we could give one prefix to the IP to C tables and even nicer if we could do the same to the NS tables so we could also share the Blocked info.....

I am sure I am not the only one that more than one nuke site.

Just My .02....

Dawg
 
View user's profile Send private message
evaders99
Former Moderator in Good Standing


Joined: Apr 30, 2004
Posts: 3221

PostPosted: Wed Dec 05, 2007 11:52 pm Reply with quote

It's a matter of finding all those ip2country definitions and changing that prefix value. I use $ip2c_prefix myself.

_________________
- Only registered users can see links on this board! Get registered or login! -

Need help? Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
Dawg
PostPosted: Wed Dec 05, 2007 11:58 pm Reply with quote

That is what I did the first time. Haveing to do it after every upgrade is a pain unless there is an easier way of doing it. Is there a program out there that will search through and entire dir and search/replace all of them?
Dawg
 
xblader
Client


Joined: Aug 17, 2006
Posts: 28

PostPosted: Thu Dec 06, 2007 9:18 am Reply with quote

If you got access to the command line on the server you can use the gawk command to search all the files within a directory and replace it with your value.
 
View user's profile Send private message
fkelly
Former Moderator in Good Standing


Joined: Aug 30, 2005
Posts: 3312
Location: near Albany NY

PostPosted: Fri Dec 07, 2007 7:34 pm Reply with quote

This is a good idea for more reasons than you state. I've been working on trying to rescue the backup application on the ACP for RN2.20. Aside from just basic bugs in the application, it also chokes on the size of a typical Nuke database, especially with the Sentinel tables, given the resource constraints of most shared servers. We can code around it by having selectable downloads, for say all tables with "nsnst" after the prefix or all tables with "bb" at the start of the name after $prefix_ but it would be much more reliable if we had dedicated prefixes to subdivide the database. That's not a short term fix though for sure.

Both editors I work with, UEstudio and PHPED will do search and replaces throughout a directory structure, although in this case I'd want to do a prompted replace rather than a global one. If this is going to be done though it would be best to get it built right in at some future release level to avoid having to redo it every time.
 
View user's profile Send private message Visit poster's website
fkelly
PostPosted: Sat Dec 08, 2007 10:19 am Reply with quote

Thinking further on this I have a simple question. Would it be possible, feasible and desirable to eliminate the $prefix entirely. Just 'hard-code' a prefix into any table names. Core Nuke stuff could be 'nuke'. Gallery is already 'g2'. Sentinel already uses 'nsnst' but the prepends the prefix. How about just 'nsnst'?

It would be a lot of work going thru the whole code base and making the changes but no worse than the recent conversion to single quotes and probably easier once we come up with a global approach to search and replaces. I doubt we'd want a separate hard coded prefix for each module but new module authors could have that option to easily distinquish their tables from the rest. Afterthought, probably the PHPBB tables should be 'phpbb' rather than just 'bb'.
 
Dawg
PostPosted: Sun Dec 09, 2007 5:26 pm Reply with quote

fkelly,
Along those same lines what about a way to SET the prefixes for select MODS in the config.

Dawg
 
fkelly
PostPosted: Mon Dec 10, 2007 9:48 am Reply with quote

Dawg, yes that would make sense. Along with the capability to not use prefixes. But this is something that will require further thought and discussion once 2.20 is out the door ... if we choose to do anything. The reason is that it affects the code base so much. Where we have a statement such as:

Code:
$result = $db->sql_query('select bid, bkey, title, url, bposition, weight, active, blanguage, blockfile, view from '.$prefix.'_blocks order by bposition, weight');


if we didn't use the prefix we could eliminate the '.$prefix.'_ entirely. In the grand scheme of things this would be a minor efficiency improvement and would require thousands of coding changes (hummm I guess for backwards compatibility we could define a variable $prefix = '' and name the table _blocks) but it still might be worth discussion. I'm really not sure we gain anything by having $prefix or that anyone uses anything other than Nuke, though what Evaders posted might indicate otherwise.

Just thinking out loud at this point.
 
Dawg
PostPosted: Mon Dec 10, 2007 10:15 am Reply with quote

fkelly,
Going back to the NS tables, It would seem to me to be a minor modification to NS to call the prefix from the config file. This would make the backup system less bloated and make it so people like me could share the NS tables across muli-sites.

On down the road if we so desired we could add support to RN for the rest of MODS as time (and desire) allowed.

For backwards compatability I would guess an if statement would have to be included somewhere so the prefix would default to nuke if it is not defined within config. I do not know NS's codeing that well so I have not idea how much time engery and effort that would require

I think the biggest issue I see is what Bob thinks about this idea. HE would be the one that would have to support the idea. Becasue if not....let's face it to would become a maintaince nghtmare to have to change the code with every new release vs having it included in the new releases....

Does he read here? his reputation is MOST excellent! Maybe he will find this idea worthy....

Dawg
 
Raven
Site Admin/Owner


Joined: Aug 27, 2002
Posts: 17077

PostPosted: Mon Dec 10, 2007 1:02 pm Reply with quote

Dawg wrote:
My suggestion is to have IP to country have a seperate prefix in the config so that sites could share the IP to C tables. Then to take that one set further and also have a prefix for the NS tables.

Here is what I have come up againest.....I wanted to share IP to C tables amoung several sites. So I went in and made all the edits...and got it working....then they released new tables ....had to update that....then I had to update NS to 14....now I am back to square 1. I am not going to do this everytime NS is updated or the IP to C tables are updated.

It would be nice if we could give one prefix to the IP to C tables and even nicer if we could do the same to the NS tables so we could also share the Blocked info.....

I am sure I am not the only one that more than one nuke site.

Just My .02....

Dawg


Dave,

What bearing does the prefix have on sharing tables? Updates are updates and either you upgrade/update or you don't. Forgetting the rest of this thread, I am not following your first post as to how changing the prefix helps/hinders what you are trying to do.

You have a special case as you are trying to share your tables among sites all on the same server. That is easy to do. It's trying to share different sites on different servers that becomes an issue.
 
View user's profile Send private message
Dawg
PostPosted: Mon Dec 10, 2007 1:41 pm Reply with quote

Raven,
I went through and changed the prefix in the NS code so that all the sites could share the IP to C tables although sharing all the NS tables would have been even nicer.....

The problem was when they released the update there are all these changed files. So my choice was to either go through all the new files everytime there is an update and do the same work over again OR have each site run on it's own set of tables. (the default setup)

NS puts out a lot of updates over the coarse of a year. IP to C puts out a lot of updated over a year.

That is a lot of work to do every time there is an update released to go through and change all the prefixes so my suggestion was to have NS run on a seperate prefix that was settable in config so people like me could share the tables right out of the box without having to alter the code. IF done with a 0-1 switch it would be something that would go unnoticed by 95% of the users but is nice for people sharing the tables like me.

I am not special....I would guess that a lot of the "Faithful" are fairly advanced Nuke users that run multi-sites.

It is just a suggestion...
worship

Dawg
 
Raven
PostPosted: Mon Dec 10, 2007 2:10 pm Reply with quote

Laughing

Let me try this again. When I said special, that was a compliment, meaning you are in more of a unique/special situation that allows you advantages over other site operators. The fact that you are able to have all of your sites on the same server makes it much easier to do this kind of a thing.

Now, to the meat of things.

Quote:
That is a lot of work to do every time there is an update released to go through and change all the prefixes

I assume you are talking about the individual sql files and not the table, correct? Assuming that to be the case there are 2 really easy options in addition to a shell command processor (like awk, sed, grep, etc.). A good text editor like Textpad allows you to mass change all occurrences of a string an all selected files. So a mass change is relatively easy and is how I handle most of my tasks. You then do a save all and Textpad makes it easy. Another option is to use php and write (or find) a script to do a find and a replace.

The $prefix is basically for organizational purposes. To have a separate prefix for NukeSentinel(tm) is a minor change and we could implement that w/o any problem at all. However (there always is one and I think it's mentioned or alluded to above), with different prefixes for different applications you will need to alter base/core code to accommodate the changes and that could get ugly because it would basically mean that all script writers would need to follow the "new" standard. As we develop RavenNuke(tm) for example, we do have a separate config but our intent is to use it to stay out of the way of the standard nuke code as much as is possible and to not make it conform to RN. That way we impose little on the community script writers.

So, I'm not sure where to go from here. it seems to me, at least for the interim, that the text editor mass change is the easiest and less impacting way to go. Something else to think about as I have been toying with this idea and if/when I get the time I am going to try it. I am very much into using MEMORY tables. The IP2C table is a prime candidate for a MEMORY table. Basically you load it into memory each time the server starts/boots and every application has access to it if you let them. You could even make it in a database of its own for even greater security. Since it's in memory the speed is as fast as your FSB and memory. It only changes when you decide to update it so there's no conflict there either. Right now the table is around 2.4meg so memory-wise it's a drop in the bucket. Of course with MySQL5 and PHP5 you have several other options but we won't go there Wink
 
kguske
Site Admin


Joined: Jun 04, 2004
Posts: 6383

PostPosted: Mon Dec 10, 2007 9:21 pm Reply with quote

Sorry to jump in here, but how about a completely different approach - having NO IP2C tables? Seriously (and I suggested this some time ago):

Why not develop a service that provides this information - one source, many clients (e.g. XML / SOAP)? Whomever maintains the server could even charge a small fee for access (or you could host your own server, if you like).

Of course, there might be performance issues with that approach, but that depends on how it's used (and may require some additional changes to be completely effective). But it certain eliminates the frequent updates and a copy for each website.

I'd bet many people would pay a small annual fee for this service without having to update the data.

_________________
I google, therefore I exist...
Only registered users can see links on this board! Get registered or login!
 
View user's profile Send private message
kguske
PostPosted: Mon Dec 10, 2007 9:23 pm Reply with quote

For reference, not only could multiple Nuke instances share an IP2C database, but multiple applications (i.e. NukeSentinel, MS Analysis, etc.) could share it on those multiple instances.
 
Raven
PostPosted: Mon Dec 10, 2007 9:36 pm Reply with quote

Good options/suggestions - food for thought. All viable with all having pros/cons. The data is available on the Internet already - that's where we get it or at least we used to. We deliver/include it for convenience.
 
Dawg
PostPosted: Mon Dec 10, 2007 10:14 pm Reply with quote

OK.....Now we are getting somewhere. If we went that path would it not make sence to also share the bann items. I am not thinking IPs as such but more things like strings and domains.

I bet there are a TON of folks that wold pay $10.00 a year for the shared info. (Raven, Bob.....Income stream)

I know I would....

Take a string of known sites like Ravens.....not just anyones site but use ones that are known and respected....have them feed the data to the server and then allow sites to subscribe to the feed with IPTC and bans.

That goes right along with Raven Nuke is all about. Nuke secured.

Just My .02

Dawg

PS....
Raven, I have not forgotten about your post above but I have to work real early in the morning. I will post a reply tomorrow evening when I get back.
 
Raven
PostPosted: Mon Dec 10, 2007 10:28 pm Reply with quote

I'm up for hosting this unless there are objections.
 
montego
Site Admin


Joined: Aug 29, 2004
Posts: 9449
Location: Arizona

PostPosted: Tue Dec 11, 2007 7:09 am Reply with quote

I think two models would be very useful:

Model 1 - IP2C "Lite"

This model would be the externally hosted web service which does the check and no IP2C data is held locally. This might possibly extend even into a centralized banned IP service, but that might be more trouble than its worth.

Model 2 - IP2C Update Service

This is the one that I would be interested in. For those who have the disk space and want the best possible performance, the issue really is getting the updates applied. I could see this service providing more of a delta update type process. The update transactions only are fed down once per defined time period and either automatically applied, or maybe the web master is notified that the updates are available and has to submit something.

I really do not like the current approach of truncating and re-loading if that can be avoided here. But, then again, I am the first to admit that I have never dissected the nature of the original updates and how difficult / easy that would be.

_________________
Only registered users can see links on this board! Get registered or login!
Only registered users can see links on this board! Get registered or login! 
View user's profile Send private message Visit poster's website
fkelly
PostPosted: Tue Dec 11, 2007 10:44 am Reply with quote

This very much "squares" with the backup project I am working on for 2.20. Right now most sites can't run the ACP based backup program because the database is too big and they run up against max cpu limits. And most sites will have problems restoring, at least until they get BigDump running, because resource limits get exceeded. Eliminating the need to backup and restore IP2Country would be a big contribution to getting around this problem. Hummm ... just thinking out loud again, but some of the Bulletin Board tables are huge also and could probably be recreated by a script after a restore of the base tables was done.
 
Raven
PostPosted: Tue Dec 11, 2007 11:55 am Reply with quote

I found this script earlier to day and haven't had time to play with it but it may be useful.
Code:
This function will read, compress, and write only small chunks at one time, this way you can compress big files without memory problems...


<?php
function gzip($src, $level = 5, $dst = false){
    if($dst == false){
        $dst = $src.".gz";
    }
    if(file_exists($src)){
        $filesize = filesize($src);
        $src_handle = fopen($src, "r");
        if(!file_exists($dst)){
            $dst_handle = gzopen($dst, "w$level");
            while(!feof($src_handle)){
                $chunk = fread($src_handle, 2048);
                gzwrite($dst_handle, $chunk);
            }
            fclose($src_handle);
            gzclose($dst_handle);
            return true;
        } else {
            error_log("$dst already exists");
        }
    } else {
        error_log("$src doesn't exist");
    }
    return false;
}
?>
 
fkelly
PostPosted: Tue Dec 11, 2007 12:11 pm Reply with quote

That could be helpful. Not to get off the topic of the thread too much but right now the backup from the ACP doesn't do compression at all ... which is another issue I suppose. I was thinking we should almost do gzip output by default since both phpMyAdmin restore and bigdump restore will handle it.
 
Gremmie
Former Moderator in Good Standing


Joined: Apr 06, 2006
Posts: 2415
Location: Iowa, USA

PostPosted: Tue Dec 11, 2007 12:32 pm Reply with quote

Perhaps it is as simple as distributing the IP2C data as a flat file, along with a PHP script. The PHP script will ask you for the database name, prefix, etc, and then drop the old data and insert the new. To accomodate several sites you simply upload the flat file to the server then run the script several times for each site.

You may still have resource limit issues crunching through all that data, but this could be addressed in the script.

And, as Devil's advocate, you could simply not use IP2C. Sentinel will run without it. I've never felt a reason to use it....yet..... Wink

Of course maybe I'm not understanding the problem since I've never had to use/maintain these tables.

_________________
Only registered users can see links on this board! Get registered or login! - An Event Calendar for PHP-Nuke
Only registered users can see links on this board! Get registered or login! - A Google Maps Nuke Module 
View user's profile Send private message
fkelly
PostPosted: Tue Dec 11, 2007 3:18 pm Reply with quote

Actually, I think the first paragraph of your response Gremmie is what Nuke Sentinel does with the import feature now. The import data is a flat file and Sentinel stuffs it into the IP2country table, if I am not mistaken.

As to using IP2country, it all depends on your site. I run a site that's primarily intended for cyclists in upstate NY. I don't need Russia or England or Spain or a bunch of other countries in. I've banned a bunch of them and now when I go into IP tracking and see a hack attempt from a country I didn't think of banning, I just ban that too. It cuts down on a lot of gunk that normally would get thrown at the site. Mostly when I look at IP tracking now I see good old USA IPs. Obviously hackers can circumvent this but every one that gets denied at the front door is one less to deal with.
 
Guardian2003
Site Admin


Joined: Aug 28, 2003
Posts: 6792
Location: Ha Noi, Viet Nam

PostPosted: Thu Dec 13, 2007 1:46 am Reply with quote

IP2C data should be written to XML.
You could then either use the data to write into your local site tables or even cache it as a file which would then auto update itself after a pre-determined time.

kguske has mentioned this before and I think we did have some good discussion on it.
One of the benefits of course is that some 'approved' sites who have their own VPS or dedicated servers could also be utilised to spread the load so the data fetches are not all coming from one place - if one service is down another would be able to provide the data seamlessly, you could even have a ping service to let you know which service is the nearest or quickest.

Other advantages of course would be for something like Spam Stopper so users could even share their own blacklists or get updates from the latest hacking attacks and block access to those sites which have been indetified as being compromised.

The possiblilities are endless and certainly within the realms of 'do-able'.
I for one would be more than willing to contribute time and resources on such a project.
 
View user's profile Send private message Send e-mail
kguske
PostPosted: Thu Dec 13, 2007 9:14 pm Reply with quote

Since both Montego and Guardian brought up the idea of a central attacker / spam database, I'll weigh in on that now. Sorry, but that's a bad idea. (you guys KNOW how much I respect you, but for everyone else this is most definitely NOT personal).

The IP2C database is official (or at least maintained by an accepted "authority"). Creating a database of blocked IPs is simply asking for trouble. It's fine for a single website to do that (or even a single webmaster who supports multiple sites), but making it available to others - as a publicly available list - is akin to publishing a list of child molesters who haven't been convicted in a court of law. With rampant IP spoofing, I have to agree with Gremmie's point (I think) that using IP addresses for blocking is practically pointless (pun intended). More to the point, I've seen especially heinous attackers spoof search engine bot IPs KNOWING that the resulting block will harm the site!

That said, having accurate IP2C data can be very useful in analyzing traffic (e.g. NukeSentinel, MS Analysis) or in localizing some site functions (e.g. default timezone for registration, etc.).

Guardian, the idea of having multiple servers for an IP2C database is very interesting. I'm not sure how that would work, and I'm concerned about performance if the first site you try (assuming you could try more than 1) isn't available. But I'd certainly be interested in ideas on how to make that work.

When I first considered a central IP2C database, I wondered what functions / transactions it would serve. That could alter the costs and benefits for various alternatives. It might also justify more significant modifications to the applications using the data. For example, you might have a screen that displays a lot of IP2C data at once. Having that on a remote server would require either a transaction with a lot of data or many single transactions. Sure, the details could be worked out, but it should force the question - does having this data on one screen justify the cost? If such conditions exist and the answer is yes, it might be better to have a local copy of the data with a way to keep it synchronized with the master database (IP2C Update Service). If not, an IP2C lite (I'd prefer to call it Remote IP2C) might be more appropriate.

I think if we do some analysis, we'll find both scenarios can be justified. The bottom line is that Montego is dead on: having both solutions would be ideal. Those who don't need local copies can save on space and backup time by using a remote IP2C service. Those who do need it can probably justify the expense of having local data. The benefit of the analysis is that is supports (and may be required anyway) for both solutions.

Anyone have the bandwidth for that kind of analysis?? I didn't when I originally suggested it to Bob some time ago, and probably don't now, either.

Finally, if we do develop such a service (or both), it could be applied to the private webmaster, shared block database approach. But I'd much rather be sharing htaccess techniques for blocking attacks at the server level than maintaining a database of blocked (spoofed) and protected IPs...
 
Display posts from previous:       
Post new topic   Reply to topic    Ravens PHP Scripts And Web Hosting Forum Index -> RN v2.20.00 - Feedback

View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2007 phpBB Group
All times are GMT - 6 Hours
 
Forums ©