There is a simple way to do this, and that is to first of all, save the birth dates in a “date” type field.  This will usually be by default a pattern like so “YEAR-MONTH-DAY” or “0000-00-00″.  Once your dates are in that format, you can use MySQL to do a comparison like so:

select * from user where birthday between DATE_SUB(CURDATE( ),INTERVAL 59 YEAR)

and DATE_SUB(CURDATE( ),INTERVAL 50 YEAR);

This will give you results from the “user” table for users who are between the ages of 50 to 59 providing that the birthday column is a “date” type.  It actually took me a while to figure this one out because I cannot ever remember needing such a data result.  Searching around the Net, I found no examples on how to exactly do this, so I hope this will help someone else a few hours of coding. :)

Dear XXXX,

We have been informed that your PayPal Debit Card number may have been obtained by an unauthorized party as a result of a security breach at a merchant where your debit card was used to make a purchase. The security of your PayPal account information was not compromised in connection with this event. To help ensure your security, we have taken the proactive step of cancelling your PayPal Debit Card. To request a new debit card, please follow the instructions below:

   1. Log in to your PayPal account by opening a new browser.

   2. Click the "ATM/Debit Card" link in the footer of any PayPal page.

   3. Click "Request Debit Card."

   4. Make sure your name and address are correct.

   5. Click to confirm that you have read and agreed to the PayPal MasterCard ATM Debit Card Agreement.

   6. Click "Submit."

We also suggest that you check your recent History Log for any unauthorized activity. If any unauthorized activity has occurred with your PayPal Debit Card, please follow the instructions below to dispute the charge:

   1. Log into your PayPal account and click the 'Contact Us' link, located at the bottom of any PayPal page.

   2. Click on the 'Contact Customer Service' link.

   3. Select 'PayPal Debit Card' from the 'Choose a Topic' box.

   4. Select 'Unauthorized Transactions' from the 'Choose a Subtopic' box.

   5. Click the 'Affidavit Of Disputed PayPal ATM/Debit Card Transactions' link.

   6. Once completed, click 'Preview this Form'

   7. Click on ‘Print Form’

You can fax the affidavit to 303-395-XXXX or mail it to:

PayPal, Inc.

Attn: Debit Card Chargebacks

PO Box 45950

Omaha, NE 68145-0950

USA

In order to process claims for unauthorized account access in a timely manner, we must receive the affidavit within the first 60 days from the date of the transaction.

We want to assure you that security is a top priority at PayPal, and we appreciate your business.

Sincerely,

PayPal Account Review Department
PayPal, an eBay Company

Assuming it to be a scam but noticing there was not even one link in the email, I go to a local ATM to see and sure enough, the card is declined. I then find this thread over at the Ebay forums. Having used my Paypal debit card at Hannafords over the last 6-8 months it seems to have gotten my card destroyed. According to those at the Ebay forums, 15-20% of all Paypal debit cards were deactivated today.

Surely Paypal and/or Mastercard could have sent out the new cards first and then waited 7 – 10 days before canceling the old one?

No official release has been made by Paypal at the time of this writing.

According to others who have called Paypal, this move was requested by Mastercard. It is not known if other cards bearing the Mastercard logo that were used at Hannafords have been affected.

MemoryError

Basically, the script is calling the entire log file into memory before parsing it, and depending on your log file size and amount of memory on your machine, this could just be too much for it. So what do you do now?

Myself, I did what I assume you are doing right now and began Googling for answers, and other than an old fix for an older version (which didn’t work for the latest), nothing fixed the issue.

I just sat there wondering how I was going to shrink this log file down so the script could handle this, especially when I wasn’t there to monitor it (i.e. a cron job). I asked myself, “I wish I could just split it into smaller files”… Finally, the light came on: Duh, isn’t this one of the reasons the split command was created?

So I wrote up this little shell script that I could call via cron and placed it in a non web-accessible directory named sitemap (download here):

cd /home/username/sitemap

rm -f log-*

rm -f access_log

cp /var/log/access_log ./

split -l 50000 access_log log-

/usr/local/bin/python2.4 /home/username/sitemap/sitemap_gen.py  --config=/home/username/sitemap/config.xml

This worked like a charm. So what is happening here?

When the script is executed, it deletes all files in the sitemap directory that start with log- and the file access_log, copies the access_log file from the /var/log directory, splits the file into smaller files of 50,000 lines each, and then executes the sitemap generator.

The sitemap generator allows you to use a wild card for access logs, so I just told it my log starts with log- in my config.xml file:

<accesslog path="/home/username/sitemap/log-*" />

That was it! It can now handle any size log file. Now if you still get the MemoryError error, try reducing the size of the split files to less than 50000.

Happy hacking!

For those of us using Sendmail on some flavor of Linux, there are a few ways to slow down the barrage of junk coming in. Spamassassin, MailScanner, etc., and of course there are always black lists. One of the more popular is Spamcop. Spamcop can plug right into the Sendmail configuration files. It checks a sender against a known database of spamming IP addresses and will deny mail delivery. (More below)

This method is fine and will work but would it not be nice to take it a step further and once Spamcop has denied a sender, to just block them from even knocking on the mail servers door? Not only would this reduce the overhead of a Sendmail child, it would also stop another query to spamcop.net saving myself and them a bit more overhead/bandwidth.

After examining the maillog file generated by Sendmail, I found I was able to easily parse out the IP addresses of offending senders. From there, it was as simple as issuing a drop to IPTables of that IP. Basically this tells your Linux firewall to refuse connections from the IP address.

This worked good at first but I found that running a cronjob to go through the mail log and ban IP’s got rather resource intensive so I decided to rewrite it a bit so it had a memory and would only ban IP addresses that haven’t already been banned. This saved a lot of overhead.

After thinking about it a bit, I also decided that dropping the IP totally was overkill. Many of these spam bots are just home users who have no clue on how to secure their PC and why should I deny these idiots access to my websites/advertising? So I rewrote it just to block them from accessing port 25 (my SMTP port).

Here is the Perl script:

## The port you wish to block (25 is the normal SMTP port).

my $port = 25;

## Path to where you want the ip_hash.txt file stored (ip database of previously dropped IP's) - NO TRAILING SLASH!

my $path_to_ip_hash = '/home/anywhere';

## You should change the path here to where you uploaded serialize.pm.  Cron env probably won't find it otherwise.

use lib '/folder_where_this_file_is_located';

## Path and name of the maillog file.

my $maillog = '/var/log/maillog';

## Location of iptables.

my $iptables = '/sbin/iptables';

####################################### No more editing required #######################################

use serialize;

## If you need the serialize module, it is located here: http://www.hurring.com/scott/code/perl/serialize/

my $y = 0;

if ($ARGV[0] eq 'flush') {

my %ip_flush = ();

my $ip_flushed = serialize(\%ip_flush);

open (IPS, '>' . $path_to_ip_hash . '/ip_hash.txt');

flock(IPS, 2);

print IPS $ip_flushed;

close IPS;

print "List flushed\n";

exit;

}

open (IPS, $path_to_ip_hash . '/ip_hash.txt');

flock(IPS, 2);

my $ip_hash = <IPS>;

close IPS;

my $ip_list = unserialize($ip_hash);

my %ip_list = %$ip_list;

open(FH,$maillog);

my @log = <FH>;

close FH;

my ($ban, $key, @ip, @temp, %ban);

## Here is where the log is parsed, searches for rejected (by spamcop) mail attempts (553).
 ##
 ## You may need to alter this to parse your log files.  This was written for for mail logs generated by
 ##
 ## Sendmail 8.14.1 (and probably updated by now).  This line gives a positive result:
 ##
 ## Oct  9 01:50:13 ns1 sendmail[26040]: ruleset=check_relay, arg1=[123.123.123.123], arg2=127.0.0.1, relay=[123.123.123.123], reject=553 5.3.0 Mail Rejected: http://spamcop.net/bl.shtml?123.123.123.123
 ##

foreach (@log) {

if ($_ =~ m/reject\=553/) {

@temp = split(/\,/, $_);

@ip = split(/[\[\]]/, $temp[3]);

if (!(exists $ip_list{$ip[1]})) {

$ban{$ip[1]}++;
 $ip_list{$ip[1]}++;

}

}

}

## End parsing.  IP's are in the %ban hash. IP's are the key values.

foreach $key (keys %ban) {

print "Issuing $iptables -I INPUT -s $key -p tcp --dport $port -j DROP\n";

$ban = `$iptables -I INPUT -s $key -p tcp --dport $port -j DROP`;

## If you want to ban them completely, comment out the above line and use the next line instead.

#$ban = `$iptables -I INPUT -s $key -j DROP`;

$y++;

}

my @months = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);

my @weekDays = qw(Sun Mon Tue Wed Thu Fri Sat Sun);

my ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();

my $year = 1900 + $yearOffset;

my $theTime = "$hour:$minute:$second, $weekDays[$dayOfWeek] $months[$month] $dayOfMonth, $year";

print "\n$theTime: $y new bans " . (keys %ip_list) . " total bans.\n";

$ip_hash = serialize(\%ip_list);

open (IPS, '>' . $path_to_ip_hash . '/ip_hash.txt');

flock(IPS, 2);

print IPS $ip_hash;

close IPS;

1;

Make sure you download the module serialize from Scott Hurrings’ website and place it in the same directory where the above code will go.

Calling this file every ten minutes or so via a cron job will execute the parsing routine. You should test this out on the command line before using it live. The parsing routines work on my version of Sendmail on my flavor of Linux. As far as I know, it should work on any other flavor too. You will need to have Sendmail configured to use Spamcop.

All it does is parse the log file searching for 553 rejects and drops them via iptables. Previous drops are saved to a file that contains a serialized hash. I chose to store it this way so it could be easily accessed by PHP scripts too. This can come in handy if you want to show people the IP’s you have banned like I have done here.

Here is the PHP code to do this:

echo "<P>List of IP's recently banned:</P>";

$ips = file_get_contents('/path/to/ip_hash.txt');

$ips = unserialize($ips);

foreach ($ips as $key => $value) {

echo "<a href=http://spamcop.net/bl.shtml?$key>$key</a><BR>";

}

I piped the output of this file to a log in my /var/log directory and set up logrotate to handle the rotation. I have been watching it slowly ice spammers and have noticed a serious improvement in server load especially during peak spam times. It also seems to have slowed down the spam itself a bit.

So you are probably saying, ok, you saved some resources and have lowered your spam intake a little too, so what?

Well, first, I know that I am probably at least slowing a bit of the spam by rejecting packets from these spammers, but I think the bigger picture here is just the concept. If this type of approach was used by a majority of mail servers, would it not take a bite out of spam right there? Imagine every Linux box running Sendmail using this? It could easily be modified to use any other mail log or firewall. Would this not also reduce the usefulness of a bot army used for the purpose of sending spam?

Now of course, one could be concerned about false positives. This is something handled more on the Spamcop end of it, but flushing your IPTables and the above perl script from time to time will lower the chances of this. (use the flush command line argument to flush the database).

Anyway, feel free to comment. :)

When an advertiser creates an ad and puts it on the AdSense network, they bid on how much they want to pay for a click. If they have the highest bid, their ads are generally placed first in the ad blocks when their ads match and are displayed. This makes perfect sense if you think about it, they are paying the most, they should get top billing. Google of course wants the highest paying advertisers to get the best exposure so they can make their money. There is more to this when it comes to ‘Smart pricing’ but we will not get into that here. It is sufficient that you know the top paying ads are usually first in an ad block.

This may make you ask “But I have three ad blocks on each page, how does this affect the pay out of a click?”. AdSense will rarely display the same advertisement twice on the same page, in fact, I have never seen them do it. What this boils down to is, not only are the best paying links at the top of an ad block, they are also contained in the FIRST ad block you display. Display is actually not the proper way to say it, what I mean is, the first one the browser reads in your HTML source code for the page. The browser will generally execute javascript and anything else in the source code in a sequential order. So if you have three ad blocks on a page, the browser executes the javascript in the same order as you have it listed in the HTML source code.

Now we have established that the highest paying clicks come from the first ad block. How can you use this to earn more?

If you use channels (which everyone should), you should be able to tell which of your ad blocks have the highest Click Through Rating (CTR). If you log into your AdSense account, view the basic report for the last month and see how it broke down with your channels. Find the channel with the highest CTR for the month. This channel should always load first in the browser. If it loads second or third, then the lower paying clicks are in these blocks. The ad block with the highest CTR is your premium advertising space and you should get a premium price for it.

Redesign your code so the highest CTR block gets loaded first. There are many ways you can do this, but I would suggest doing it with CSS so if the CTR’s should change in the future, it will not take much work to adjust which block loads first.

In conclusion, make sure the area that gets the highest Click Through Rating loads the AdSense javascript first. Adjust your other areas so they also load in the order of which has the better CTR.

Once you find the IP address of the user(s) you wish to ban (example: 123.255.123.1), just add them to the .htaccess file with an entry like so:

deny from 123.255.123.1

Perhaps this isn’t enough because the user has a dialup account with their ISP and just hangs up, calls back, and gets a new IP. Most of the time, the ISP will have the same first 2 sets of numbers, as our example shows: 123.255. You can eliminate that entire range:

deny from 123.255.

You should be VERY CAREFUL when you do this because you could have just denied access to everyone from a popular ISP such as AOL, MSN, etc.. Sometimes if you are under attack from such users you have no choice but to temporarily ban an entire ISP. There are a few good tools to find out the ISP of an IP address:

American Registry for Internet Numbers (ARIN): http://arin.net/

&

RIPE NCC: http://ripe.net/

Generally, these services will not tell you any information about the user themselves, but will list out ISP information and how to contact their abuse departments.

Wouldn’t it be nice if the output could somehow be separated from the code so coders and designers don’t have to clash?

Well there is a way, and it is called The Smarty Template Engine.  What is the Smarty Template Engine?  The technical definition of a template engine is code that facilitates a manageable way to separate application logic and content from its presentation.  Basically, your script provides all the information to the engine, and Smarty uses it, along with a template file to display it with standard HTML tags.   There are many built in Smarty tags that allow you to perform many functions found in PHP if necessary.  You can even add your own functions to Smarty!

Smarty allows you to separate code from layout.  You do not need to add anything special to your PHP installation, and can include all the Smarty files with your application.  Just require smarty in your scripts and you can use the code to handle all of your output.  It even facilitates output buffering with filters.

One large benefit to using Smarty is caching.  Smarty can cache pages or even just parts of pages to drastically increase load times, especially on pages where content doesn’t change frequently.

You can find more on Smarty at the PHP web site here: http://smarty.php.net.

This is where a CMS will shine, but what is a CMS? A Content Management System primarily does exactly what it says: manages content! Generally a CMS is software that you install on your site that will allow you to add/edit content, manage layout, manage users, and add extra functionality to your site. Instead of creating a new page by hand, with all the HTML tags, etc., the CMS will generally present you with a WYSIWYG editor and let you enter the content and presents it on your site in a predetermined style. This is very similar say, to a message board or forum except with greater control.

One of the most popular CMS systems today is Joomla. The Joomla team is filled with most of the development staff of the CMS Mambo. The company that distributes Mambo parted ways with their developers and Joomla was born. Joomla is an open source CMS and is completely free. There are many other such systems out there such as PHPNuke, PostNuke, Wordpress, etc.. For this article, we will use Joomla as our primary example.

With Joomla as your CMS, you can write your content all day and save it to your servers SQL database and display it to your visitors. You can also create many users with different responsibilities such as authors, administrators, editorial, etc..

All of your content is now instantly searchable since Joomla comes with powerful search capabilities. This is something you just cannot do with static HTML pages without a lot of work, or waiting for a search engine to spider your site.

As with most CMS software, you can add on software to do many different things. In Joomla, these are called Modules, Components, and Mambots. With these add-ons, you can incorporate software such as forums, classifieds, webmail, etc., to your CMS giving you and your users many features. You can even write your own add-ons if you are proficient with PHP.

Another benefit using a CMS is consistency. Your pages will generally use the same style, keeping your layout consistent throughout your site. Want to change your layout? You do not have to change thousands of HTML pages, just one or two layout pages and they will instantly be used in all of your current content. Most CMS software uses templates, so you can get pre made styles for your site, or even hire a designer with knowledge of your CMS to design you something unique.

In the end, a good CMS will save you hours of time maintaining a site and give you and your users a much better experience.

For any operating system based on Unix (i.e. Linux), permissions are very important. There are many files you don’t want other files to touch. It wouldn’t be a good situation is for example, your forum PHP script started overwriting, or reading your servers password files, or deleted directories all over the place.

When using Perl, generally most operating systems will require you CHMOD a Perl script executable before it can be run. This is a security plus compared to PHP where generally, if the PHP file exists, it is automatically executable. So if you have a PHP script that can write to your server, and that script creates another PHP script, it could run arbitrary commands on your server. This is of course, not a good idea.

So what do the numbers mean when you see “CHMOD 755″. This is a bit technical, as the numbers are just a binary representation of a permission scheme.

The CHMOD command takes three arguments, for example:

“CHMOD rwx r-x r-x somefile.cgi” tells the server that the owner of the file can read it, write to it, and execute it. If this was a directory, substitute search for execute.

Breaking it down:

rwx <- file owner
r-x <- user group
r-x <- everyone else

How does this convert to numbers? Basically the "-" symbol equals 0, and everything else equals 1. So the first part of our CHMOD command: rwx equals 111, the second and third part, r-x equals 101. In binary, this comes to 755. Here is a simple binary chart:

So instead of writing “CHMOD rwx r-x r-x somedirectory” you can write “CHMOD 755 somedirectory”. Many FTP programs such as WS_FTP allow you to use check boxes to set permissions.

You should take care in setting files CHMOD 777. This basically means anyone can read/write/execute/search the file/directory.

Permission structures can be a bit tricky from server to server depending on file ownership. Usually only the file owner can change permissions. Most FTP programs will let you view the entire directory listing which should show you who owns the file, and what group has access to the file as an owner.

This comes in very handy for many reasons.  Perhaps you have written a CGI or PHP script that reads the server load, and will notify you by email if that load is high.  It doesn’t do you much good unless it runs every few minutes.  With Crontab, you can execute it as often as you wish.

What do you do if you have no access to Crontab?  Well, many server admins would rather not give access to their users because almost any command can be issued from it.  Of course, almost any command can be issued in a Perl or PHP script, but we can save that for another article.  There is an alternative!  A free PHP script called Virtual Crontab.  This script is the next best thing to Crontab and runs almost exactly like it.   Using your web browser, you can schedule tasks to execute at any time of the hour, day, and month.

Many PHP scripts run certain processes on a schedule.  For example, the popular Vbulletin forum software has a section called “Schedules Tasks” where certain operations are run at certain times.  One task is to recount all of the posts, and all of the users.  Since this is a bit CPU intensive, it is only run about once per hour.  The problem is, it requires a user activate the script before the action can take place.  With Virtual Crontab, you can set up a task to call the URL to the script, in essence activating it.

If your PHP installation has not restricted access by turning safe mode on, you can even execute software and commands on the server.

For those of you who do not have access to Crontab, or are baffled by how it works, Virtual Crontab is the answer!