Friday, October 25, 2013

Jenkins, Maven error: Target repository cannot be empty

After applying the previous three fixes (see my other recent posts) to my Jenkins instance my Maven builds were still failing. I hope these posts can help others.

Issue:

mavenExecutionResult exceptions not empty
message : Internal error: java.lang.IllegalArgumentException: Target repository cannot be empty
cause : Target repository cannot be empty
Stack trace : 
org.apache.maven.InternalErrorException: Internal error: java.lang.IllegalArgumentException: Target repository cannot be empty
....snip...
at org.jfrog.build.client.DeployDetails$Builder.build(DeployDetails.java:115)


Discussion:
After asking Uncle Google, I only turned up the jfrog code that returns the "Target repository cannot be empty" message.  After a little more prodding of Uncle Google, Uncle finally revealed that the Jenkins Artifactory Plug-in uses jfrog  JENKINS Artifactory Plugin. Now I had somewhere to look.  I looked at my Jenkins instance and attempted to set the Artifactory target repository, but the Artifactory plugin appeared to be corrupt, as I wasn't getting any place to enter the target repository. Thus it appeared that the Artifactory plugin was probably corrupt. This was most likely due to my need to downgrade Jenkins to an older version.


Solution:  I first attempted to use the Manage Plugin feature to uninstall the Artifactory plugin but it didn't actually remove it.  Thus I forcibly removed Artifactory Jenkins plug-in, by removing JENKINS_HOME/plugins/artifactory directory and the artifactory.jpi file. I believe there are still some xml files I need to hack but I'm finally back to building my software.


git: Failed to connect to repository

My Jenkins instance was failing to connect to a git repository using a supplied username and private key. The Credentials plugin appeared to be getting confused. I hope the following can help others avoid a bit of head banging against your desk.

Issue: 

Failed to connect to repository : Command "ls-remote -h ssh://myuser@repo HEAD" returned status code 128:
stderr: Permission denied, please try again. 
Permission denied, please try again. 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 
fatal: The remote end hung up unexpectedly

Discussion: 

In my situation the Jenkins instances is pointing to several different repositories (svn and git)
The connection to the svn repository uses svn+ssh. The connection to the git repository ,using ssh, was failing. It appears that the Jenkins instance is having trouble figuring out which credentials to use or private key to use.


Solution: 
The solution is to define the private key to use for the user that is failing. I found the solution here at: stackoverflow  then modified for my situation.

Create a JENKINS_HOME/.ssh/config file to point to the correct private key.
Set permissions to 600 ( rw------- )
Set the contents of the file to:

Host source.server.com
HostName 123.456.789.012
User myuser
IdentityFile /home/jenkins/.ssh/id_rsa

Where: Host:            Can be anything is it just a place holder I generally use the DNS name 
                                 or short name of the server.
            HostName:    IP addresss or fully qualified domain name if you have DNS 
                                 enabled on your network
           User:             The user that will be connecting to the repo
  IdentityFile: The ssh private key that will be used to connect to the repository.
                                      In my case the user  and key are not the jenkins user and id_rsa. 

Finally: 

You must add the public key to your known_hosts file. The easiest way to add this is to execute: ssh myuser@source.server.com   
Then answer yes.

Alternatively if you have the public key you can directly edit your known_hosts file and paste the public key to the bottom of the file.

Jenkins Maven Jobs fail to Parse POMS

I had several recent Issues with Jenkins and Maven I hope this will help someone else out there with the same issue.

Issue: 

Jenkins Maven Jobs fail to Parse POMS
Caused by: java.lang.ClassNotFoundException: org.apache.maven.cli.MavenLoggerManager

Discussion: 

It appears that Maven 3.1.0, and 3.1.1 (I tried both) have not been compiled into Jenkins yet.
More can be found here:JENKINS-15935  as of today 10/25/2013 the fix version has not been set. So I'm guessing it will be fixed soon.

Solution: 

Roll back Maven to 3.0.5, This assumes that you installed maven yourself,(ie /usr/local/maven) rather than having Jenkins do it for you which I didn't try.

Jenkins Maven jobs do not log errors

I recently had a few issues with my Jenkins, Maven, Git environment failing builds without displaying any actual errors. To say the least this was very frustrating especially since the Maven builds ran just fine from the command line in the Jenkins work space.

Issue: Jenkins Maven jobs do not log errors.

Discussion:

 Apparently a bug was introduced in Jenkins with the release of 1.526 which caused Jenkins to not display errors generated by maven, Only displaying the build as a failure. This is discussed in the Jenkins Jira instance here:JENKINS=19352 

Solution: 

Roll back Jenkins to version 1.525

Monday, July 22, 2013

Simple Perl Time stamp function

##################
# timestamp: Simplified function returns a timestamp 
# calling profile: mytimestamp=&timestamp();
# returns: timestamp
##################
sub timestamp {
my ($flag,$message) = @_;
my $timestamp;
my $timedate;
my $date;
my $time;
my $sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst;
my $thisday, $thismon;
my $yy,$yyyy;


($sec,$min,$hour,$mday,$mon,$yy,$wday,$yday,$isdst) = localtime(time);
$thisday= (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[$wday]; 
$thismon= (Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec)[$mon];

#If the year month, day, hour, minute, or second, are less than 10 prepend with a 0

$yyyy = $yy + 1900; #add 1900 to get 4 digit year
$yy   -= 100;       #subtract 100 to get 2 digit year

if ($yy < 10 ) {
  $yy = "0$yy";
}

$mon++;   #add 1 to month as month array starts at 0
if ($mon < 10 ) {
  $mon = "0$mon";
}

# Day
if ($mday < 10 ) {
  $mday = "0$mday";
}

# Hour
if ($hour < 10 ) {
  $hour = "0$hour";
}

# Min
if ($min < 10 ) {
  $min = "0$min";
}

# Second
if ($sec < 10 ) {
  $sec = "0$sec";
}

# TO GET FORMAT                 USE THIS
#=================================================
# hh:mm:ss           $timestamp="$hour:$min:$sec";
# hhmmss             $timestamp="$hour$min$sec";
# dd/mm/yy           $timestamp="$mday/$mon/$yy";
# mm/dd/yy")         $timestamp="$mon/$mday/$yy";
# mm/dd/yyyy         $timestamp="$mon/$mday/$yyyy";
# yyyymmdd           $timestamp="$yyyy$mon$mday";
# yyyymmddhhmm       $timestamp="$yyyy$mon$mday$hour$min";
# yyyymmddhhmmss     $timestamp="$yyyy$mon$mday$hour$min$sec";
# yyyy-mm-dd         $timestamp="$yyyy-$mon-$mday";
# mmddyyyy           $timestamp="$mon$mday$yyyy";
# dd-mm-yy           $timestamp="$mday-$mon-$yy";
# dd-mm-yyyy         $timestamp="$mday-$mon-$yyyy";
# mm-dd-yy           $timestamp="$mon-$mday-$yy";
# mm-dd-yyyy         $timestamp="$mon-$mday-$yyyy";
# dd-mmm-yyyy        $timestamp="$mday-$thismon-$yyyy";
# mmm dd, yyyy       $timestamp="$thismon $mday, $yyyy";

#yyyymmddhhmm
$timestamp="$yyyy$mon$mday$hour$min";

return ($timestamp);
}

Monday, July 1, 2013

Apache Error [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

Problem: One day I woke up and my Apache server which had been running for several years was down and wouldn't restart. I checked the Apache error log and found the following error message:

[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

This particular error message caused Uncle Google to speak in riddles, many of the forum posts that were found led to non answers. After fumbling around a bit, I noticed that another Apache log " nss_error_log" was being updated and displayed the following error messages:

[error] Certificate not verified: 'Server-Cert'
[error] SSL Library Error: -8181 Certificate has expired
[error] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.


Now we've found the root cause of our issue it is in a module called nss.  I never noticed it before so another visit to Uncle Google actually turned up some interesting reading and several solutions. In short mod_nss is an alternative to mod_ssl it does a few things that mod_ssl doesn't do, and can run along side mod_ssl. 

Since I'm not now an expert on mod_nss, I'll leave the gory details of how to set it up and what it does to these articles [What is mod_nssOn Setting up Mod_NSS ,More detail ] but suffice it to say chances are if you didn't know you were using it you may not need it. If you are inheriting a system you probably want to dig deeper into why it is there. Below I'm listing four possible solutions. 

  1. Simply do as the error message above says add NSSEnforceValidCerts off to your nss.conf file which is usually located in /etc/httpd/conf.d.   The downside to this is that your nss_error_log will continue to get messages like "SSL Library Error: -8181 Certificate has expired"
  2. Remove nss.conf from /etc/httpd/conf.d this will of course cause mod_nss to not be loaded.
  3. Uninstall/re-install your nss rpm modules on re-installation a new certificate will be generated and your problem will go away for a few more years
  4. Properly setup your nss db or trust store using the certutil command. (you can Google for more details) 


Wednesday, May 29, 2013

Raspberry Pi Digital Picture Frame (Part 2)

 9/15/2013 I have posted new code to handle an issue with Iphones. When sending a picture from an IPhone the picture is always named Photo.jpg, this causes pictures to be overwritten. The solution was two fold, 1) add a time stamp to each file, 2) Check to see the file already exists then add an additional, index if two timestamps are the same.

Notes:
      Editing files: You can use either nano or vi if you know it.  I used vi, but will be referencing nano in this post.
       Sudo vs Root user: If you are comfortable working as the root (or super user) you can avoid typing sudo before every command by executing sudo bash

Gmail Setup:
If Mom doesn't have a Gmail account create one for her. Then follow these instructions for enabling IMAP.

Enable IMAP in your Gmail settings

  1. Sign in to Gmail.
  2. Click the gear icon  in the upper right, then select Settings.
  3. Click Forwarding and POP/IMAP.
  4. Select Enable IMAP.
  5. Click Save Change

Pi Setup:
After boot up you will be presented with a a menu to finish setup. The first thing you should do is expand the root partition to fill the SD card, set your keyboard layout, and your timezone. Be patient as this will take a couple of minutes.  You also have the option to have the GUI desktop enabled at startup, you can set this to NO as we won't need it.  However I did use the GUI to create the initial WI-FI connection, and installed most of the required software from a terminal window.  For the rest of this post I will assume you are at a command line.

Log In:
The default username is pi and the default password is raspberry. 

Network Setup:
Before you can install any software you need to be connected to the internet. If you are directly connected to your router with an ethernet cable execute the following command to see if you've been assigned an IP address.

sudo ifconfig

There are two groups of output one for eth0 the other for wlan0. For a cable connected Pi you should see an ipaddress beginning with 10 or 198 in the eth0 block.  If you purchased a Wi-Fi dongle and don't see an ipaddress, follow these instructions.

1. edit the file /etc/network/interfaces 
sudo nano /etc/network/interfaces

2. Delete everything in the file, then copy and paste the below text in instead. Then modify the last two lines to reflect your actual network name "ssid" and password (keeping the double quotes on both):
3. Save the file using CTRL-X to exit, Y to save the file and hit the Carriage Return to verify the filename.
4. Restart wlan0 by executing;   sudo ifup wlan0
5. Verify you're router has given you an ipaddress by executing; sudo ifconfig
6. Verify internet connection by executing; sudo ping www.google.com

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0

iface wlan0 inet dhcp
    wpa-ssid "ssid"
    wpa-psk "password"

Notes: Some additional commands to help you debug.
    Shutdown wireless connection: ifdown wlan0
    List available networks: iwlist wlan0 scan

Install some software:
Hopefully everything worked and you are now connected to the web.
  • Install any Raspberry Pi upgrades:  sudo apt-get update
  • Install Linux Frame Buffer image viewer: sudo apt-get install fbi
  • Install perl ssleay libraries: sudo apt-get install libnet-ssleay-perl

Install Perl Modules:
The Perl script that logs into your Mom's email account needs to call a few functions that are not already installed on your Raspberry Pi. Thus we need to install what are known as Perl modules. The tool we use is called cpan, the good folks who built the wheezy installation have included the cpan command in the installation. Once started the cpan tool will search the known online Perl module libraries for the modules and any dependencies we need and install them for us.  To install the needed perl modules do the following.  A couple of notes before you begin..


  1. CPAN generates lots of output, you can ignore most of it.
  2. Linux and Perl are case sensitive it is important to enter your install commands exactly as listed below. If you mis-spell the module name cpan will not be able to find it.
  3. Answer "Yes" or "Y" when prompted to install any dependencies.
  4. Answer "NO" or "N" when asked if you want to run additional tests.


sudo cpan
cpan[1]> install MIME::Parser
....
cpan[2]> install Mail::IMAPClient
....
cpan[3]> install IO::Socket::SSL
....
cpan[4]> exit

Install the scripts
   We are going to create two shell scripts and one Perl script. The shell scripts are short and can be typed in. The Perl script is rather large I list it at the bottom of this post or you can download it from the link below

Create the directory to hold your scripts and pictures.
sudo mkdir /usr/local/getpics
sudo chown pi:pi /usr/local/getpics

Create the scripts:
 The first script is called downloadpic.sh, the second slideshow.sh the third picturedownloader.pl. Use nano to create your scripts. Remember to quit nano use CTRL-X, answer Y and then hit Carriage Return to confirm.

To download each script you can use the wget command as follows.
wget https://sites.google.com/site/relengetc/downloadpics.sh
wget https://sites.google.com/site/relengetc/slideshow.sh
wget https://sites.google.com/site/relengetc/picturedownloader.pl

Or enter them manually
cd /usr/local/getpics
nano downloadpics.sh

#!/bin/sh
cd /usr/local/getpics
/usr/local/getpics/picturedownloader.pl
sudo reboot

nano slideshow.sh
#!/bin/sh
cd /usr/local/getpics/pictures
/usr/bin/fbi -noverbose -t 10 -n 800x600 -a *

Add execution permissions to your scripts
chmod 755 downloadpics.sh slideshow.sh picturedownloader.pl

Finishing Up
The Raspberry Pi will prompt you for your username and password every time it boots up. For everything to work we need the "pi" user to be automatically logged in at boot time.  We need to modify one of the system startup scripts.

sudo nano /etc/inittab

Find the line that looks like this and comment it out by placing a # in front of it.
1:2345:respawn:/sbin/getty --noclear 38400 tty1
It should look like this
#1:2345:respawn:/sbin/getty --noclear 38400 tty1

Now insert the line below

1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

Save your changes (CTRL-X, Y, CR)

Next we need to enable the linux command scheduler called cron and start our slide show on startup
We need to modify another startup file

sudo nano /etc/rc.local

Scroll to the bottom then insert the following two lines.

/etc/init.d/cron/start
bash /usr/local/getpics/slideshow.sh

Finally we need to configure automatic downloads and a reboot by setting the crontab for the pi user. To do this we run the crontab -e command, which will use nano to open a file. Scroll to the bottom and add the line below. This will schedule your system to check for and download pictures if any at 8:30 am, 12:30pm and 8:30pm. Save as before (Ctrl-X, Y, CR)

crontab -e
30 8,12,20 * * * /usr/local/getpics/downloadpics.sh 2&>1 /dev/null



Picture Downloader Script
The only change you will need to make to this script is to set the proper username and password on lines 6 and 7.

#!/usr/bin/perl
#Perl Modules needed
use MIME::Parser;
use Mail::IMAPClient;

#Connection information
my $username = 'username';               #your gmail username
my $password = 'password';              #your gmail password
my $mailhost = 'imap.gmail.com';         #Only change if not using gmail
my $debug = 0;                           #Set to 0 to turn off debugging

#Environment setup
# Directory where you want your pictures
my $outputdir = "./pictures";

# Directory where the message text and header information is placed
# Note: The contents of messagedir will be removed after picture download
# There may be a way to keep the message text ,header and extra copy of the imagei
# from being downloaded  but I haven't figured it out.

my $messagedir = "./tmp";

#Setup some variables that we will need later
my (@body, $i, $subentity);
my ($x, $newx, @attachment, $attachment, @attname, $bh, $nooatt);
my $image="";
my $from="";
my $subject="";
my $timestamp="";

#These are the types of attachments allowed
#List of mime types found here: http://www.freeformatter.com/mime-types-list.html
#ex: If you want to add pdfs you would add application/pdf
my @attypes= qw(image/bmp
                image/gif
                image/jpeg
                image/png
                image/tiff
);

#Create ouput directories if they don't exist.
#Not the most efficient piece of code but effective and readable
if (! -d $outputdir ) {
  if (!mkdir ("$outputdir", 0755)) {
    die "Unable to create directory $outputdir\n";
  }
}
if (! -d $messagedir ) {
  if (!mkdir ("$messagedir", 0755)) {
    die "Unable to create directory $messagedir\n";
  }
}



#Connect to your mail host
my $imap = Mail::IMAPClient->new(
    Debug => $debug,
    User => $username,
    Password => $password,
    Server => $mailhost,
    port    => 993,
    Uid  => 1,
    ssl => 1
) || die "Unable to connect to $mailhost \n";


my $newm=0;
   $newm = $imap->select('INBOX');

#need to verify if this code actually works
if ($newm==0) {
  $imap->logout();
  print "No New Messages.\n";
  exit;
  }

my $parser = new MIME::Parser;
#my $error = ($@ || $parser->last_error);

$parser->output_dir($messagedir);

#Search SUBJECT lines for pattern "PIC"
#my @messages = $imap->search('SUBJECT',"PIC");
#Search for NOT SEEN emails
my @messages = $imap->search('NOT',"SEEN");


#print "Number of Messages: $#messages \n";

#Search thru each New message
foreach my $id (@messages) {
  print "message id: $id\n" if ($debug);

  my $entity = $parser->parse_data($imap->message_string($id));

  #Lets see if we are able to parse anything
  print "====================================================== \n" if ($debug);

  chomp( $from = $entity->head->get('FROM') );
  chomp( $subject = $entity->head->get('SUBJECT') );
  chomp( $timestamp = $entity->head->get('Date') );

  print " FROM: $from \n SUBJECT: $subject \n DATE: $timestamp \n" if ($debug);
  print "====================================================== \n\n\n\n" if ($debug);

#Get email body borrowed most of the code
#from here: http://www.perlmonks.org/index.pl?node_id=195442

if ($entity->parts > 0){
    for ($i=0; $i<$entity->parts; $i++){

        $subentity = $entity->parts($i);

        if (($subentity->mime_type =~ m/text\/html/i) || ($subentity-> mime_type =~ m/text\/plain/i)){
            $body = join "",  @{$subentity->body};
            next;
        }

        #this elsif is needed for Outlook's nasty multipart/alternative messages
        elsif ($subentity->mime_type =~ m/multipart\/alternative/i){

            $body = join "",  @{$subentity->body};

            #split html and text parts
            @body = split /------=_NextPart_\S*\n/, $body;

            #assign the first part of the message,
            #hopefully the text, part as the body
            $body = $body[1];

            #remove leading headers from body
            $body =~ s/^Content-Type.*Content-Transfer-Encoding.*?\n+//is;
            next;
        }

        #new attachment code start
        #grab attachment name and contents
        foreach $x (@attypes){
            if ($subentity->mime_type =~ m/$x/i){
                $bh = $subentity->bodyhandle;
                $attachment = $bh->as_string;
                push @attachment, $attachment;
                push @attname, $subentity->head->mime_attr('content-disposition.filename');
            }else{
                #some clients send attachments as application/x-type.
                #checks for that
                $newx = $x;
                $newx =~ s/application\/(.*)/application\/x-$1/i;
                if ($subentity->mime_type =~ m/$newx/i){
                    $bh = $subentity->bodyhandle;
                    $attachment = $bh->as_string;
                    push @attachment, $attachment;
                    push @attname, $subentity->head->mime_attr('content-disposition.filename');
                }
            }

        }
       $nooatt = $#attachment + 1;
       #new attachment code end
    }
} else {
   $body = join "",  @{$entity->body};
}

#body may contain html tags. they will be stripped here
$body =~ s/(<br>)|(<p>)/\n/gi;           #create new lines
$body =~ s/<.+\n*.*?>//g;                #remove all <> html tages
$body =~ s/(\n|\r|(\n\r)|(\r\n)){3,}//g; #remove any extra new lines
$body =~ s/\&nbsp;//g;                   #remove html &nbsp characters

#remove trailing whitespace from body
$body =~ s/\s*\n+$//s;

if ( $debug ) {
print "Messege was contructed as follows:
\$from:    $from
\$to:      $to
\$subject: $subject

\$body:    $body
number of attachments: $nooatt
\$attachment(s): ".join ", ", @attname;

}
}
$imap->logout();

#write contents of each attachment to a file
  for ($x = 0; $x < $nooatt; $x++){
      $image = $attname[$x];

      print "\n $x attachmentname:$image \n" if ($debug);

      #strip one or more spaces in the image name and replace with _
      $image =~ s/\s+/_/g;
      $timestamp=&timestamp();
      $image="$timestamp"."_"."$image";

      if ( -e "$outputdir/$image" ) {
         $image="$x" . "_" . "$image";
         print "\n hello: $image \n" if ($debug);
      }

      print "\n $x attachmentname:$image \n" if ($debug);

      open FH, ">$outputdir/$image" || die "cannot open FH:$!\n";
      print FH "$attachment[$x]";
      close FH;
  }


#cleanup
if (!opendir (TMPDIR, $messagedir)) {
    die "unable to open $messagedir\n";
 }
 else {
    @cleanup_list= grep !/^\.\.?$/, readdir(TMPDIR);
 }
 closedir(TMPDIR);

 if ( defined @cleanup_list) {
   foreach my $filename (@cleanup_list) {
      if ( !unlink("$messagedir/$filename") ) {
        warn("Unable to remove $messagedir/$filename\n");
      }
   }
 }


##################
# timestamp: Simplified function returns a timestamp
# calling profile: mytimestamp=&timestamp();
# returns: timestamp
##################
sub timestamp {
my ($flag,$message) = @_;
my $timestamp;
my $timedate;
my $date;
my $time;
my $sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst;
my $thisday, $thismon;
my $yy,$yyyy;


($sec,$min,$hour,$mday,$mon,$yy,$wday,$yday,$isdst) = localtime(time);
$thisday= (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[$wday];
$thismon= (Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec)[$mon];

#If the year month, day, hour, minute, or second, are less than 10 prepend with a 0

$yyyy = $yy + 1900; #add 1900 to get 4 digit year
$yy   -= 100;       #subtract 100 to get 2 digit year

if ($yy < 10 ) {
  $yy = "0$yy";
}

$mon++;   #add 1 to month as month array starts at 0
if ($mon < 10 ) {
  $mon = "0$mon";
}

# Day
if ($mday < 10 ) {
  $mday = "0$mday";
}

# Hour
if ($hour < 10 ) {
  $hour = "0$hour";
}

# Min
if ($min < 10 ) {
  $min = "0$min";
}

# Second
if ($sec < 10 ) {
  $sec = "0$sec";
}

# TO GET FORMAT                 USE THIS
#=================================================
# hh:mm:ss           $timestamp="$hour:$min:$sec";
# hhmmss             $timestamp="$hour$min$sec";
# dd/mm/yy           $timestamp="$mday/$mon/$yy";
# mm/dd/yy")         $timestamp="$mon/$mday/$yy";
# mm/dd/yyyy         $timestamp="$mon/$mday/$yyyy";
# yyyymmdd           $timestamp="$yyyy$mon$mday";
# yyyymmddhhmm       $timestamp="$yyyy$mon$mday$hour$min";
# yyyymmddhhmmss     $timestamp="$yyyy$mon$mday$hour$min$sec";
# yyyy-mm-dd         $timestamp="$yyyy-$mon-$mday";
# mmddyyyy           $timestamp="$mon$mday$yyyy";
# dd-mm-yy           $timestamp="$mday-$mon-$yy";
# dd-mm-yyyy         $timestamp="$mday-$mon-$yyyy";
# mm-dd-yy           $timestamp="$mon-$mday-$yy";
# mm-dd-yyyy         $timestamp="$mon-$mday-$yyyy";
# dd-mmm-yyyy        $timestamp="$mday-$thismon-$yyyy";
# mmm dd, yyyy       $timestamp="$thismon $mday, $yyyy";

#yyyymmddhhmm
$timestamp="$yyyy$mon$mday$hour$min$sec";
print "\n TIMESTAMP: $timestamp \n";
return ($timestamp);
}

Tuesday, May 28, 2013

Raspberry Pi Digital Picture Frame (Part 1)

A few years ago I purchased a Kodak Pulse email addressable picture frame for my mother.  Though a little expensive at $200 I thought this was a great product. It allowed my siblings and I to send pictures to Mom from our cellphones and other devices. It was fun for Mom as well as she could see pictures of the grandkids almost immediately after an event happened.  Unfortunately the picture frame stopped connecting to my mothers WI-FI network and I was unable to fix it.  I didn't want to spend another $200 thus I began looking at alternatives, old desktops/laptops/tablets and perhaps using flikr, or some other photo sharing service. I didn't want to be tied down to yet another online service.  As I was researching alternatives the Raspberry PI B was announced, this was the perfect solution, small footprint, linux based and cheap, and I probably had stuff laying about the house.

Before I begin describing the build I'd like to give a shout out to Cameron Wiebe for his excellent post on building a Raspberry Pi picture frame some of the instructions below come directly from his article.

What I Built

A digital picture frame that automatically checks a gmail account for new messages then downloads any attached images and automatically displays them. I configured the gmail account for IMAP, and used a Perl script to query the account.

What I Used

Things I bought:
Raspberry PI B (512 MB): $39.99
NetGear USB wireless card. $25.00.
USB keyboard: $7
USB mouse: $3

Things I had laying around:
4GB micro SD card with a bunch of adapters. See pic below.
7" Polaroid monitor, left over from a two screen auto DVD player
Video cable
5V 700mA wall wart left over from an old Nokia cell phone
12 V wall wart to power the monitor borrowed from a USB switch.
4 port unpowered usb hub

Setting up the Raspberry Pi
I started with the instructions here , then went here to download Raspberry Pi Wheezy. I have a MacBook so I followed the excellent instructions here to copy the image to the SD card. Be patient the transfer takes a few minutes. Please read the instructions carefully. If you do it wrong you could trash your system.

Now plug everything in and power up. I was pleasantly surprised everything started up without any issues.  Read Part 2 for complete details to get your Digital Picture Frame up and running.















Tuesday, May 7, 2013

Makespec.py: error: Requires at least one scriptname file

Recently I inherited a set of python scripts with corresponding windows executable files. To create the executable (.exe) file one can used the tool pyinstaller. The scripts were maintained by multiple people using a shared network drive. A development team on the other side of the world needed access to the network share.  To give them access I would have had to work with the corporate network team, which isn't always easy. However, this team has access to our centralized SCM system. As a Release Engineer and Subversion admin, I felt that these scripts should be put into source control. I proceeded to place the scripts in Subversion without any issues. I then needed to create a Jenkins job to execute the pyinstaller scripts. I spent nearly a day hacking the build script, and using uncle Google searching for the error in the header. After hitting my head repeatedly against the wall I discovered the simplest of solutions, and probably the most obvious.

I simply changed this:
C:\apps\pyinstaller\Makespec.py --onefile --out=. --noconsole --name=cl_downloader downloader.py 


Downloader.py
usage: python Makespec.py [opts] <scriptname> [<scriptname> ...]
Makespec.py: error: Requires at least one scriptname file


To this:
python C:\apps\pyinstaller\Makespec.py --onefile --out=. --noconsole --name=cl_downloader downloader.py 

I'm hoping that this simple solution will keep someone else from getting a headache.


Monday, February 25, 2013

MySQL Replication


Configuring Master/Master replication

The following procedure can be used to setup MySQL in a master/master replication configuration.   We'll call the two MySQL servers Master1 and Master2. In a dual master setup each server functions as both a master and a slave to the other server. The following assumptions are being made:
·         You have two MySQL servers running the same MySQL version.
·         Neither server is currently active (i.e. they are not currently accepting inserts or deletions or modifications of any type.)
·         Master1 refers to the first server that has data that you wish to replicate.
·         Master2 refers to the second server.
·         Master/Slave or Master/Master replication has not already been implemented.

NOTE: Change IP addreses and Server-ID’s in the following steps to match your environment.

1.    Configure Master1 by adding the following to /etc/my.cnf
#Replication Setup
#set server id to the last octet of the servers ip address
server-id = 150
replicate-same-server-id = 0
auto-increment-increment = 2
auto-increment-offset = 1

# change this to a path/name appropriate to your system
log-bin       = /usr/local/mysql/data/mysql-bin
log-bin-index = /usr/local/mysql/data/mysqld-bin.index
relay-log     = /usr/local/mysql/data/mysqld-relay-bin
relay-log-index = /usr/local/mysql/data/mysqld-relay-bin.index
binlog-format        = MIXED
binlog-cache-size    = 1M
expire-logs-days     = 7
sync-binlog          = 1

skip-slave-start
log-slave-updates

2.    Configure Master2 by adding the following to /etc/my.cnf.
#Replication Setup
#set server id to the last octet of the servers ip address
server-id = 148
replicate-same-server-id = 0
auto-increment-increment = 2
auto-increment-offset = 2

# change this to a path/name appropriate to your system
log-bin       = /usr/local/mysql/data/mysql-bin
log-bin-index = /usr/local/mysql/data/mysqld-bin.index
relay-log     = /usr/local/mysql/data/mysqld-relay-bin
relay-log-index = /usr/local/mysql/data/mysqld-relay-bin.index
binlog-format        = MIXED
binlog-cache-size    = 1M
expire-logs-days     = 7
sync-binlog          = 1

skip-slave-start
log-slave-updates


3.    Start MySQL on both servers.
Shell> cd /usr/local/mysql
Shell> ./bin/mysqld_safe --user=mysql &

4.  Backup the Database on Master 1
        Shell> mysqldump -u root -ppassword –-log-error=dump.log --all-databases -–master_data > master1_snapshot_master_data.sql


5.    Capture the master_log_file name, and position from the newly created backup

   Shell>grep MASTER_LOG_FILE master1_snapshot_master_data.sql | head -1

           CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=390;

6.    Install the Database on Master2
Shell> mysql –uroot –ppassword < master1_snapshot_master_data.sql

7.    Create slave user on Master 1:
      Adjust the IP address to reflect your systems.

mysql> use mysql;
mysql> CREATE USER 'slave_user'@'10.48.92.%' IDENTIFIED BY 'password';
mysql> GRANT REPLICATION SLAVE ON *.* to 'slave_user'@'10.48.92.%' identified by 'password';                         
mysql> FLUSH PRIVILEGES;


8.    Create slave user on Master 2:
      Adjust the IP address to reflect your systems.
mysql> use mysql;
mysql> CREATE USER 'slave_user'@'10.48.92.%' IDENTIFIED BY 'password';
mysql> GRANT REPLICATION SLAVE ON *.* to 'slave_user'@'10.48.92.%' identified by 'password';                 
mysql> FLUSH PRIVILEGES;


9.    Synchronize the Servers,

Master1:
mysql> STOP SLAVE;


Master2:  Set Master 2 to be a slave of Master 1
Set the value of “MASTER_LOG_FILE “ and “MASTER_LOG_POS”  with the values discovered in step 5 above.

mysql> STOP SLAVE;
mysql> SHOW MASTER STATUS\G;
mysql> CHANGE MASTER TO MASTER_HOST=’<MASTER1-IP>’, MASTER_USER=’slave_user’, MASTER_PASSWORD=’password’, MASTER_LOG_FILE=’mysqld-bin.XXXXXX’, MASTER_LOG_POS=NNNN;

Master1:  Set Master 1 to be a slave of Master 2
We don’t need the Position: value because NOPS only writes to MASTER 1.
mysql> CHANGE MASTER TO MASTER_HOST =’<MASTER2-IP>’, MASTER_USER=’slave_user’, MASTER_PASSWORD=’password’;

        Master2:
mysql> START SLAVE;

               Master1:
mysql> START SLAVE;
       
10.  Verify Replication is running.
Execute the following on both Master1 and Master2.  For “SHOW SLAVE STATUS\G;” look for  “Seconds_Behind_Master: 0”.  For “SHOW PROCESSLIST;” look for “Master has sent all binlog to slave; waiting for binlog to be updated”

mysql> SHOW SLAVE STATUS\G;
…snip…
Seconds_Behind_Master: 0
…snip…

mysql> SHOW PROCESSLIST;
…snip…
Master has sent all binlog to slave; waiting for binlog to be updated
Slave has read all relay log; waiting for the slave I/O thread to update it
…snip…