Wednesday, May 29, 2013

Raspberry Pi Digital Picture Frame (Part 2)

 9/15/2013 I have posted new code to handle an issue with Iphones. When sending a picture from an IPhone the picture is always named Photo.jpg, this causes pictures to be overwritten. The solution was two fold, 1) add a time stamp to each file, 2) Check to see the file already exists then add an additional, index if two timestamps are the same.

Notes:
      Editing files: You can use either nano or vi if you know it.  I used vi, but will be referencing nano in this post.
       Sudo vs Root user: If you are comfortable working as the root (or super user) you can avoid typing sudo before every command by executing sudo bash

Gmail Setup:
If Mom doesn't have a Gmail account create one for her. Then follow these instructions for enabling IMAP.

Enable IMAP in your Gmail settings

  1. Sign in to Gmail.
  2. Click the gear icon  in the upper right, then select Settings.
  3. Click Forwarding and POP/IMAP.
  4. Select Enable IMAP.
  5. Click Save Change

Pi Setup:
After boot up you will be presented with a a menu to finish setup. The first thing you should do is expand the root partition to fill the SD card, set your keyboard layout, and your timezone. Be patient as this will take a couple of minutes.  You also have the option to have the GUI desktop enabled at startup, you can set this to NO as we won't need it.  However I did use the GUI to create the initial WI-FI connection, and installed most of the required software from a terminal window.  For the rest of this post I will assume you are at a command line.

Log In:
The default username is pi and the default password is raspberry. 

Network Setup:
Before you can install any software you need to be connected to the internet. If you are directly connected to your router with an ethernet cable execute the following command to see if you've been assigned an IP address.

sudo ifconfig

There are two groups of output one for eth0 the other for wlan0. For a cable connected Pi you should see an ipaddress beginning with 10 or 198 in the eth0 block.  If you purchased a Wi-Fi dongle and don't see an ipaddress, follow these instructions.

1. edit the file /etc/network/interfaces 
sudo nano /etc/network/interfaces

2. Delete everything in the file, then copy and paste the below text in instead. Then modify the last two lines to reflect your actual network name "ssid" and password (keeping the double quotes on both):
3. Save the file using CTRL-X to exit, Y to save the file and hit the Carriage Return to verify the filename.
4. Restart wlan0 by executing;   sudo ifup wlan0
5. Verify you're router has given you an ipaddress by executing; sudo ifconfig
6. Verify internet connection by executing; sudo ping www.google.com

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0

iface wlan0 inet dhcp
    wpa-ssid "ssid"
    wpa-psk "password"

Notes: Some additional commands to help you debug.
    Shutdown wireless connection: ifdown wlan0
    List available networks: iwlist wlan0 scan

Install some software:
Hopefully everything worked and you are now connected to the web.
  • Install any Raspberry Pi upgrades:  sudo apt-get update
  • Install Linux Frame Buffer image viewer: sudo apt-get install fbi
  • Install perl ssleay libraries: sudo apt-get install libnet-ssleay-perl

Install Perl Modules:
The Perl script that logs into your Mom's email account needs to call a few functions that are not already installed on your Raspberry Pi. Thus we need to install what are known as Perl modules. The tool we use is called cpan, the good folks who built the wheezy installation have included the cpan command in the installation. Once started the cpan tool will search the known online Perl module libraries for the modules and any dependencies we need and install them for us.  To install the needed perl modules do the following.  A couple of notes before you begin..


  1. CPAN generates lots of output, you can ignore most of it.
  2. Linux and Perl are case sensitive it is important to enter your install commands exactly as listed below. If you mis-spell the module name cpan will not be able to find it.
  3. Answer "Yes" or "Y" when prompted to install any dependencies.
  4. Answer "NO" or "N" when asked if you want to run additional tests.


sudo cpan
cpan[1]> install MIME::Parser
....
cpan[2]> install Mail::IMAPClient
....
cpan[3]> install IO::Socket::SSL
....
cpan[4]> exit

Install the scripts
   We are going to create two shell scripts and one Perl script. The shell scripts are short and can be typed in. The Perl script is rather large I list it at the bottom of this post or you can download it from the link below

Create the directory to hold your scripts and pictures.
sudo mkdir /usr/local/getpics
sudo chown pi:pi /usr/local/getpics

Create the scripts:
 The first script is called downloadpic.sh, the second slideshow.sh the third picturedownloader.pl. Use nano to create your scripts. Remember to quit nano use CTRL-X, answer Y and then hit Carriage Return to confirm.

To download each script you can use the wget command as follows.
wget https://sites.google.com/site/relengetc/downloadpics.sh
wget https://sites.google.com/site/relengetc/slideshow.sh
wget https://sites.google.com/site/relengetc/picturedownloader.pl

Or enter them manually
cd /usr/local/getpics
nano downloadpics.sh

#!/bin/sh
cd /usr/local/getpics
/usr/local/getpics/picturedownloader.pl
sudo reboot

nano slideshow.sh
#!/bin/sh
cd /usr/local/getpics/pictures
/usr/bin/fbi -noverbose -t 10 -n 800x600 -a *

Add execution permissions to your scripts
chmod 755 downloadpics.sh slideshow.sh picturedownloader.pl

Finishing Up
The Raspberry Pi will prompt you for your username and password every time it boots up. For everything to work we need the "pi" user to be automatically logged in at boot time.  We need to modify one of the system startup scripts.

sudo nano /etc/inittab

Find the line that looks like this and comment it out by placing a # in front of it.
1:2345:respawn:/sbin/getty --noclear 38400 tty1
It should look like this
#1:2345:respawn:/sbin/getty --noclear 38400 tty1

Now insert the line below

1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

Save your changes (CTRL-X, Y, CR)

Next we need to enable the linux command scheduler called cron and start our slide show on startup
We need to modify another startup file

sudo nano /etc/rc.local

Scroll to the bottom then insert the following two lines.

/etc/init.d/cron/start
bash /usr/local/getpics/slideshow.sh

Finally we need to configure automatic downloads and a reboot by setting the crontab for the pi user. To do this we run the crontab -e command, which will use nano to open a file. Scroll to the bottom and add the line below. This will schedule your system to check for and download pictures if any at 8:30 am, 12:30pm and 8:30pm. Save as before (Ctrl-X, Y, CR)

crontab -e
30 8,12,20 * * * /usr/local/getpics/downloadpics.sh 2&>1 /dev/null



Picture Downloader Script
The only change you will need to make to this script is to set the proper username and password on lines 6 and 7.

#!/usr/bin/perl
#Perl Modules needed
use MIME::Parser;
use Mail::IMAPClient;

#Connection information
my $username = 'username';               #your gmail username
my $password = 'password';              #your gmail password
my $mailhost = 'imap.gmail.com';         #Only change if not using gmail
my $debug = 0;                           #Set to 0 to turn off debugging

#Environment setup
# Directory where you want your pictures
my $outputdir = "./pictures";

# Directory where the message text and header information is placed
# Note: The contents of messagedir will be removed after picture download
# There may be a way to keep the message text ,header and extra copy of the imagei
# from being downloaded  but I haven't figured it out.

my $messagedir = "./tmp";

#Setup some variables that we will need later
my (@body, $i, $subentity);
my ($x, $newx, @attachment, $attachment, @attname, $bh, $nooatt);
my $image="";
my $from="";
my $subject="";
my $timestamp="";

#These are the types of attachments allowed
#List of mime types found here: http://www.freeformatter.com/mime-types-list.html
#ex: If you want to add pdfs you would add application/pdf
my @attypes= qw(image/bmp
                image/gif
                image/jpeg
                image/png
                image/tiff
);

#Create ouput directories if they don't exist.
#Not the most efficient piece of code but effective and readable
if (! -d $outputdir ) {
  if (!mkdir ("$outputdir", 0755)) {
    die "Unable to create directory $outputdir\n";
  }
}
if (! -d $messagedir ) {
  if (!mkdir ("$messagedir", 0755)) {
    die "Unable to create directory $messagedir\n";
  }
}



#Connect to your mail host
my $imap = Mail::IMAPClient->new(
    Debug => $debug,
    User => $username,
    Password => $password,
    Server => $mailhost,
    port    => 993,
    Uid  => 1,
    ssl => 1
) || die "Unable to connect to $mailhost \n";


my $newm=0;
   $newm = $imap->select('INBOX');

#need to verify if this code actually works
if ($newm==0) {
  $imap->logout();
  print "No New Messages.\n";
  exit;
  }

my $parser = new MIME::Parser;
#my $error = ($@ || $parser->last_error);

$parser->output_dir($messagedir);

#Search SUBJECT lines for pattern "PIC"
#my @messages = $imap->search('SUBJECT',"PIC");
#Search for NOT SEEN emails
my @messages = $imap->search('NOT',"SEEN");


#print "Number of Messages: $#messages \n";

#Search thru each New message
foreach my $id (@messages) {
  print "message id: $id\n" if ($debug);

  my $entity = $parser->parse_data($imap->message_string($id));

  #Lets see if we are able to parse anything
  print "====================================================== \n" if ($debug);

  chomp( $from = $entity->head->get('FROM') );
  chomp( $subject = $entity->head->get('SUBJECT') );
  chomp( $timestamp = $entity->head->get('Date') );

  print " FROM: $from \n SUBJECT: $subject \n DATE: $timestamp \n" if ($debug);
  print "====================================================== \n\n\n\n" if ($debug);

#Get email body borrowed most of the code
#from here: http://www.perlmonks.org/index.pl?node_id=195442

if ($entity->parts > 0){
    for ($i=0; $i<$entity->parts; $i++){

        $subentity = $entity->parts($i);

        if (($subentity->mime_type =~ m/text\/html/i) || ($subentity-> mime_type =~ m/text\/plain/i)){
            $body = join "",  @{$subentity->body};
            next;
        }

        #this elsif is needed for Outlook's nasty multipart/alternative messages
        elsif ($subentity->mime_type =~ m/multipart\/alternative/i){

            $body = join "",  @{$subentity->body};

            #split html and text parts
            @body = split /------=_NextPart_\S*\n/, $body;

            #assign the first part of the message,
            #hopefully the text, part as the body
            $body = $body[1];

            #remove leading headers from body
            $body =~ s/^Content-Type.*Content-Transfer-Encoding.*?\n+//is;
            next;
        }

        #new attachment code start
        #grab attachment name and contents
        foreach $x (@attypes){
            if ($subentity->mime_type =~ m/$x/i){
                $bh = $subentity->bodyhandle;
                $attachment = $bh->as_string;
                push @attachment, $attachment;
                push @attname, $subentity->head->mime_attr('content-disposition.filename');
            }else{
                #some clients send attachments as application/x-type.
                #checks for that
                $newx = $x;
                $newx =~ s/application\/(.*)/application\/x-$1/i;
                if ($subentity->mime_type =~ m/$newx/i){
                    $bh = $subentity->bodyhandle;
                    $attachment = $bh->as_string;
                    push @attachment, $attachment;
                    push @attname, $subentity->head->mime_attr('content-disposition.filename');
                }
            }

        }
       $nooatt = $#attachment + 1;
       #new attachment code end
    }
} else {
   $body = join "",  @{$entity->body};
}

#body may contain html tags. they will be stripped here
$body =~ s/(<br>)|(<p>)/\n/gi;           #create new lines
$body =~ s/<.+\n*.*?>//g;                #remove all <> html tages
$body =~ s/(\n|\r|(\n\r)|(\r\n)){3,}//g; #remove any extra new lines
$body =~ s/\&nbsp;//g;                   #remove html &nbsp characters

#remove trailing whitespace from body
$body =~ s/\s*\n+$//s;

if ( $debug ) {
print "Messege was contructed as follows:
\$from:    $from
\$to:      $to
\$subject: $subject

\$body:    $body
number of attachments: $nooatt
\$attachment(s): ".join ", ", @attname;

}
}
$imap->logout();

#write contents of each attachment to a file
  for ($x = 0; $x < $nooatt; $x++){
      $image = $attname[$x];

      print "\n $x attachmentname:$image \n" if ($debug);

      #strip one or more spaces in the image name and replace with _
      $image =~ s/\s+/_/g;
      $timestamp=&timestamp();
      $image="$timestamp"."_"."$image";

      if ( -e "$outputdir/$image" ) {
         $image="$x" . "_" . "$image";
         print "\n hello: $image \n" if ($debug);
      }

      print "\n $x attachmentname:$image \n" if ($debug);

      open FH, ">$outputdir/$image" || die "cannot open FH:$!\n";
      print FH "$attachment[$x]";
      close FH;
  }


#cleanup
if (!opendir (TMPDIR, $messagedir)) {
    die "unable to open $messagedir\n";
 }
 else {
    @cleanup_list= grep !/^\.\.?$/, readdir(TMPDIR);
 }
 closedir(TMPDIR);

 if ( defined @cleanup_list) {
   foreach my $filename (@cleanup_list) {
      if ( !unlink("$messagedir/$filename") ) {
        warn("Unable to remove $messagedir/$filename\n");
      }
   }
 }


##################
# timestamp: Simplified function returns a timestamp
# calling profile: mytimestamp=&timestamp();
# returns: timestamp
##################
sub timestamp {
my ($flag,$message) = @_;
my $timestamp;
my $timedate;
my $date;
my $time;
my $sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst;
my $thisday, $thismon;
my $yy,$yyyy;


($sec,$min,$hour,$mday,$mon,$yy,$wday,$yday,$isdst) = localtime(time);
$thisday= (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[$wday];
$thismon= (Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec)[$mon];

#If the year month, day, hour, minute, or second, are less than 10 prepend with a 0

$yyyy = $yy + 1900; #add 1900 to get 4 digit year
$yy   -= 100;       #subtract 100 to get 2 digit year

if ($yy < 10 ) {
  $yy = "0$yy";
}

$mon++;   #add 1 to month as month array starts at 0
if ($mon < 10 ) {
  $mon = "0$mon";
}

# Day
if ($mday < 10 ) {
  $mday = "0$mday";
}

# Hour
if ($hour < 10 ) {
  $hour = "0$hour";
}

# Min
if ($min < 10 ) {
  $min = "0$min";
}

# Second
if ($sec < 10 ) {
  $sec = "0$sec";
}

# TO GET FORMAT                 USE THIS
#=================================================
# hh:mm:ss           $timestamp="$hour:$min:$sec";
# hhmmss             $timestamp="$hour$min$sec";
# dd/mm/yy           $timestamp="$mday/$mon/$yy";
# mm/dd/yy")         $timestamp="$mon/$mday/$yy";
# mm/dd/yyyy         $timestamp="$mon/$mday/$yyyy";
# yyyymmdd           $timestamp="$yyyy$mon$mday";
# yyyymmddhhmm       $timestamp="$yyyy$mon$mday$hour$min";
# yyyymmddhhmmss     $timestamp="$yyyy$mon$mday$hour$min$sec";
# yyyy-mm-dd         $timestamp="$yyyy-$mon-$mday";
# mmddyyyy           $timestamp="$mon$mday$yyyy";
# dd-mm-yy           $timestamp="$mday-$mon-$yy";
# dd-mm-yyyy         $timestamp="$mday-$mon-$yyyy";
# mm-dd-yy           $timestamp="$mon-$mday-$yy";
# mm-dd-yyyy         $timestamp="$mon-$mday-$yyyy";
# dd-mmm-yyyy        $timestamp="$mday-$thismon-$yyyy";
# mmm dd, yyyy       $timestamp="$thismon $mday, $yyyy";

#yyyymmddhhmm
$timestamp="$yyyy$mon$mday$hour$min$sec";
print "\n TIMESTAMP: $timestamp \n";
return ($timestamp);
}

Tuesday, May 28, 2013

Raspberry Pi Digital Picture Frame (Part 1)

A few years ago I purchased a Kodak Pulse email addressable picture frame for my mother.  Though a little expensive at $200 I thought this was a great product. It allowed my siblings and I to send pictures to Mom from our cellphones and other devices. It was fun for Mom as well as she could see pictures of the grandkids almost immediately after an event happened.  Unfortunately the picture frame stopped connecting to my mothers WI-FI network and I was unable to fix it.  I didn't want to spend another $200 thus I began looking at alternatives, old desktops/laptops/tablets and perhaps using flikr, or some other photo sharing service. I didn't want to be tied down to yet another online service.  As I was researching alternatives the Raspberry PI B was announced, this was the perfect solution, small footprint, linux based and cheap, and I probably had stuff laying about the house.

Before I begin describing the build I'd like to give a shout out to Cameron Wiebe for his excellent post on building a Raspberry Pi picture frame some of the instructions below come directly from his article.

What I Built

A digital picture frame that automatically checks a gmail account for new messages then downloads any attached images and automatically displays them. I configured the gmail account for IMAP, and used a Perl script to query the account.

What I Used

Things I bought:
Raspberry PI B (512 MB): $39.99
NetGear USB wireless card. $25.00.
USB keyboard: $7
USB mouse: $3

Things I had laying around:
4GB micro SD card with a bunch of adapters. See pic below.
7" Polaroid monitor, left over from a two screen auto DVD player
Video cable
5V 700mA wall wart left over from an old Nokia cell phone
12 V wall wart to power the monitor borrowed from a USB switch.
4 port unpowered usb hub

Setting up the Raspberry Pi
I started with the instructions here , then went here to download Raspberry Pi Wheezy. I have a MacBook so I followed the excellent instructions here to copy the image to the SD card. Be patient the transfer takes a few minutes. Please read the instructions carefully. If you do it wrong you could trash your system.

Now plug everything in and power up. I was pleasantly surprised everything started up without any issues.  Read Part 2 for complete details to get your Digital Picture Frame up and running.















Tuesday, May 7, 2013

Makespec.py: error: Requires at least one scriptname file

Recently I inherited a set of python scripts with corresponding windows executable files. To create the executable (.exe) file one can used the tool pyinstaller. The scripts were maintained by multiple people using a shared network drive. A development team on the other side of the world needed access to the network share.  To give them access I would have had to work with the corporate network team, which isn't always easy. However, this team has access to our centralized SCM system. As a Release Engineer and Subversion admin, I felt that these scripts should be put into source control. I proceeded to place the scripts in Subversion without any issues. I then needed to create a Jenkins job to execute the pyinstaller scripts. I spent nearly a day hacking the build script, and using uncle Google searching for the error in the header. After hitting my head repeatedly against the wall I discovered the simplest of solutions, and probably the most obvious.

I simply changed this:
C:\apps\pyinstaller\Makespec.py --onefile --out=. --noconsole --name=cl_downloader downloader.py 


Downloader.py
usage: python Makespec.py [opts] <scriptname> [<scriptname> ...]
Makespec.py: error: Requires at least one scriptname file


To this:
python C:\apps\pyinstaller\Makespec.py --onefile --out=. --noconsole --name=cl_downloader downloader.py 

I'm hoping that this simple solution will keep someone else from getting a headache.