No Space Left on Device

Issue: No Space Left on Device

Discussion:

I have a server which has been setup to capture uploaded crash logs. Every hour I transfer these files (using rsync) to another server for statistical analysis.  One of our end users noticed that the current day statistics were not as expected.  I manually ran the script and noticed the No space left on device error below.  At first didn't understand how I could be out of space when a df -h returned 14 Gigabytes free.  After a reboot, and a little Googling I realized that the system was out of available Inodes.  

For those new to Linux an Inode is simply a pointer to a file, or more precisely pointer or index to a block on a disk which contains either part of or all of a file. In the old days a writable block on a system might be 512 Kb, now it is typical to have a block size of 4096 Kb or even 8192 Kb.  If a file doesn't fit into a single block then the next available block is written to and the system "magically" keeps track of the next block. On my system each of these crash logs was very small, and over the previous 6 months we had over 1.6 million uploaded to the server each file took up one Inode, and eventually I ran out of Inodes even though I didn't run out of diskspace.

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135212_0.log.MdO3DE" failed: No space left on device (28)

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135217_0.log.qz7uz4" failed: No space left on device (28)

rsync: mkstemp "/usr/local/crashlogs/20141120/.20112014135219_0.log.73mgvu" failed: No space left on device (28)

#df -h

Filesystem                   Size  Used Avail Use% Mounted on

/dev/mapper/vg_tern-lv_root   26G   11G   14G  45% /

The out of disk space error was related to the lack of available Inodes.

#df -ih

Filesystem                  Inodes IUsed IFree IUse% Mounted on

/dev/mapper/vg_tern-lv_root   1.6M  1.6M     1  100% /

Solution:

  There are some versions of Linux, and file system types that will allow one to increase the number of available Inodes. Mine was not one of them.  One option would be to back everything up reformat the disk with more Inodes then restore the system.  This was not an option for me.  The final solution is to remove unnecessary files.  Lucky for me the end user didn't care about old data thus she gave me permission to remove three months worth of crash logs. This freed up over 500,000 inodes.

IF you don't know the directory that is causing you issues below is a one line script that can help you find the location of your many small files.

for i in /*;do echo $i; find $i | wc -l; done

Jenkins, Maven error: Target repository cannot be empty

After applying the previous three fixes (see my other recent posts) to my Jenkins instance my Maven builds were still failing. I hope these posts can help others.

Issue:

mavenExecutionResult exceptions not empty
message : Internal error: java.lang.IllegalArgumentException: Target repository cannot be empty
cause : Target repository cannot be empty
Stack trace : 
org.apache.maven.InternalErrorException: Internal error: java.lang.IllegalArgumentException: Target repository cannot be empty
....snip...

at org.jfrog.build.client.DeployDetails$Builder.build(DeployDetails.java:115)

Discussion:
After asking Uncle Google, I only turned up the jfrog code that returns the "Target repository cannot be empty" message.  After a little more prodding of Uncle Google, Uncle finally revealed that the Jenkins Artifactory Plug-in uses jfrog  JENKINS Artifactory Plugin. Now I had somewhere to look.  I looked at my Jenkins instance and attempted to set the Artifactory target repository, but the Artifactory plugin appeared to be corrupt, as I wasn't getting any place to enter the target repository. Thus it appeared that the Artifactory plugin was probably corrupt. This was most likely due to my need to downgrade Jenkins to an older version.


Solution:  I first attempted to use the Manage Plugin feature to uninstall the Artifactory plugin but it didn't actually remove it.  Thus I forcibly removed Artifactory Jenkins plug-in, by removing JENKINS_HOME/plugins/artifactory directory and the artifactory.jpi file. I believe there are still some xml files I need to hack but I'm finally back to building my software.

git: Failed to connect to repository

My Jenkins instance was failing to connect to a git repository using a supplied username and private key. The Credentials plugin appeared to be getting confused. I hope the following can help others avoid a bit of head banging against your desk.

Issue: 

Failed to connect to repository : Command "ls-remote -h ssh://myuser@repo HEAD" returned status code 128:
stderr: Permission denied, please try again. 
Permission denied, please try again. 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 
fatal: The remote end hung up unexpectedly

Discussion: 

In my situation the Jenkins instances is pointing to several different repositories (svn and git)
The connection to the svn repository uses svn+ssh. The connection to the git repository ,using ssh, was failing. It appears that the Jenkins instance is having trouble figuring out which credentials to use or private key to use.


Solution: 
The solution is to define the private key to use for the user that is failing. I found the solution here at: stackoverflow  then modified for my situation.

Create a JENKINS_HOME/.ssh/config file to point to the correct private key.
Set permissions to 600 ( rw------- )
Set the contents of the file to:

Host source.server.com
HostName 123.456.789.012
User myuser
IdentityFile /home/jenkins/.ssh/id_rsa

Where: Host:            Can be anything is it just a place holder I generally use the DNS name 
                                 or short name of the server.
            HostName:    IP addresss or fully qualified domain name if you have DNS 
                                 enabled on your network
           User:             The user that will be connecting to the repo
  IdentityFile: The ssh private key that will be used to connect to the repository.
                                      In my case the user  and key are not the jenkins user and id_rsa. 

Finally: 

You must add the public key to your known_hosts file. The easiest way to add this is to execute: ssh myuser@source.server.com   
Then answer yes.

Alternatively if you have the public key you can directly edit your known_hosts file and paste the public key to the bottom of the file.

Jenkins Maven Jobs fail to Parse POMS

I had several recent Issues with Jenkins and Maven I hope this will help someone else out there with the same issue.

Issue: 

Jenkins Maven Jobs fail to Parse POMS
Caused by: java.lang.ClassNotFoundException: org.apache.maven.cli.MavenLoggerManager

Discussion: 

It appears that Maven 3.1.0, and 3.1.1 (I tried both) have not been compiled into Jenkins yet.
More can be found here:JENKINS-15935  as of today 10/25/2013 the fix version has not been set. So I'm guessing it will be fixed soon.

Solution: 

Roll back Maven to 3.0.5, This assumes that you installed maven yourself,(ie /usr/local/maven) rather than having Jenkins do it for you which I didn't try.

Jenkins Maven jobs do not log errors

I recently had a few issues with my Jenkins, Maven, Git environment failing builds without displaying any actual errors. To say the least this was very frustrating especially since the Maven builds ran just fine from the command line in the Jenkins work space.

Issue: Jenkins Maven jobs do not log errors.

Discussion:

 Apparently a bug was introduced in Jenkins with the release of 1.526 which caused Jenkins to not display errors generated by maven, Only displaying the build as a failure. This is discussed in the Jenkins Jira instance here:JENKINS=19352 

Solution: 

Roll back Jenkins to version 1.525

Simple Perl Time stamp function

##################
# timestamp: Simplified function returns a timestamp 
# calling profile: mytimestamp=&timestamp();
# returns: timestamp
##################
sub timestamp {
my ($flag,$message) = @_;
my $timestamp;
my $timedate;
my $date;
my $time;
my $sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst;
my $thisday, $thismon;
my $yy,$yyyy;


($sec,$min,$hour,$mday,$mon,$yy,$wday,$yday,$isdst) = localtime(time);
$thisday= (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[$wday]; 
$thismon= (Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec)[$mon];

#If the year month, day, hour, minute, or second, are less than 10 prepend with a 0

$yyyy = $yy + 1900; #add 1900 to get 4 digit year
$yy   -= 100;       #subtract 100 to get 2 digit year

if ($yy < 10 ) {
  $yy = "0$yy";
}

$mon++;   #add 1 to month as month array starts at 0
if ($mon < 10 ) {
  $mon = "0$mon";
}

# Day
if ($mday < 10 ) {
  $mday = "0$mday";
}

# Hour
if ($hour < 10 ) {
  $hour = "0$hour";
}

# Min
if ($min < 10 ) {
  $min = "0$min";
}

# Second
if ($sec < 10 ) {
  $sec = "0$sec";
}

# TO GET FORMAT                 USE THIS
#=================================================
# hh:mm:ss           $timestamp="$hour:$min:$sec";
# hhmmss             $timestamp="$hour$min$sec";
# dd/mm/yy           $timestamp="$mday/$mon/$yy";
# mm/dd/yy")         $timestamp="$mon/$mday/$yy";
# mm/dd/yyyy         $timestamp="$mon/$mday/$yyyy";
# yyyymmdd           $timestamp="$yyyy$mon$mday";
# yyyymmddhhmm       $timestamp="$yyyy$mon$mday$hour$min";
# yyyymmddhhmmss     $timestamp="$yyyy$mon$mday$hour$min$sec";
# yyyy-mm-dd         $timestamp="$yyyy-$mon-$mday";
# mmddyyyy           $timestamp="$mon$mday$yyyy";
# dd-mm-yy           $timestamp="$mday-$mon-$yy";
# dd-mm-yyyy         $timestamp="$mday-$mon-$yyyy";
# mm-dd-yy           $timestamp="$mon-$mday-$yy";
# mm-dd-yyyy         $timestamp="$mon-$mday-$yyyy";
# dd-mmm-yyyy        $timestamp="$mday-$thismon-$yyyy";
# mmm dd, yyyy       $timestamp="$thismon $mday, $yyyy";

#yyyymmddhhmm
$timestamp="$yyyy$mon$mday$hour$min";

return ($timestamp);
}

Apache Error [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

Problem: One day I woke up and my Apache server which had been running for several years was down and wouldn't restart. I checked the Apache error log and found the following error message:

[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

This particular error message caused Uncle Google to speak in riddles, many of the forum posts that were found led to non answers. After fumbling around a bit, I noticed that another Apache log " nss_error_log" was being updated and displayed the following error messages:

[error] Certificate not verified: 'Server-Cert'
[error] SSL Library Error: -8181 Certificate has expired
[error] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.

Now we've found the root cause of our issue it is in a module called nss.  I never noticed it before so another visit to Uncle Google actually turned up some interesting reading and several solutions. In short mod_nss is an alternative to mod_ssl it does a few things that mod_ssl doesn't do, and can run along side mod_ssl. 

Since I'm not now an expert on mod_nss, I'll leave the gory details of how to set it up and what it does to these articles [What is mod_nss , On Setting up Mod_NSS ,More detail ] but suffice it to say chances are if you didn't know you were using it you may not need it. If you are inheriting a system you probably want to dig deeper into why it is there. Below I'm listing four possible solutions. 

1. Simply do as the error message above says add NSSEnforceValidCerts off to your nss.conf file which is usually located in /etc/httpd/conf.d.   The downside to this is that your nss_error_log will continue to get messages like "SSL Library Error: -8181 Certificate has expired"
2. Remove nss.conf from /etc/httpd/conf.d this will of course cause mod_nss to not be loaded.
3. Uninstall/re-install your nss rpm modules on re-installation a new certificate will be generated and your problem will go away for a few more years
4. Properly setup your nss db or trust store using the certutil command. (you can Google for more details)