Upgrading from Subversion 1.7 to 1.8

I recently worked with an organization to upgrade their Subversion repository from 1.7 to 1.8.  The steps involved, if everything goes according to plan, are:

1. Dump the repository:

   svnadmin dump repo1_7 > repo1_7.dump

   Or you can reduce the dump size and calculate the deltas between revisions and only dump them:

   svnadmin dump --deltas repo1_7 > repo1_7.dump

2. Create the new repository, which by default uses the fsfs file system:

   svnadmin create repo1_8

3. If you need to transfer the dump file to a new host you should take care of that first but then you need load the dump:

   svnadmin load repo1_8 < repo1_7.dump

Now that would generally be it as far as the repository itself is concerned, unless you get errors.  I got errors and it is not uncommon when upgrading.  I ran into issue on step 3, loading the dump file.  It ran for about 2,000+ revisions and then stopped with these errors:

   Cannot accept non-LF line endings in 'svn:log' property
   Cannot accept non-LF line endings in 'svn:ignore' property

This occurred because the repository contained the older ^M carriage return.  This is no longer allowed after version 1.6 but the previous upgrades used the"--bypass-prop-validation" option when loading the repository which just delayed dealing with the issue.  The fix for this is to replace the ^M line endings found in the repository.  To do this you take your dump file you created in step 1 and use the following sed command to replace the ^M line endings.

   sed -e '/^svn:log$/,/^PROPS-END$/ s/^M/ /' -e '/^svn:ignore$/,/^PROPS-END$/ s/^M/\n/' repo1_7.dump > repo1_7-fix.dump

Now I want to make note that the ^M in the command above is not created with Shift+6 and Shift+M.  The ^M character is a carriage return control character that means 0D in hex.  In order to create the character you can either type CTRL+V and CTRL+M or reference it as \x0D in the sed command as shown below.

   sed -e '/^svn:log$/,/^PROPS-END$/ s/\x0D/ /' -e '/^svn:ignore$/,/^PROPS-END$/ s/\x0D/\n/' repo1_7.dump > repo1_7-fix.dump




Once that command finishes you will have repaired copy of the dump file which should load now.


Configuring Jenkins to use HTTPS on CentOS 6.6

These instructions are using Jenkins 1.595 from the Jenkins Yum repository http://pkg.jenkins-ci.org/redhat/. 

1. Create certificate for host with whatever tool you choose.  Personally I like XCA but I don't have a PKI system in place.
   
   
2. Export PKCS12 certificate with chain.
   
   
3. Convert PKCS12 certifcate to java keystore using the following command:
   keytool -importkeystore -srckeystore certificate.p12 -srcstoretype PKCS12 -destkeystore jenkinsstore
   
   
4. Copy the keystore to a permanent location (ex. /var/lib/jenkins).
   
   
5. Import your CA certificate into Java cacerts keystore:
   keytool -import -file CA.crt -keystore /usr/java/latest/jre/lib/security/cacerts
   
   
6. Configure /etc/sysconfig/jenkins with the following settings:
   JENKINS_JAVA_CMD="/usr/java/latest/bin/java"
   JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xrs -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=512m"
   JENKINS_HTTPS_PORT="8843"
   JENKINS_HTTPS_KEYSTORE="/var/lib/jenkins/jenkinsstore"
   JENKINS_HTTPS_KEYSTORE_PASSWORD="thePassword"
   JENKINS_HTTPS_LISTEN_ADDRESS="0.0.0.0"
   
   
7. Configure iptables to redirect 443 to 8843 and to block tcp 8080 if you want to use the standard port 443:
   -A INPUT -i eth0 -p tcp -m tcp --dport 8080 -j DROP
   -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8843
   
   
8. Start Jenkins:
   service jenkins start

Quick OpenLDAP Setup on CentOS 6.6

1. Install the necessary files:
   

   yum install openldap openldap-clients openldap-servers

2. Modify the following options in the /etc/openldap/slapd.d/cn\=config.ldif configuration file:
   

   #olcAllows: bind_v2  

   olcIdleTimeout: 60

3. Generate the SSHA hash for the admin user:
   

   slappasswd -s password

   (example output: {SSHA}abunchofhash)
   

4. Modify the following configuration options in /etc/openldap/slapd.d/cn\=config/olcDatabase\=\{2\}bdb.ldif. The domain will be test.com
   

   olcSuffix: dc=test,dc=com
   olcRootDN: cn=admin,dc=test,dc=com
   olcRootPW: {SSHA}abunchofhash
   

5. Modify the olcAccess option in /etc/openldap/slapd.d/cn\=config/olcDatabase={1}monitor.ldif so the dn is correct:
   

   olcAccess: {0}to *  by/ dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"/
   read  by dn.base="cn=admin,dc=test,dc=com" read  by * none

6. Start the OpenLDAP server and configure it to start at boot time:
   

   chkconfig slapd on
   service slapd start

7. Create an LDIF (LDAP Interchange Format) file with the configuration for our organization LDAP tree. Here we will create two organizational units one called People, where all users be a member of this ou, and another ou called Groups, which will be used to create groups for our organization. At the end of the file specify who is the RootDN for this LDAP tree (cn=admin,dc=example,dc=com).  I named this file ldapconfig.ldif:
   

   dn: dc=test,dc=com
   objectclass: dcObject
   objectclass: organization
   o: Test Org
   dc: test
   
   dn: ou=Users,dc=test,dc=com
   objectClass: organizationalUnit
   objectClass: top
   ou: Users
   
   dn: ou=Groups,dc=test,dc=com
   objectClass: organizationalUnit
   objectClass: top
   ou: Groups
   
   dn: cn=admin,dc=test,dc=com
   objectclass: organizationalRole
   cn: admin

8. Apply our LDIF file and test if the LDAP tree is ready with the ldapsearch command:
   

   ldapadd -x -D "cn=admin,dc=test,dc=com" -W -f ldapconfig.ldif

   ldapsearch -x -b 'dc=test,dc=com' '(objectclass=*)'

9. Create an ldap user by adding the below to an ldif file and running ldapadd as above:
   

   dn: uid=user1,ou=Users,dc=test,dc=com
   objectclass: top
   objectclass: person
   objectclass: inetOrgPerson
   objectclass: organizationalPerson
   uid: user1
   cn: User 1
   sn: 1
   givenName: User 1

10. Assign a password for the user:
    

    ldappasswd -S -x -D "cn=admin,dc=test,dc=com" -W/ uid=user1,ou=People,dc=test,dc=com

11. Create a group in the Groups organizational unit by adding the below to an ldif file and running ldapadd as above:
    

    dn: cn=group1,ou=Groups,dc=test,dc=com
    cn: group1
    objectclass: groupofnames
    member: uid=user1,ou=Users,dc=test,dc=com

12. To add a newly created user to the group after the initial creation create another ldif file and add the below text to it.  Then modify the group with ldapmodify:
    

    dn: cn=group1,ou=Groups,dc=test,dc=com
    changetype: modify
    add: member
    member: uid=user2,ou=Users,dc=test,dc=com

    
    

    ldapmodify -x -D "cn=admin,dc=test,dc=com" -W -f addto_group1.ldif

13. To remove a user from a group create another ldif file and add the below text.  Use ldapmodify to again modify the group:
    

    dn: cn=group1,ou=Groups,dc=test,dc=com
    changetype: modify
    delete: member
    member: uid=user2,ou=Users,dc=test,dc=com

    
    

    ldapmodify -x -D "cn=admin,dc=test,dc=com" -W -f removefrom_group1.ldif

Generating GPG Keys on CentOS 6

I recently ran into an issue where I was having problems creating a new set of GPG keys on a CentOS host that I was using.  Here is the process that eventually worked for me.

1. Run the following gpg-agent command:

gpg-agent --daemon --use-standard-socket --pinentry-program /usr/bin/pinentry-curses

2. Run the following rngd command to ensure there is enough entropy being generated:

sudo rngd -r /dev/urandom

3. Finally run the gpg command to generate the key:

gpg --gen-key

If you are running the gpg-agent as root you may run into other issue that prevent you from completing the process.  It is best to run as a non-root user.

Nate

Java Keystore Import Error

When trying I started recieveing the following error:

keytool error: java.lang.Exception: Input not an X.509 certificate

My certificate was in the .pem format and all that I had to do was convert the certificate using the following command:

openssl x509 -outform der -in mycert.pem -out mycert.der

Then I was able to import the certificate using:

keytool -keystore /location/of/keystore -import -alias mycertalias -file mycert.der

XCA Certificate Authority

After much searching and testing various Linux GUI certificate authority applications I finally found one that works consistently and gives me all of the options that I need.  I am not exactly sure how I missed it as it has been out for some time but XCA has turned out to be exactly what I was looking for.  The biggest issue I had was creating certificates for Windows Servers in order to setup LDAPS.  I had used TinyCA2, GnoMint, and various others.  Both were either limited in their capabilities or failed to work consistently.  I even tried to go setup full PKI systems like EJBCA and Windows Certificate Services but found them to be to difficult to pick up in the limited amount of time I had to learn how to use it.

XCA uses a Qt based GUI interface that is well organized and full featured.  I found the ability to create templates a great time saver when you have to crank out lots of certificates. Plus the ability to export to any format is also very handy as sometimes I never know what I need and I can never remember the correct openssl commands to convert to different formats.

http://sourceforge.net/projects/xca/

http://xca.sourceforge.net/

Importing Key and Certificate into Java Keystore

For whatever reason I had a hell of a time trying to import the key and certificate from a third party CA into a Java keystore (JKS) that was used by a java web application.  I just kept finding incorrect steps on how to do it and then after about 3 days of monkeying around with it I finally found what worked.  

The first step was to export the key and certificate into PKCS12 format (either .pfx or .p12 file extension).  The keytool treats the PKCS12 file as a keystore.  If you are using Tomcat or some other Java application server you can actually use the file as a keystore without importing it into a JKS. The keytool gives you the ability to merge keystores which is what I ended up doing.  After several attempts the below command was what I ended up with.

keytool -importkeystore -destkeystore /certs/my.javakeystore -srckeystore certAndKey.p12  -srcstoretype PKCS12

I tried several times with the -alias option and it kept failing giving this error:

keytool error: java.lang.Exception: Alias <alias1> does not exist

I found that the alias that is used initially is in the PKCS12 file.  If the alias you are using does not match up to the alias, which I am assuming is the internal or friendly name, then it will fail.  So if you leave the alias option off then it will import using the default alias that is sees in the PKCS12 file.  In my case it was identical to one I had in my keystore already and it let me change the alias name during the import process.

Setting GRUB2 Password on Debian Squeeze (6.0.5)

To generate the hashed password run the following and enter the password you want to use:

grub-mkpasswd-pbkdf2

Example output:

grub.pbkdf2.sha512.10000.AKSDAKSDJ939302502NKJKANVKANVWIEN2034928345092835ADSIFUA09UFN23548098DF0SDFLKSJDRO0395203WELRFISDUF09U45

Then edit /etc/grub.d/40_custom and append the following lines to the end of the file.  I am using the user root here in this instance.  It is not the same as the system's root account and you can create multiple accounts if you want with whatever names you wish.  Make sure you replace the entry after root with your own output from the grub-mkpasswd-pbkdf2 command.

set superusers="root"

password_pbkdf2 root grub.pbkdf2.sha512.10000.AKSDAKSDJ939302502NKJKANVKANVWIEN2034928345092835ADSIFUA09UFN23548098DF0SDFLKSJDRO0395203WELRFISDUF09U45

Once you have saved that run update-grub and reboot your system.  When it gets to the boot menu then you should try to edit the entries and see if it asks for your username and password in order to test the configuration.

How to Add a Jailed User to the SFTP Only

*Note* These instructions work only with OpenSSH version 5.0 and newer.

Now when you create users that need to be jailed, make sure they belong to the 'sftponly' group. For the user "denis" with the password "test", you will need to do the following steps.

useradd denis
usermod -g sftponly denis
usermod -s /bin/false denis
usermod -d /home/denis denis
passwd denis

Add the User to the SSHD Config

nano /etc/ssh/sshd_config

Add the new user to the end of the list with AllowUser before their name.

Then restart sshd by running: /etc/init.d/sshd restart

To set up the jail run the following commands

chmod 755 /home/denis
chown root:root /home/denis
mkdir /home/denis/data
chown denis:sftponly /home/denis/data

Setting up the atftpd server on Debian Squeeze

I have recently had some issues trying to use tftpd server in conjunction with some of my Cisco equipment.  I kept getting these errors regardless of actual permissions on the system:

%Error opening tftp://192.168.130.6/c1841-ipbase-mz.124-1c.bin (Permission denied)

So I uninstalled that and installed the atftpd server package.  That seemed to do the trick.  I did have to change the default directory as I did already have a directory full of IOS files that I was just too lazy to move where the maintainers think I should put it.  So here are the steps I took to get things working.

1. Install the atftpd package from the repository.

apt-get install atftpd

2. Add this line into /etc/inetd.conf:

tftp dgram udp4 wait nobody /usr/sbin/tcpd /usr/sbin/in.tftpd --tftpd-timeout 300 --retry-timeout 5 --mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1 --maxthread 100 --verbose=5 /tftpboot

3. Create /tftpboot directory and change permissions to allow writing to the directory.

mkdir /tftpboot

chmod 777 /tftpboot

4. Restart the inetd server.  To restart the server on Debian run:

/etc/init.d/openbsd-inetd restart

After I followed those steps I was able to connect to the server from my Cisco equipment and transfer files both ways.  

You may want to disable the server when you are not actively using it.  To do this comment out the line you added to /etc/inetd.conf and restart or stop inetd.

Changing umask setting for all users

In order to have the umask setting change for all users at login the umask setting in /etc/profile needs to be changed.

1. Use text editor to edit /etc/profile. *Note: You need root access to edit this file.

sudo nano -w /etc/profile

2. Edit the umask line located at the end of the file. *Note: This example is setting the umask to allow the owner and group have read/write while others have no permissions.

umask 007

3. Type in Crtl-X, Y, and hit the Enter key to save and exit.

Changing Subversion revision times

Occasionally revision entries in Subversion will not have a time/date included in the entry.  In my experience they are not entered when filtering information from other repositories or converting other types of version control data to Subversion.  If you are using Apache2 to handle the authentication errors will appear on the server's /var/log/httpd/error_log that look like this one:

[Thu Jan 13 07:31:32 2011] [error] [client 192.168.13.89] Could not access revision times.  [500, #0]

*Note* This only appears to be an issue for users and systems that use SVNKit as their Subversion client as most people will still be able to connect with Eclipse and other clients to check in data.  Hudson will usually have the most problems with this.

In order to correct the problem the entries with the missing dates will need to be located.  This is easily done with the following command string which needs to be run on the Subversion server itself.

svn log --xml -v -r {2010-12-01T00:00:00Z}:{2011-01-13T15:00:00Z} file:///svn/repo

If there is any issue related to the time on a revision then the following error will be shown when you run the above command.

<?xml version="1.0"?>

<log>

svn: Failed to find time on revision 34534

The first step in making the necessary corrections is to edit the /svn/repos/mosaic/hooks/post-revprop-change file so that it contains the following information.

#!/bin/sh
exit 0

If the previous steps had not be performed the hook files would have prevented this step from actually occurring. So to make the change to the revision entry the svn command is used on the server itself using the propset subcommand. When entering the date try to select a date that is in between the dates that proceed and follow this revision entry.

svn propset -r34534 --revprop svn:date '2010-11-29T19:55:44.000220Z' file:///svn/repo

If the command is successful then following message will appear.

property 'svn:date' set on repository revision 34534

After that re-run the svn log command to verify there are no further entries that will cause errors. If there are none that the Subversion server are registering then you should see all of the log entries for that time period displayed.  This is not always a positive indication. It has been my experience that Subversion does not always register an error for all missing date entries. The best way to find out if there are any missing date entries is to copy the output of the svn log command to a searchable text document where <date></date> can be searched for.  If there are no other missing dates in the log then the final steps are to reverse the changes made to hook files and return them to their previous state

Setting InnoDB as the default MySQL Engine

In order to set InnoDB as the default storage engine in MySQL version 5.0.x the process is simple.

Open the my.cnf file in a text editor such as vi.  If the following line is in the config it should be commented out or deleted:

skip-innodb

Add this line under [mysqld] to make InnoDB the default engine:

default-table-type=innodb

Save the file and restart mysqld.

To verify that the default has been changed from MyISAM  to InnoDB login to MySQL and run the following:

show engines;

You should see that InnoDB is now the default.

+------------+---------+----------------------------------------------------------------+

| Engine     | Support | Comment                                                        |

+------------+---------+----------------------------------------------------------------+

| MyISAM     | YES     | Default engine as of MySQL 3.23 with great performance         | 

| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      | 

| InnoDB     | DEFAULT | Supports transactions, row-level locking, and foreign keys     | 

| BerkeleyDB | YES     | Supports transactions and page-level locking                   | 

| BLACKHOLE  | NO      | /dev/null storage engine (anything you write to it disappears) | 

| EXAMPLE    | NO      | Example storage engine                                         | 

| ARCHIVE    | NO      | Archive storage engine                                         | 

| CSV        | NO      | CSV storage engine                                             | 

| ndbcluster | NO      | Clustered, fault-tolerant, memory-based tables                 | 

| FEDERATED  | NO      | Federated MySQL storage engine                                 | 

| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          | 

| ISAM       | NO      | Obsolete storage engine                                        | 

+------------+---------+----------------------------------------------------------------+

Increasing VMware virtual disks in Linux

The process of increasing a VMware virtual disk in Linux is fairly straight forward.  I have VMware Workstation 7 installed and it comes with the tool vmware-vdiskmanager which is what I will use to grow the virtual disk.  To perform a size increase of 300GB run the following command:

vmware-vdiskmanager -x 300GB virtual-disk-name.vmdk

Manually installing Sun JDK in Debian/Ubuntu

In order to manually install the JDK .bin file and change the default Java directory you need to do the following.

1. Download the jdk-****.bin installer (I am using jdk-1.6.0_20-linux-i586.bin) to your download directory (I am using /home/myaccount/downloads)

2. Run:

cd /opt

sudo sh /home/myaccount/downloads/jdk-1.6.0_20-linux-i586.bin

3. Hit the spacebar a few times and type yes when prompted.

4. Run:

update-alternatives --install /usr/bin/java java /opt/jdk1.6.0_20/bin/java 500

update-alternatives --set java /opt/jdk1.6.0_20/bin/java
OR
update-alternatives --config java

**Note: The --set option will not prompt you while the --config option will present you with a list of option.

Changing Hudson Context Path on CentOS

I have recently had to setup Hudson running on a CentOS server as well as some other web based applications.  I found that in order to get Apache to proxy Hudson I had to change the context path for Hudson.

Now I simply used the Hudson repository for RedHat/Fedora/CentOS (http://hudson-ci.org/redhat/) so I didn't just unpack Hudson on the system and go from there.  That being said the installation files were spread across the file system.  The file that needs to be edited in order to change the context path is: /etc/sysconfig/hudson 

To change the context path simply add --prefix=/hudson to the value of that variable like this:

HUDSON_ARGS="--prefix=/hudson"

For more information on setting up Apache to proxy Hudson here are a couple of links that I found helpful though geared toward Ubuntu:

http://wiki.hudson-ci.org/display/HUDSON/Running+Hudson+behind+Apache

http://www.zzorn.net/2009/11/setting-up-hudson-on-port-80-on-debian.html