1. smithy(1)
  2. smithy(1)


smithy - build, test, and install software with ease


smithy formula install APPLICATION/VERSION/BUILD


smithy is a command line software installation tool that borrows ideas heavily from the excellent homebrew package management system for Mac OS X and SWTools.

This is still a work in progress. If you have any questions, suggestions, or encounter any errors please open an issue on github at https://github.com/AnthonyDiGirolamo/smithy/issues.


For help on all options and commands run smithy help
For help on a specific command run smithy help COMMAND or smithy help COMMAND SUBCOMMAND


There are two ways of installing software using smithy:


Build scripts are shell scripts that live inside of a package's PREFIX and execute the steps required to perform the compilation. A hierarchy might look something like this:

`--- sles11.1_gnu4.3.4
   |--- bin
   |--- build-notes
   |--- dependencies
   |--- include
   |--- lib
   |--- rebuild
   |--- relink
   |--- remodule
   |--- retest
   |--- share
   `--- subversion-1.6.17

The prefix for the above package is /sw/xk6/subversion/1.6.17/sles11.1_gnu4.3.4 within this directory you can see the usual bin/include/lib/share folders. Additionally there is the source directory subversion-1.6.17 and three build scripts rebuild, relink, and remodule. The rebuild script will compile the software and set the prefix while the remodule script sets up the environment including loading modulefiles.


The Problem with build scripts is that they are duplicated for every software installation. This makes installing new software difficult because you will have to go back and look at old ones and copy relevant steps to a new rebuild script. So much for don't repeat yourself!

An alternative is to use formulas. This idea is heavily borrowed from the wonderful homebrew package management system for Mac OS X. With a formula you can specify the modules to load, extra package dependencies, installation steps, and the modulefile in a single file. The formulas are written in ruby and can be as flexible and dynamic as you like.

A package installed with a formula has a simpler prefix directory structure. The only addition is the source files under the source directory.

`--- sles11.1_gnu4.3.4
   |--- bin
   |--- include
   |--- lib
   |--- source
   `--- share


All software builds have the following name format: APPLICATION/VERSION/BUILD That is three different parts separated by forward slashes / Each part consists of:


The name using lowercase characters


Numbers with periods. I recommend that whatever you choose as a version allows it to be lexigraphically sorted from oldest to newest. For example use 2013-02-14 instead of Feb14-2013.


Build is somewhat different, it consists of the intended operating system and compiler separated by underscores _. It is important to use compiler version numbers that correspond to module versions. This will allow smithy to auto-generate modulefiles for software with multiple builds.

sles11.1_gnu4.6.2 corresponds to SuSE Linux Enterprise Server 11.1 and the GNU gcc compiler 4.6.2

cle4.0_pgi12.1.0 corresponds to Cray Linux Environment 4.0 and the PGI 12.1.0 compile

You may also add any other details relevant to the build. For instance, add python2.7 to the build name of a python module compiled for python2.7 and python3.3 for python3.3.


For details on writing formulas please see the smithyformula(5) man page. This section covers only how to install software using existing formulas.

Smithy formula sub-commands include:

Create a new formula
List known formulas
Display a formula location
Display a formula
Install a package using a formula
Create a modulefile for a given package

Once a formula has been written installing is straightforward using the install command. For example to install subversion to your software root under the subversion/1.7.8/sles11.1_gnu4.3.4 directory you might run:

smithy formula install subversion/1.7.8/sles11.1_gnu4.3.4

See NAMING SOFTWARE BUILDS for details on naming a software build. The format of the install sub-command is: smithy formula install [command options] APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD. The options and arguments include:

This will delete all existing files in the target directory before performing the installation.
By default smithy will guess the formula name based on the target directory APPLICATION/VERSION/BUILD argument. You may wish you install to a location named differently than a formula. In this case, use --formula-name to define which formula to use.
This is the destination directory that the software will be installed in. If you omit the BUILD or VERSION/BUILD directories smithy will try to guess the version based on the formula, and the build based on the operating system and version of gcc available. If in doubt, specify the full destination.

Assuming we install subversion with the previous command and our software-root is /sw/xk6/ the finished directory structure would look like:

|--- modulefile
|  `--- subversion
|     `--- 1.7.8
`--- sles11.1_gnu4.3.4
   |--- bin
   |--- include
   |--- lib
   |--- source
   `--- share

A modulefile folder will be created alongside the sles11.1_gnu4.3.4 build directory so that the modulefile can be tested. To test the modulefile you need to add the modulefile folder to the $MODULEPATH environment variable. Running any of the following will do this:

smithy module use last
smithy module use subversion/1.7.8/sles11.1_gnu4.3.4
module use /sw/xk6/subversion/1.7.8/modulefile/subversion

last is an alias to the last software build smithy worked on. smithy show last will display the last build you worked on.

Once loaded, you should be able to interact with the new module file as normal using:

module avail subversion/1.7.8
module display subversion/1.7.8
module load subversion/1.7.8

You can now deploy the module to make it available to other users. This should be done once you're confident the modulefile is working properly. To do so run either:

smithy module deploy last
smithy module deploy subversion/1.7.8/sles11.1_gnu4.3.4


This section shows the typical work-flow for adding a new software build using build scripts. Let's imagine we want to install petsc 3.2 using the cray compiler. We must create a place for the software build, build it, and create a module.

1. Creating The Build

Let's use petsc/3.2/cle4.0_cray8.0.1 as the name for our new package. See NAMING SOFTWARE BUILDS for details on naming a software build. This would be petsc designed to run on a cle4.0 compute node and compiled with the cray cce 8.0.1 compiler. This can be accomplished by running:

smithy new petsc/3.2/cle4.0_cray8.0.1

You can save yourself some extra time by telling smithy where the source tar file for petsc is:

smithy new -t petsc-3.2-p7.tar.gz petsc/3.2/cle4.0_cray8.0.1

When using the -t, --tarfile= option smithy will extract the given archive to the source directory. For the petsc example above this would be /sw/xk6/petsc/3.2/cle4.0_cray8.0.1/source

The -t option can also download an archive from a given URL. The archive is saved along side the source directory. As an example:

smithy new -t http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.2-p7.tar.gz petsc/3.2/cle4.0_cray8.0.1

This command will download petsc-3.2-p7.tar.gz, save it to /sw/xk6/petsc/3.2/cle4.0_cray8.0.1/petsc-3.2-p7.tar.gz and extract it to /sw/xk6/petsc/3.2/cle4.0_cray8.0.1/source. This feature requires the curl command to work properly.

Additionally, if this is a brand new piece of software add the --web-description switch. This will create the application description files too.

2. Editing and Building the Software

Once you have created the build you may need to update the build (rebuild) and environment (remodule) scripts before building the software. Both files live within the software prefix. For our example it is located in /sw/xk6/petsc/3.2/cle4.0_cray8.0.1. You can edit this and other related files using the edit command:

smithy edit build petsc/3.2/cle4.0_cray8.0.1

This is just a convenience command to editing the rebuild script. It will expect the $EDITOR environment variable to be set to the edior of your choice. You can alternatively use the -e,--editor= option. To make things quicker you can run:

smithy edit last

By omitting the build argument, smithy will prompt you to select the file to edit. The last argument will tell smithy to use the last software build it worked with. If you are not sure what the last one is you can run smithy show last to see.

Once you have edited the files, it's time to build. Do so by running:

smithy build last

The results of the run will be shown on the screen and simultaneously logged to build.log withing the software prefix folder.

3. Create and edit a modulefile

This step is best done after all builds for a particular application have been created. When you create a new software build a modulefile is created too. For our petsc install it lives in: /sw/xk6/petsc/3.2/modulefile All builds of a particular application share a single modulefile.

Smithy does some guessing as to what should be inside the modulefile based on what builds you have. If there is only one build the modulefile is relatively simple. If you have multiple builds for different compilers then it will be more complicated.

To be sure the modulefile is up to date on each build, regenerate it with:

smithy module create last

Launch your editor with:

smithy edit modulefile last

Add any relevant variables to your application. Next you will need to test it to make sure it is loading properly for each build and that there are no errors. You can add the modulefile to your $MODULEPATH using:

smithy module use last

Now you are free to display and load it using the normal module commands:

module avail      petsc
module display    petsc/3.2
module load       petsc/3.2

Once the modulefile is ready, you can copy it into place using:

smithy module deploy last

For out petsc example, this command will copy /sw/xk6/petsc/3.2/modulefile/petsc/3.2 to /sw/xk6/modulefiles/petsc/3.2

4. Website Description

If this is a new application you will need to add some information to the description file. For petsc this lives in: /sw/xk6/petsc/description This is an html formatted file. Alternatively, it can live in /sw/xk6/petsc/description.markdown this file is in markdown format and is a bit simpler to write than html. See http://kramdown.rubyforge.org/quickref.html for more information on markdown syntax. If both files exist, the markdown file takes precedence.

If the description file is missing you can generate one by running:

smithy repair last

To publish to the website run:

smithy publish petsc


To see an up to date example of what a smithy config file should look like run smithy show example_config. Most of the options specified in a config file can be overridden by global command line options. See smithy help for info on global command line options. The smithy config file uses the yaml syntax and looks something like this:

software-root: /sw
download-cache: /sw/sources
- /sw/tools/smithy/formulas
- /sw/tools/smithy/another_formula_directory
global-error-log: /sw/tools/smithy/exceptions.log
file-group-name: ccsstaff
descriptions-root: /sw/descriptions
web-root: /sw/descriptions_in_html
  xk6: titan
  xk7: titan
  analysis-x64: lens
  smoky: smoky
  everest-login: redhat6
  everest: redhat6
  lens: analysis-x64
  sith-login: redhat6
  sith: redhat6
  smoky-login: smoky
  titan-login: xk6
  titan-ext: xk6
  yona-login: yona
  yona: yona


This option defines the top level directory used as the location of architecture directories. Good choices might be /opt/smithy or /usr/local/smithy. Someplace under your home directory would work too.


Software compiled with smithy is intended to be group writable so that more that one user may install software in the same location. This option defines the group name used for installing software.


This option defines what hostnames map to which architecture directories. Smithy will use the mapped directories whenever it is run on machines with hostnames specified by this option. Smithy will ignore trailing digits on hostnames. For example:

  everest: redhat6
  titan-login: xk6
  titan-ext: xk6
  yona-login: yona
  yona: yona

Would produce the following matches:


This directory is used when software archives are downloaded via formulas or through the smithy new --tarfile= option.


This option specifies one or more directories where formulas are stored. If multiple directories are used then smithy will use the first formula it finds in the order directories are specified.


This file will log internal errors smithy encounters. It is usually used for development. You can also get extra error output by setting GLI_DEBUG=true in your shell.


The smithy publish command outputs architecture names at the top of description files. If you wish to use a different name this option allows for that. For example:

  xk6: titan
  xk7: titan
  analysis-x64: lens


This directory is the target output directory when running smithy publish


Setting this option changes the way smithy handles description files. When NOT set smithy will expect descriptions to live inside each application folder inside each architecture folder. For example, assuming a software-root of /sw and an architecture of xk6 smithy would expect the description files in these locations:

|--- redhat6
|  |--- subversion
|  |  |--- 1.6.17
|  |  |--- 1.7.8
|  |  `--- description.markdown
|  `--- git
|     |---
|     |---
|     `--- description.markdown
`--- xk6
   |--- subversion
   |  |--- 1.6.17
   |  |--- 1.7.8
   |  `--- description.markdown
   `--- git
      `--- description.markdown

This setup has multiple description files per application. This might make sense if your documentation is highly dependent on the architecture. If we set descriptions-root to /sw/descriptions smithy would expect the descriptions in:

|--- redhat6
|  |--- subversion
|  |  |--- 1.6.17
|  |  `--- 1.7.8
|  `--- git
|     |---
|     `---
|--- xk6
|  |--- subversion
|  |  |--- 1.6.17
|  |  `--- 1.7.8
|  `--- git
|     |---
|     `---
`--- descriptions
   |--- subversion
   |  `--- description.markdown
   `--- git
      `--- description.markdown

With this we have only one description file per application.


This option allows you to override the complier definition for creating modulefiles. The syntax is yaml and the default values are:

- :prg_env: PrgEnv-gnu
  :compiler_name: gcc
  :human_name: gnu
  :regex: !ruby/regexp /(gnu|gcc)(.*)/
  :build_name_regex: !ruby/regexp /(gnu|gcc)([\d\.]+)/
- :prg_env: PrgEnv-pgi
  :compiler_name: pgi
  :human_name: pgi
  :regex: !ruby/regexp /(pgi)(.*)/
  :build_name_regex: !ruby/regexp /(pgi)([\d\.]+)/
- :prg_env: PrgEnv-intel
  :compiler_name: intel
  :human_name: intel
  :regex: !ruby/regexp /(intel)(.*)/
  :build_name_regex: !ruby/regexp /(intel)([\d\.]+)/
- :prg_env: PrgEnv-cray
  :compiler_name: cce
  :human_name: cray
  :regex: !ruby/regexp /(cce|cray)(.*)/
  :build_name_regex: !ruby/regexp /(cce|cray)([\d\.]+)/



  1. March 2015
  2. smithy(1)