smithy - build, test, and install software with ease
smithy new APPLICATION/VERSION/BUILD
smithy edit APPLICATION/VERSION/BUILD
smithy build APPLICATION/VERSION/BUILD
smithy formula install APPLICATION/VERSION/BUILD
smithy is a command line software installation tool that borrows ideas heavily
from the excellent homebrew package management system for Mac OS X and SWTools.
This is still a work in progress. If you have any questions, suggestions, or encounter any errors please open an issue on github at https://github.com/AnthonyDiGirolamo/smithy/issues.
For help on all options and commands run smithy help
For help on a specific command run smithy help COMMAND or smithy help COMMAND SUBCOMMAND
There are two ways of installing software using smithy:
Build scripts are shell scripts that live inside of a package's PREFIX and execute the steps required to perform the compilation. A hierarchy might look something like this:
/sw/xk6/subversion/1.6.17
`--- sles11.1_gnu4.3.4
|--- bin
|--- build-notes
|--- dependencies
|--- include
|--- lib
|--- rebuild
|--- relink
|--- remodule
|--- retest
|--- share
`--- subversion-1.6.17
The prefix for the above package is
/sw/xk6/subversion/1.6.17/sles11.1_gnu4.3.4 within this directory you can see
the usual bin/include/lib/share folders. Additionally there is the source
directory subversion-1.6.17 and three build scripts rebuild, relink, and
remodule. The rebuild script will compile the software and set the prefix
while the remodule script sets up the environment including loading modulefiles.
The Problem with build scripts is that they are duplicated for every software installation. This makes installing new software difficult because you will have to go back and look at old ones and copy relevant steps to a new rebuild script. So much for don't repeat yourself!
An alternative is to use formulas. This idea is heavily borrowed from the wonderful homebrew package management system for Mac OS X. With a formula you can specify the modules to load, extra package dependencies, installation steps, and the modulefile in a single file. The formulas are written in ruby and can be as flexible and dynamic as you like.
A package installed with a formula has a simpler prefix directory structure. The only addition is the source files under the source directory.
/sw/xk6/subversion/1.7.8
`--- sles11.1_gnu4.3.4
|--- bin
|--- include
|--- lib
|--- source
`--- share
All software builds have the following name format: APPLICATION/VERSION/BUILD
That is three different parts separated by forward slashes / Each part consists of:
The name using lowercase characters
Numbers with periods. I recommend that whatever you choose as a version allows it to be lexigraphically sorted from oldest to newest. For example use 2013-02-14 instead of Feb14-2013.
Build is somewhat different, it consists of the intended operating system and
compiler separated by underscores _. It is important to use compiler version
numbers that correspond to module versions. This will allow smithy to
auto-generate modulefiles for software with multiple builds.
sles11.1_gnu4.6.2 corresponds to SuSE Linux Enterprise Server 11.1 and the GNU gcc compiler 4.6.2
cle4.0_pgi12.1.0 corresponds to Cray Linux Environment 4.0 and the PGI 12.1.0 compile
You may also add any other details relevant to the build. For instance, add python2.7 to the build name of a python module compiled for python2.7 and python3.3 for python3.3.
For details on writing formulas please see the smithyformula(5) man page. This section covers only how to install software using existing formulas.
Smithy formula sub-commands include:
newlistwhichdisplayinstallcreate_modulefileOnce a formula has been written installing is straightforward using the install
command. For example to install subversion to your software root under the
subversion/1.7.8/sles11.1_gnu4.3.4 directory you might run:
smithy formula install subversion/1.7.8/sles11.1_gnu4.3.4
See NAMING SOFTWARE BUILDS for details on naming a software build.
The format of the install sub-command is: smithy formula install [command
options] APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD. The options and arguments
include:
--[no-]clean--formula-name--formula-name to define
which formula to use.APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILDAssuming we install subversion with the previous command and our software-root
is /sw/xk6/ the finished directory structure would look like:
/sw/xk6/subversion/1.7.8
|--- modulefile
| `--- subversion
| `--- 1.7.8
`--- sles11.1_gnu4.3.4
|--- bin
|--- include
|--- lib
|--- source
`--- share
A modulefile folder will be created alongside the sles11.1_gnu4.3.4 build
directory so that the modulefile can be tested. To test the modulefile you need
to add the modulefile folder to the $MODULEPATH environment variable. Running
any of the following will do this:
smithy module use last
smithy module use subversion/1.7.8/sles11.1_gnu4.3.4
module use /sw/xk6/subversion/1.7.8/modulefile/subversion
last is an alias to the last software build smithy worked on. smithy show
last will display the last build you worked on.
Once loaded, you should be able to interact with the new module file as normal using:
module avail subversion/1.7.8
module display subversion/1.7.8
module load subversion/1.7.8
You can now deploy the module to make it available to other users. This should be done once you're confident the modulefile is working properly. To do so run either:
smithy module deploy last
smithy module deploy subversion/1.7.8/sles11.1_gnu4.3.4
This section shows the typical work-flow for adding a new software build using build scripts. Let's imagine we want to install petsc 3.2 using the cray compiler. We must create a place for the software build, build it, and create a module.
Let's use petsc/3.2/cle4.0_cray8.0.1 as the name for our new package. See
NAMING SOFTWARE BUILDS for details on naming a software build. This would
be petsc designed to run on a cle4.0 compute node and compiled with the cray cce
8.0.1 compiler. This can be accomplished by running:
smithy new petsc/3.2/cle4.0_cray8.0.1
You can save yourself some extra time by telling smithy where the source tar file for petsc is:
smithy new -t petsc-3.2-p7.tar.gz petsc/3.2/cle4.0_cray8.0.1
When using the -t, --tarfile= option smithy will extract the given archive
to the source directory. For the petsc example above this would be
/sw/xk6/petsc/3.2/cle4.0_cray8.0.1/source
The -t option can also download an archive from a given URL. The archive is
saved along side the source directory. As an example:
smithy new -t http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.2-p7.tar.gz petsc/3.2/cle4.0_cray8.0.1
This command will download petsc-3.2-p7.tar.gz, save it to
/sw/xk6/petsc/3.2/cle4.0_cray8.0.1/petsc-3.2-p7.tar.gz and extract it to
/sw/xk6/petsc/3.2/cle4.0_cray8.0.1/source. This feature requires the curl
command to work properly.
Additionally, if this is a brand new piece of software add the
--web-description switch. This will create the application description files
too.
Once you have created the build you may need to update the build (rebuild) and
environment (remodule) scripts before building the software. Both files live
within the software prefix. For our example it is located in
/sw/xk6/petsc/3.2/cle4.0_cray8.0.1. You can edit this and other related files
using the edit command:
smithy edit build petsc/3.2/cle4.0_cray8.0.1
This is just a convenience command to editing the rebuild script. It will expect
the $EDITOR environment variable to be set to the edior of your choice. You
can alternatively use the -e,--editor= option. To make things quicker you
can run:
smithy edit last
By omitting the build argument, smithy will prompt you to select the file to
edit. The last argument will tell smithy to use the last software build it
worked with. If you are not sure what the last one is you can run smithy show
last to see.
Once you have edited the files, it's time to build. Do so by running:
smithy build last
The results of the run will be shown on the screen and simultaneously logged to
build.log withing the software prefix folder.
This step is best done after all builds for a particular application have been
created. When you create a new software build a modulefile is created too. For
our petsc install it lives in: /sw/xk6/petsc/3.2/modulefile All builds of a
particular application share a single modulefile.
Smithy does some guessing as to what should be inside the modulefile based on what builds you have. If there is only one build the modulefile is relatively simple. If you have multiple builds for different compilers then it will be more complicated.
To be sure the modulefile is up to date on each build, regenerate it with:
smithy module create last
Launch your editor with:
smithy edit modulefile last
Add any relevant variables to your application. Next you will need to test it to
make sure it is loading properly for each build and that there are no errors.
You can add the modulefile to your $MODULEPATH using:
smithy module use last
Now you are free to display and load it using the normal module commands:
module avail petsc
module display petsc/3.2
module load petsc/3.2
Once the modulefile is ready, you can copy it into place using:
smithy module deploy last
For out petsc example, this command will copy
/sw/xk6/petsc/3.2/modulefile/petsc/3.2 to /sw/xk6/modulefiles/petsc/3.2
If this is a new application you will need to add some information to the
description file. For petsc this lives in: /sw/xk6/petsc/description This is
an html formatted file. Alternatively, it can live in
/sw/xk6/petsc/description.markdown this file is in markdown format and is a
bit simpler to write than html. See http://kramdown.rubyforge.org/quickref.html
for more information on markdown syntax. If both files exist, the markdown file
takes precedence.
If the description file is missing you can generate one by running:
smithy repair last
To publish to the website run:
smithy publish petsc
To see an up to date example of what a smithy config file should look like run
smithy show example_config. Most of the options specified in a config file can
be overridden by global command line options. See smithy help for info on
global command line options. The smithy config file uses the
yaml syntax and looks something like this:
---
software-root: /sw
download-cache: /sw/sources
formula-directories:
- /sw/tools/smithy/formulas
- /sw/tools/smithy/another_formula_directory
global-error-log: /sw/tools/smithy/exceptions.log
file-group-name: ccsstaff
descriptions-root: /sw/descriptions
web-root: /sw/descriptions_in_html
web-architecture-names:
xk6: titan
xk7: titan
analysis-x64: lens
smoky: smoky
hostname-architectures:
everest-login: redhat6
everest: redhat6
lens: analysis-x64
sith-login: redhat6
sith: redhat6
smoky-login: smoky
titan-login: xk6
titan-ext: xk6
yona-login: yona
yona: yona
This option defines the top level directory used as the location of architecture
directories. Good choices might be /opt/smithy or /usr/local/smithy.
Someplace under your home directory would work too.
Software compiled with smithy is intended to be group writable so that more that one user may install software in the same location. This option defines the group name used for installing software.
This option defines what hostnames map to which architecture directories. Smithy will use the mapped directories whenever it is run on machines with hostnames specified by this option. Smithy will ignore trailing digits on hostnames. For example:
hostname-architectures:
everest: redhat6
titan-login: xk6
titan-ext: xk6
yona-login: yona
yona: yona
Would produce the following matches:
This directory is used when software archives are downloaded via formulas or
through the smithy new --tarfile= option.
This option specifies one or more directories where formulas are stored. If multiple directories are used then smithy will use the first formula it finds in the order directories are specified.
This file will log internal errors smithy encounters. It is usually used for
development. You can also get extra error output by setting GLI_DEBUG=true in
your shell.
The smithy publish command outputs architecture names at the top of
description files. If you wish to use a different name this option allows for
that. For example:
web-architecture-names:
xk6: titan
xk7: titan
analysis-x64: lens
This directory is the target output directory when running smithy publish
Setting this option changes the way smithy handles description files. When NOT
set smithy will expect descriptions to live inside each application folder
inside each architecture folder. For example, assuming a software-root of /sw
and an architecture of xk6 smithy would expect the description files in these
locations:
/sw
|--- redhat6
| |--- subversion
| | |--- 1.6.17
| | |--- 1.7.8
| | `--- description.markdown
| `--- git
| |--- 1.7.9.5
| |--- 1.8.2.1
| `--- description.markdown
`--- xk6
|--- subversion
| |--- 1.6.17
| |--- 1.7.8
| `--- description.markdown
`--- git
|--- 1.7.9.5
|--- 1.8.2.1
`--- description.markdown
This setup has multiple description files per application. This might make sense
if your documentation is highly dependent on the architecture. If we set
descriptions-root to /sw/descriptions smithy would expect the descriptions
in:
/sw
|--- redhat6
| |--- subversion
| | |--- 1.6.17
| | `--- 1.7.8
| `--- git
| |--- 1.7.9.5
| `--- 1.8.2.1
|--- xk6
| |--- subversion
| | |--- 1.6.17
| | `--- 1.7.8
| `--- git
| |--- 1.7.9.5
| `--- 1.8.2.1
`--- descriptions
|--- subversion
| `--- description.markdown
`--- git
`--- description.markdown
With this we have only one description file per application.
This option allows you to override the complier definition for creating modulefiles. The syntax is yaml and the default values are:
compilers:
- :prg_env: PrgEnv-gnu
:compiler_name: gcc
:human_name: gnu
:regex: !ruby/regexp /(gnu|gcc)(.*)/
:build_name_regex: !ruby/regexp /(gnu|gcc)([\d\.]+)/
- :prg_env: PrgEnv-pgi
:compiler_name: pgi
:human_name: pgi
:regex: !ruby/regexp /(pgi)(.*)/
:build_name_regex: !ruby/regexp /(pgi)([\d\.]+)/
- :prg_env: PrgEnv-intel
:compiler_name: intel
:human_name: intel
:regex: !ruby/regexp /(intel)(.*)/
:build_name_regex: !ruby/regexp /(intel)([\d\.]+)/
- :prg_env: PrgEnv-cray
:compiler_name: cce
:human_name: cray
:regex: !ruby/regexp /(cce|cray)(.*)/
:build_name_regex: !ruby/regexp /(cce|cray)([\d\.]+)/
smithyformula(5)