PMF-AMS Analysis Guide

Introduction

The PMF Evaluation Panel consists of 3 Igor procedure files (ipfs) called PMF_Execution, PMF_ViewResults, and PMF_Scatter. This wiki serves as the help and documentation for the software. To run PMF with the panel, the PMF executable and associated files, accessed separately, are required (see Section 3, Installing PMF with Igor). The PMF executable is compiled only for Windows/DOS. It is possible to execute PMF on a Windows computer and then analyze the experiment on a Macintosh.

The ipfs were written by Ingrid Ulbrich and Donna Sueper (Jimenez Group, University of Colorado, Boulder) and Greg Brinkman (Hannigan Group, University of Colorado, Boulder). Questions about this codecan be addressed to Ingrid or Donna at ulbrich@colorado.edu or dsueper@colorado.edu.

PMF (Positive Matrix Factorization) was developed by Dr. P. Paatero (Dept. of Physics, University of Helsinki). One of the original papers describing this method is Paatero, 1997 P. Paatero, Least squares formulation of robust non-negative factor analysis, Chemometrics and Intelligent Laboratory Systems 37 (1997), pp. 23–35. First time users to PMF are encouraged to first read the documentation by Paatero regarding PMF (see Section 9, Other Resources).

This Igor toolkit was intended for use in analyzing AMS data, but there are only few assumptions in the toolkit relating to AMS-type data. Non-AMS users of this software can skip Section 4, Creating the Organics and Error Matrices.

A Message to Contributors

We want to encourage active participation by all users in the evolution of the information contained within this wiki and welcome the addition of content that is beneficial to the community as a whole. However, please DO NOT delete any content from this page!! Significant time, effort, and deliberation has gone into the information contained in this page. Rather than deleting content, please feel free to voice your concerns by posting a comment to the discussion page where others can contribute (please be sure to include a topic to be referenced in responses).

Installing PMF with Igor

Setting up PMF on Your Computer

1. Download the BareBones PMF Starter Kit
2. Create a new folder on your computer where you'll store the PMF files and all files output by PMF (this should NOT be the folder where you store your Igor experiments). This folder must contain:
   • PMF2wtst.exe (obtained from P. Paatero or the BareBones Starter Kit)
   • imupmf.ini (obtained from the BareBones Starter Kit)
   • pmf2key.key (obtained from P. Pattero, U. Helsinki)
3. Start a new Igor experiment and load the following files from the BareBones Starter Kit:
   • DataAndErrorForBareBonesPMF.itx
   • BareBonesPMFExecution_1_00B.ipf

Running the PMF Test Case

1. Start the Panel from the BareBonesPMF menu on the menu bar.
   File:BareBonesPanel.png
   
2. Use the first button and select the path to the folder you created with the PMF2wtst.exe file.
3. Use the second button to execute PMF.

You should see a black DOS window pop up, scroll a lot out output, and then close. Igor will then tell you, "The PMF barebones analysis was successfully completed within X seconds."

X should be > 0!

If the window goes away immediately and the Igor message says that the analysis was completed within 0 seconds, the execution was not successful. Try the following steps to solve the problem.

If PMF does not Run Properly

1. Look in the folder you created with the PMF2wtst.exe file for the existence of the files

 Matrix.dat and StdDev.dat

If these files exist, Igor was able to access the correct folder. Continue with Step 2.

If these files do not exist, Igor was not able to access the correct folder. Go back to the first button on the panel and check that you've given the correct path to the folder with the PMF2wtst.exe file. Run PMF again by pressing the second button.

2. Look in the folder you created with the PMF2wtst.exe file for the existence of the file

 PMF2.LOG

If this file does not exist, PMF was not run in this folder. Go to step 3.

If this file does exist, PMF attempted to run in this folder. Open the file PMF2.LOG (it is a text file).

Glance down the contents of the file and look for many lines of sequential numbered output, such as

 1 rank1 step chi2=   9282.6     Penalty=  1.5287E+04 Flags GF
 2 rank1 step chi2=   7411.7     Penalty=  1.4084E+04 Flags GF

If these lines are in the file, PMF ran successfully on your computer, which must be fast enough to run this data in less than 1 second. You're done!

If the lines of sequential numerical output are not in the file, you should find the following lines within this file (note that every line is NOT included here, but the lines selected here are in the order they appear in the file):

2a)

 ##PMF2 .ini file for: IMUPMF.INI  --- BareBonesPMF
 Successfully read task initialization file imupmf.ini
 titled:  ##PMF2 .ini file for: IMUPMF.INI  --- BareBonesPMF

If these lines appear in the file, PMF found the .ini file. Continue with Step 2b.

If these lines do not appear, you should see a message about not finding an appropriate .ini file.

• Make sure that this folder contains the file imupmf.ini, provided with the BareBones PMF Starter Kit. If it did not, copy the file to this folder, delete the file PMF2.LOG and press the second button in the panel again to see whether PMF runs successfully.

If the file imupmf.ini already exists in the same folder as the PMF2wtst.exe file, continue with Step 3.

2b)

 Successfully opened input file      30
 with name MATRIX.DAT
 Successfully opened input file      31
 with name STD_DEV.DAT

If these lines appear in the file, everything should have run correctly. Look at the rest of the PMF2.LOG file to see whether other errors are reported. If you still encounter difficulty, contact Ingrid Ulbrich at <Ingrid dot Ulbrich at colorado dot edu> for assistance and attach the PMF2.LOG file to your email.

If these lines do not appear in the file, you will see a message about PMF not being able to access one of these files. Check that the files are not being used by other programs, delete the file PMF2.LOG, and press the second button on the panel again to see whether PMF runs successfully.

3. Look in the C:\ directory of your computer for the existence of the file

 runpmf.bat

If this file exists, Igor was able to write to your C:\ drive. Continue with Step 3a.

If this file does not appear, Igor was not able to write this file to your C:\ drive. This might be due to high security settings on your computer. You should create a text file with this name (NOT runpmf.bat.txt) and choose to edit it (not open it) with a text editor (e.g., Notepad, WordPad, Emacs, etc.). The file should contain the lines

 cd C:\Documents and Settings\Ingrid Ulbrich\Desktop\
 pmf2wtst imupmf  

NOTE that you must change the path in this example to the path to the folder where you have put the file PMF2wtst.exe!

3a) Execute runpmf.bat by double-clicking on its icon. You should see the black DOS window pop up, scroll output, and close again.

If this happens, PMF has run successfully from the batch. In the Igor experiment, press the second button on the panel to see whether PMF runs successfully.

If the black window does not show scrolled output, continue with Step 3b.

3b) You now need to execute PMF from the command line.

To access the DOS command line, launch the Command Prompt from your Windows Start menu, located in

 Start -> Programs -> Accessories -> Command Prompt

Change from the path displayed in the prompt to the folder where you put the PMF2wtst.exe file. Use the command

 cd

to change directories (e.g.,

 cd Desktop\PMF

). You can change one directory at a time, or several at a time as shown in the example. To move up 1 directory, use

 cd ..	

Be sure that the folder contains the three files

 PMF2wtst.exe (obtained from Paatero or the BareBones Starter Kit)
 imupmf.ini  (obtained from the BaseBones Starter K
 pmf2key.key (obtained from P. Pattero, U. Helsinki)

by listing the contents of the current directory, using the command

 dir	

Then type this command at the prompt to run PMF:

 pmf2wtst imupmf

You should see scrolling output in the command window. The window will not disappear and you can scroll back through the output (which is also saved in the file PMF2.LOG).

If the output contains many lines of sequential numbered output, such as

 1 rank1 step chi2=   9282.6     Penalty=  1.5287E+04 Flags GF
 2 rank1 step chi2=   7411.7     Penalty=  1.4084E+04 Flags GF

PMF ran successfully here. Delete the file PMF2.LOG and in the Igor experiment, press the second button on the panel to whether PMF runs successfully from Igor.

If the output does not contain these lines, go back to Step 2 of this section to examine errors that might be reported by PMF in the file PMF2.LOG.

Creating the Organics and Error Matrices (Step 0)

In the Q-AMS Software (James')

Be sure to use v1.41 or later.

spikes

smoothing

In SQUIRREL

• There is currently not a 'one-button' piece of code to extract the matrices in Squirrel for use in PMF analysis. Several parts of the code exist, so you won't have to do any coding yourself.

• Here is the outline:

• (1) Create and export the organic matrix, which is a 2-d matrix, rows = runs and cols = m/z (remember that Igor counts from 0 so column 0 is m/z 1). You can do this in the squirrel panel, ms tab, in the ms average spectra tab. Choose the todo wave you want, enter Org into the species name, and press the export matrices button.

• (2) Create the error matrix, which should be of the same dimensions as the org matrix. Go to the corrections-error tab, select MS errors and press the corrections button. Make sure you are *not* doing other corrections, like the airbeam correction without meaning to. This error matrix will reflect the errors inherent in the calculation of the sticks, and is independent of any species like organics or nitrate.

• (3) Use the function Make_Org_err() below to multiply the frag Org matrix by the error matrix to get the Org_error matrix. This simply does the matrix multiplication of the error matrix with the 'frag organic' matrix. This 'frag_organic' matrix is generated when converting the frag_org text wave into a numerical 2-d representation of the text wave frag_organic. The result is a 2 d matrix (rows = runs, columns = m/z). Note that if you are using a time-dependency in your organic frag wave (such as when you have an external gas phase CO2 measurement that has been mapped onto the AMS time wave), this algorithm won't work because the frag_org matrix changes at each point in time.

  Function Make_Org_err()
    wave MSSDiff_p_err
    wave Org_mat=root:ms_mats:Org_mat
    redimension /n=(dimsize(mssdiff_p_err,0),dimsize(mssdiff_p_err,1),1) MSSDiff_p_err
    imagetransform /g=3 transposevol MSSDiff_p_err
    wave m_volumetranspose
    matrixop /o mass_tmp=m_volumetranspose x Org_mat
    imagetransform /g=3 transposevol mass_tmp
    matrixop /o Org_err=m_volumetranspose[][][0]
    killwaves/z m_volumetranspose, mass_tmp
  End

• (4) Now you have the basic pieces - the Org matrix and the Org error matrix. There is still more work that needs to be done: removing columns with zeros/nans, and perhaps doing some editing on the error matrix.

Deleting NaNs/zeros

In the PMF_Execution_XX.ipf, use the function

pmf_ams_deleteNaNs_mxWvs(dataMx, errMx, rowDescrWv, colDescrWv)

where

dataMx and errMx are your data and error matrices,

rowDescrWv is a 1-D wave that gives the indices for the rows (usually t_series), and

colDescrWv is a 1-D wave that gives the indices for the columns (usually amus).

The function chooses the first row and column of the data that contain a mix of NaNs and values and uses this as a template to delete rows or columns (respectively) that contain NaNs. Note that if a data value in the first row was 0 and you have replaced zeros by NaNs, a column may be deleted inadvertently!

The function creates new versions of the input waves with names noNaNs_amus, noNaNs_t_series, etc. from which the NaN rows and/or columns have been deleted. Global strings called NaNsList_amus and NaNsList_tseries have been created and are used by related functions to delete items from other waves or reinsert the original NaNs into waves.

Related Functions

Also found in PMF_Execution_XX.ipf:

• pmf_ams_deleteNaNs_Wvs
• pmf_ams_insertNaNs_Mxs
• pmf_ams_insertNaNs_Wvs

Perform PMF Analysis *Step 1*

Additional File for PMF Executable Directory

Be sure that the file

 mypmft.ini

(provided on the PMF data page) is located in the directory with the PMF Executable.

Recommended Practice: Organizing Your Folders

 + root
    + TemplateData          to copy for new versions of analysis
    + Variation1            e.g., your basis case
    + Variation2            e.g., with different error estimates
    + External_MassSpectra  for use with the Scatter Panel
    + External_Tseries      for use with the Scatter Panel

NOTE: Data for running PMF can be in root: or a subfolder of root: , but not any lower folder.

The Executable Panel

• set path to the PMF executable
• Choose dataMatrix and Error Matrix (use noNaNs_ versions)
• Choose model error
• Choose a range for number of factors. To make sure that everything runs properly, you may want to run just one case (Min p = 2, Max p = 2). We recommend running 1 factor to have a context for the meaning of the 2-factor solution.

Choose FPEAK or SEED values

• "FPEAK" is a tool used to explore rotations of the solutions of a given number of factors. Note that FPEAK does not explore all possible rotations of a solution. FPEAK = 0 does not apply any rotational forcing. Non-zero values of FPEAK create near-zero values in the factor profiles (mass spectra) or time series. More information about FPEAK can be found in the PMF Users Manual Part 1 (pp. 9,12,14,21) and Part 2 (p. 24), and in several papers by P. Paatero (see [Additional Resources]).

A good first set of FPEAK values is -1.0 to +1.0 with a delta value of 0.1 or 0.2. For a full analysis, a wide enough range of FPEAKs to achieve Q/Qexp of at least 1% above the minimum value is recommended.

• "SEED" is a tool used to choose different random starts (initial values) for the PMF algorithm. Using different seeds may lead to solutions in different local minima (Q/Qexp) in the solution space. One set of solutions may have more physical meaning than another, or multiple sets may make physical sense. It is impossible to test all start values, but testing many seeds may give an indication of local minima for your dataset. More information about seeds can be found in the PMF Users Manual Part 1 (p. 11) and Part 2 (p. 16).

Run seeds from 0 to your preferred maximum with a delta value of 1.

What the Software Does When You Press the Button...

The basic version

The software will execute PMF once for every combination of number of factors and FPEAK/seed. So if you run 1-5 factors and 5 FPEAKs, PMF will run 5x5=25 times. Each run starts a new black DOS window that will close when the run is completed. The duration of each run is printed in the history at the end of each run. In general, runs which solve for more factors and runs with FPEAK farther from 0 take longer. The code runs all of the FPEAKS or seeds at for one number of factors, then advances to the next number of factors (e.g., run 1 factor with each of 5 FPEAK values, then 2 factors with each of 5 FPEAK values, etc.).

When each DOS window pops up, it "grabs the focus" and makes itself the top window. This makes it hard to use the computer for anything else. You may want to run these analyses when you're going to way away from the computer for several hours or on a dedicated analysis computer.

A little more detail

The software writes the files

C:\delete_log.bat    C:\runPMF.bat

and writes your DataMatrix and ErrorMatrix as MATRIX.DAT and STD_DEV.DAT, respectively to the folder with your PMF Executable. The software also writes a file to that folder called STD_DEV_PROP.DAT, which has the same number of points as the DataMatrix and in which every element is equal to the ModelError.

The software then enters a pair of nested loops in which the following steps occur:

• for each number of factors
  • for each FPEAK or SEED
    • use the file

 mypmft.ini

as a template to create the file

 imupmf.ini

which is used as the control file for PMF.

• • • Delete the old PMF2.LOG file by running delete_log.bat
    • Execute PMF by running run_PMF.bat.
    • Wait for PMF to complete its run.
    • Load PMF output (including log file and factors)

At the completion of the loops, the software calculates some statistics from the output and then creates a panel to select data for viewing.

Other Resources

• Paateto documentation, ftp? Hopke ftp?
• Users Meeting presentation
• Software page
• Datasets page