Difference between revisions of "PMF-AMS Analysis Guide"
(→PMF Evaluation Tool Software) |
(→View PMF Analysis Results *Step 2*) |
||
(267 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
A shortcut to this page is [http://tinyurl.com/PMF-guide http://tinyurl.com/PMF-guide] | A shortcut to this page is [http://tinyurl.com/PMF-guide http://tinyurl.com/PMF-guide] | ||
− | The PMF Evaluation Tool (PET) was described in [http://cires.colorado.edu/jimenez/ams-papers.html#Database_PMFTool Ulbrich et al., ACP 2009]. Please cite the tool with this work in publications in which you have used the PET. The PET consists of several [http://www.wavemetrics.com Igor] procedure files (ipfs). This wiki serves as the help and documentation for the software. To run PMF with the panel, the PMF executable and associated files, accessed separately, are required (see Section | + | The PMF Evaluation Tool (PET) was described in [http://cires.colorado.edu/jimenez/ams-papers.html#Database_PMFTool Ulbrich et al., ACP 2009]. Please cite the tool with this work in publications in which you have used the PET. The PET consists of several [http://www.wavemetrics.com Igor] procedure files (ipfs). This wiki serves as the help and documentation for the software. To run PMF with the panel, the PMF executable and associated files, accessed separately, are required (see Section 2, [[#Installing PMF with Igor| Installing PMF with Igor]]). |
− | The ipfs were written by Ingrid Ulbrich and Donna Sueper (Jimenez Group, University of Colorado, Boulder) and Greg Brinkman (Hannigan Group, University of Colorado, Boulder). Questions about this code can be addressed to [mailto: | + | The ipfs were written by Ingrid Ulbrich (formerly of the Jimenez Group, University of Colorado, Boulder) and Donna Sueper (Aerodyne Research and Jimenez Group, University of Colorado, Boulder) and Greg Brinkman (Hannigan Group, University of Colorado, Boulder). Questions about this code can be addressed to [mailto:support@aerodyne.com?Subject=Igor-PMF%20Question Donna]. |
− | PMF (Positive Matrix Factorization) was developed by Dr. P. Paatero (Dept. of Physics, University of Helsinki). Links to Paatero's PMF documentation | + | PMF (Positive Matrix Factorization) was developed by Dr. P. Paatero (retired, Dept. of Physics, University of Helsinki). Links to Paatero's PMF documentation and many PMF method papers can be found in Section 9, [[#Other Resources |Other Resources]]. |
− | This Igor toolkit was intended for use in analyzing AMS data, but there are only few assumptions in the toolkit relating to AMS-type data. Some information on ways to create the necessary waves and matrices from non-AMS data are found in | + | Originally, the PMF executable and license files could be purchased by sending an email to Paatero. Now one can freely download the pmf executable files below, but a license file is needed for it to run. Please email [mailto:support@aerodyne.com?Subject=PMF%20Operating%20System%20Issue Donna] for credentials to download the executables. The Swiss company Datalystica is now the sole official seller of the multi-linear engine (ME-2) solver package that contains a license that is used for the PMF executable. Once the ME-2 package is downloaded, copy and rename the ME2key.key file to pmf2key.key and place this in the same folder as the PMF executable files. Please go to [http://datalystica.com/me-2-solver/ https://datalystica.com/me-2-solver/] to purchase ME-2. One license file can be used for all in the same research group. |
+ | |||
+ | This Igor toolkit was intended for use in analyzing AMS data, but there are only few assumptions in the toolkit relating to AMS-type data. Some information on ways to create the necessary waves and matrices from non-AMS data are found in Section 3.2.4 below. | ||
== A Message to Contributors == | == A Message to Contributors == | ||
Line 14: | Line 16: | ||
== Installing PMF with Igor == | == Installing PMF with Igor == | ||
+ | |||
+ | <strong>This section has been modified with new information on 23 October 2024 by [mailto:support@aerodyne.com?Subject=PMF%20Operating%20System%20Issue Donna]</strong> | ||
=== PMF and Operating Systems === | === PMF and Operating Systems === | ||
− | The PMF executable is compiled only for Windows/DOS. | + | The PMF executable is compiled only for Windows/DOS; it has been used with Windows 11, 10 and older operating systems Windows XP and Windows Vista. Using OneDrive to access the PMF executable file within PET has been problematic for some users. If possible, it is recommended that users access the PMF executable and the Igor template experiment PET from a local drive. |
+ | |||
+ | For users with Macs, two options have proven successful. One method is to execute PMF on a Windows computer and then analyze the experiment on a Macintosh. The other method is to execute PMF under a Windows emulator on a Mac. Note that in this latter case, you may need to have the PMF executable somewhere in the c:\ directory; this isn't necessary when running Windows on a PC. | ||
+ | |||
+ | === Setting up PMF and PET on Your Computer the First Time === | ||
+ | |||
+ | ==== Download the PMF executable package ==== | ||
+ | The file pmf2wopt.exe is the PMF solver. It is downloadable via the PMF executable package below for free; however a username and password is required. Email [mailto:sueper@colorado.edu?Subject=PMF%20Operating%20System%20Issue Donna] for these credentials. An older executable file named pmf2wtst.exe also comes with this package but is not typically used. The PMF executable package will also contain a initialization/configuration file named mypmft.ini. This pmf initialization file contains parameters that is used by PET but is not explicitly needed for the executable to run. For every call to the PMF executable PET will generate a 'working' initialization file based on mypmft.ini. This file, named imupmf.ini, is specific to the PET package. | ||
+ | |||
+ | PMF executable file step: | ||
+ | |||
+ | Step 1: [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/PMF_executable_package.zip Download the zipped PMF executable package] and save locally on the computer from where it will run. | ||
+ | |||
+ | ==== Purchase a group PMF license ==== | ||
+ | Along with the PMF executable file a user will need a license file named pmf2key.key. This pmf2key.key file is checked when the PMF executable begins and ensures that a license is valid. If this key file is missing the PMF executable will abort without any analysis taking place. The license, or key file, is purchased through [https://datalystica.com/me-2-solver/ Datalystica], a Swiss company that obtained the rights from the originator of the PMF code, Penti Paatero (University of Helsinki, retired). For PMF usage, one need not purchase SoFi from Datalystica, only ME-2 (multi-linear engine 2), a linear solver for which PMF is a part. This ME-2 package will contain not only executable files such as me2gf*.exe, but will come with a key file named me2key.key. This same key file for ME-2 executable files will also work for PMF executable files, but it must be copied and renamed pmf2key.key and placed in the same folder as the PMF executable files. Every single end user within an organization is not required to obtain a license; users within the same research group can share a license. In Paatero's words "If you drink coffee together in a real room, you are a group." Datalystica does not support PET and PMF but Datalystica employees may be able to answer questions regarding ME-2. The me2key.key and pmf2key.key license file is small and will likely have characters #PMF2KEY#PMF3KEY#ME-2KEY#3127#MSDOS somewhere in the first line. | ||
+ | |||
+ | PMF license file steps: | ||
+ | |||
+ | Step 1: Purchase and download ME-2 solver from [https://datalystica.com/me-2-solver/ Datalystica] | ||
+ | |||
+ | Step 2: Copy the file named me2key.key into the PMF executable folder and rename as pmf2key.key | ||
+ | |||
+ | ==== Download PET Igor template experiment or zipped file of ipfs (ipf = Igor Procedure File) ==== | ||
+ | The Igor interface for viewing and evaluating PMF solutions is called PET (PMF Evaluation Tool). | ||
+ | |||
+ | PET step: | ||
+ | Step 1: [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/IgorPMF_Template_v3_08E.pxt Download the PET Igor template file (IgorPMF_Template_v*.pxt)] and save locally on the computer. It is not recommended that the Igor template experiment reside in the same folder as the PMF executable package. | ||
+ | |||
+ | From the top most Igor menu there is a section named "PMF". From this menu there are these options: | ||
+ | *"Perform PMF analysis *Step 1*" | ||
+ | *"View PMF analysis results *Step 2*" | ||
+ | *"Small Pop-Plots Panel for *Step 2*", | ||
+ | *"Compare PMF results with external factors *Step 3*" | ||
+ | |||
+ | Selecting "Perform PMF analysis" will bring up a panel from which you can preform data preparation steps and select data and conditions for running PMF. | ||
+ | |||
+ | Selecting "View PMF analysis results" will bring up a panel from which you can select waves that were previously input into PMF. Once choices have been selected the large PET panel with multiple graphs showing results and diagnostics will appear. Within one PET experiment, it is possible to run PMF either on multiple data sets or under different conditions. This panel allows one to select the PMF results from possible different data sets or conditions. | ||
+ | |||
+ | Selecting "Small Pop-Plots Panel" will generate a mini version of the big PMF results panel. This option was initially intended for users with small screen sizes. | ||
+ | |||
+ | Selecting "Compare PMF results with external factors *Step 3*" will generate a panel from which one can compare PMF results with other data that was not included in the PMF analysis. For example, if one had some gas phase or meteorological measurements with the same time stamps as the input data, this option will show a panel of correlation plots of PMF factors with this 'external' data. | ||
− | + | ==== Set up options ==== | |
− | === | + | It is recommended that users keep the PET Igor template experiment either within a Wavemetrics folder/subfolder or adjacent to the PMF executable package folder. Once PET data preparation steps have been completed and the gold "Being PMF..." button is pushed, the user will be prompted to save the template experiment with a unique name. |
+ | |||
+ | One may have many copies of the PMF executable packages on one computer. A file organization suggestion is to set up multiple copies like so: C:\Users\*\Documents\PMF\PMF Executable1, C:\Users\*\Documents\PMF\PMF Executable2 , etc, each folder with a copy of the pmf2key.key file. One can have multiple versions of Igor open at once; this allows one to run multiple copies of PMF (from different Igor experiments) at the same time. | ||
+ | |||
+ | ===== Upgrade PET version options ===== | ||
+ | One can upgrade versions of the PET software without having to redo any PMF analysis steps. Once can download the latest PET ipfs (Igor Procedure Files) below and load the latest versions of the ipfs into an existing PET experiment and then kill all the old ipfs. You may want to kill and recreate PET panels such as the "View Results" and "Setup" panels to take advantage of new features. | ||
+ | <!-- | ||
(The BareBones version is currently out of date. You can skip the BareBones version and install the full version of PMF.) | (The BareBones version is currently out of date. You can skip the BareBones version and install the full version of PMF.) | ||
− | + | It is recommended that you use the BareBones version to make sure that PMF and Igor interact correctly on your computer. | |
#Download the [[#BareBones Starter Kit | BareBones Starter Kit]] | #Download the [[#BareBones Starter Kit | BareBones Starter Kit]] | ||
#Create a new folder on your computer where you'll store the PMF files and all files output by PMF (this should NOT be the folder where you store your Igor experiments). This folder must contain: | #Create a new folder on your computer where you'll store the PMF files and all files output by PMF (this should NOT be the folder where you store your Igor experiments). This folder must contain: | ||
Line 45: | Line 95: | ||
''If the window goes away immediately'' and the Igor message says that the analysis was completed within 0 seconds, the execution was not successful. Follow the steps for what to do [[#If PMF does not Run Properly| If PMF does not Run Properly]] to correct the problem. | ''If the window goes away immediately'' and the Igor message says that the analysis was completed within 0 seconds, the execution was not successful. Follow the steps for what to do [[#If PMF does not Run Properly| If PMF does not Run Properly]] to correct the problem. | ||
− | + | ||
+ | |||
=== Setting Up the Full Version === | === Setting Up the Full Version === | ||
+ | |||
After you have confirmed that PMF and Igor interact correctly on your computer with the BareBones version, you are ready to use the full version for analysis. | After you have confirmed that PMF and Igor interact correctly on your computer with the BareBones version, you are ready to use the full version for analysis. | ||
# Download the .ipf (or .pxt) files from [[#PMF_Evaluation_Tool_Software | the software section of this wiki]]. | # Download the .ipf (or .pxt) files from [[#PMF_Evaluation_Tool_Software | the software section of this wiki]]. | ||
Line 65: | Line 117: | ||
Secondly, when you runPMF with your selected choices (the gold 'Begin PMF analysis with these choices' button), the code perform some calculations on the PMF results that have been loaded into Igor. An error message "StatsRankCorrelation Test error; Does not support Nan or INF" may appear. It is safe to ignore this error and continue. Press the green right-pointing arrow or the Continue button. | Secondly, when you runPMF with your selected choices (the gold 'Begin PMF analysis with these choices' button), the code perform some calculations on the PMF results that have been loaded into Igor. An error message "StatsRankCorrelation Test error; Does not support Nan or INF" may appear. It is safe to ignore this error and continue. Press the green right-pointing arrow or the Continue button. | ||
+ | |||
+ | It is recommended that one puts all the PMF executable-related files (*.exe and *.ini and *.key etc) in one folder, perhaps named 'PMF_executable', for which you never add or remove other files (except those done automatically by PMF or PET). This PMF_executable folder does not have to be in the 'Program' folder on windows machines; in can reside in within a user's 'Documents' folder. It also should not reside in the Wavemetrics folder or any Igor Pro folder. The location of the Igor template experiment should also not be within the PMF_executable folder. | ||
==== Important Note about PMF2___.exe Files ==== | ==== Important Note about PMF2___.exe Files ==== | ||
Line 76: | Line 130: | ||
//strconstant PMF_EXEC_FILE_NAME="pmf2wopt.exe" //imu2.03 NOTE that if you use this line, you need to upgrade your .ini file to v2.03!! | //strconstant PMF_EXEC_FILE_NAME="pmf2wopt.exe" //imu2.03 NOTE that if you use this line, you need to upgrade your .ini file to v2.03!! | ||
− | + | --> | |
− | + | ====If PMF does not Run Properly within PET==== | |
+ | |||
+ | 0. Make sure that the path to the executable is correct. If red text appears in the section 'PMF Executable File Path' then something is wrong. Look in the folder you created with the PMF2wtst.exe or pmf2wopt.exe file. If necessary go to the tab entitled "PMF defaults and advanced options' and reselect the executable file; then go to the run tab, 'PMF Executable File Path' and reselect the folder with your chosen .exe file. | ||
+ | |||
+ | 1. Look in the PMF executable folder that contains pmf2wopt.exe (and/or pmf2wtst.exe) and check for the existence of the files | ||
Matrix.dat and StdDev.dat | Matrix.dat and StdDev.dat | ||
− | :If these files exist, Igor was able to access the | + | :If these files exist, Igor was able to access the pmf executable and key file. Continue with Step 2. |
− | :If these files do not exist, Igor was not able to access the correct folder. | + | :If these files do not exist, Igor was not able to access the correct folder. Check the existence and spelling of the license file pmf2key.key |
− | 2. Look in the folder | + | 2. Look in the PMF executable folder that contains PMF2*.exe for the existence of the file |
PMF2.LOG | PMF2.LOG | ||
Line 97: | Line 155: | ||
2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF | 2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF | ||
− | : If these lines are in the file, PMF ran successfully on your computer | + | : If these lines are in the file, PMF ran successfully on your computer. |
:If the lines of sequential numerical output are not in the file, you should find the following lines within this file (note that every line is NOT included here, but the lines selected here are in the order they appear in the file): | :If the lines of sequential numerical output are not in the file, you should find the following lines within this file (note that every line is NOT included here, but the lines selected here are in the order they appear in the file): | ||
Line 103: | Line 161: | ||
2a) | 2a) | ||
− | ##PMF2 .ini file for: IMUPMF.INI --- | + | ##PMF2 .ini file for: IMUPMF.INI --- PMF Evaluation Panel v2.3 |
Successfully read task initialization file imupmf.ini | Successfully read task initialization file imupmf.ini | ||
− | titled: ##PMF2 .ini file for: IMUPMF.INI --- | + | titled: ##PMF2 .ini file for: IMUPMF.INI --- PMF Evaluation Panel v2.3 |
:If these lines appear in the file, PMF found the .ini file. Continue with Step 2b. | :If these lines appear in the file, PMF found the .ini file. Continue with Step 2b. | ||
Line 111: | Line 169: | ||
:If these lines do not appear, you should see a message about not finding an appropriate .ini file. | :If these lines do not appear, you should see a message about not finding an appropriate .ini file. | ||
− | :*Make sure that this folder contains the file imupmf.ini, provided with the | + | :*Make sure that this folder contains the file imupmf.ini, provided with the PMF executable package. |
− | :If the file imupmf.ini already exists in the same folder as the | + | :If the file imupmf.ini already exists in the same folder as the PMF2*.exe file, continue with Step 3. |
2b) | 2b) | ||
Line 122: | Line 180: | ||
with name STD_DEV.DAT | with name STD_DEV.DAT | ||
− | :If these lines appear in the file, everything should have run correctly. Look at the rest of the PMF2.LOG file to see whether other errors are reported. If you still encounter difficulty, contact [mailto: | + | :If these lines appear in the file, everything should have run correctly. Look at the rest of the PMF2.LOG file to see whether other errors are reported. If you still encounter difficulty, contact [mailto:support@aerodyne.com?Subject=PMF%20Operating%20System%20Issue%20Donna Donna] for assistance and attach the PMF2.LOG file to your email. |
:If these lines do not appear in the file, you will see a message about PMF not being able to access one of these files. Check that the files are not being used by other programs, ''delete the file PMF2.LOG'', and press the second button on the panel again to see whether PMF runs successfully. | :If these lines do not appear in the file, you will see a message about PMF not being able to access one of these files. Check that the files are not being used by other programs, ''delete the file PMF2.LOG'', and press the second button on the panel again to see whether PMF runs successfully. | ||
− | 3. Look in the C:\ directory of your computer (C: | + | 3. Look in the C:\ directory of your computer (C:...\My Documents with the full version) for the existence of the file |
runpmf.bat | runpmf.bat | ||
Line 134: | Line 192: | ||
:If this file does not appear, Igor was not able to write this file to your C:\ drive. This might be due to high security settings on your computer. You should create a text file with this name (NOT runpmf.bat.txt) and choose to '''edit''' it (not open it) with a text editor (e.g., Notepad, WordPad, Emacs, etc.). The file should contain the lines | :If this file does not appear, Igor was not able to write this file to your C:\ drive. This might be due to high security settings on your computer. You should create a text file with this name (NOT runpmf.bat.txt) and choose to '''edit''' it (not open it) with a text editor (e.g., Notepad, WordPad, Emacs, etc.). The file should contain the lines | ||
− | cd C:\Documents and Settings\ | + | cd C:\Documents and Settings\*\Desktop\ |
− | + | pmf2wopt imupmf | |
− | :'''NOTE''' that you must change the path in this example to the path to the folder where you have put the file | + | :'''NOTE''' that you must change the path in this example to the path to the folder where you have put the file PMF2*.exe! |
3a) Execute runpmf.bat by double-clicking on its icon. You should see the black DOS window pop up, scroll output, and close again. | 3a) Execute runpmf.bat by double-clicking on its icon. You should see the black DOS window pop up, scroll output, and close again. | ||
Line 151: | Line 209: | ||
Start -> Programs -> Accessories -> Command Prompt | Start -> Programs -> Accessories -> Command Prompt | ||
− | :Change from the path displayed in the prompt to the folder where you put the | + | :Change from the path displayed in the prompt to the folder where you put the PMF2*.exe file. Use the command |
cd | cd | ||
− | :to change directories (e.g. | + | :to change directories (e.g.) |
cd Desktop\PMF | cd Desktop\PMF | ||
− | : | + | :You can change one directory at a time, or several at a time as shown in the example. To move up 1 directory, use |
cd .. | cd .. | ||
Line 165: | Line 223: | ||
:Be sure that the folder contains the three files | :Be sure that the folder contains the three files | ||
− | PMF2wtst.exe | + | pmf2wopt.exe and/or PMF2wtst.exe |
− | imupmf.ini | + | imupmf.ini |
− | pmf2key.key | + | pmf2key.key |
:by listing the contents of the current directory, using the command | :by listing the contents of the current directory, using the command | ||
Line 174: | Line 232: | ||
:Then type this command at the prompt to run PMF: | :Then type this command at the prompt to run PMF: | ||
+ | |||
+ | pmf2wopt imupmf | ||
+ | |||
+ | or | ||
pmf2wtst imupmf | pmf2wtst imupmf | ||
Line 184: | Line 246: | ||
2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF | 2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF | ||
− | :PMF ran successfully here. | + | :PMF ran successfully here. Make sure to close this file after you are finished examining it so that the next time PMF runs it won't complain about writing to this file. |
:If the output does not contain these lines, go back to Step 2 of this section to examine errors that might be reported by PMF in the file PMF2.LOG. | :If the output does not contain these lines, go back to Step 2 of this section to examine errors that might be reported by PMF in the file PMF2.LOG. | ||
− | == | + | == Data Preparation *Step 1*== |
+ | |||
+ | === Exporting the Data by Instrument, Analysis Software Type into PET === | ||
+ | |||
+ | * PMF analysis requires two 2-dimensional matrices of the same dimensions, a data matrix and a matrix corresponding to uncertainties, which is typically the calculated errors. Both of these matrices must not have any nans (Not a Number) values. In the data preparation section of PET there is a step for the automated removal of rows containing all nans and columns which contain all zeros or nans. The data matrix may contain negative values (PMF solutions will always be >=0) and the uncertainty matrix must have all values greater than zero. In the data preparation section of PET there is a step for the automated setting of a minimum non-zero error values. | ||
+ | |||
+ | The Igor PET PMF tool needs additional 1-dimensional input waves to facilitate the plotting and interpretation of PMF solutions. Required are a 1-dimenensional numeric wave corresponding to index values for the rows, which typically a indicates measurement time in increasing order and 1-dimenensional numeric wave corresponding to index values for the columns, which is typically an m/q in increasing order. For data sets where the columns correspond to high resolution data, i.e. each column indicates a chemical formula such as C2H3O, a 1-dimensional text wave will also be required. | ||
+ | |||
+ | ==== ToF AMS, Q-AMS & ToF ACSM, Q-ACSM ==== | ||
− | + | * Regardless as to whether the data has been generated from a quadrupole or ToF, users should select to use MS airbeam correction option for generating the matrices. This is because the error matrices will be generated using the airbeam correction factor if it exists. In the various analysis software packages users may generate the data in units of ug/m3 or Hz. An Igor text file can be generate to facilitate the exporting of all relevant input information for PET. | |
− | * | + | * Some steps are slightly different between manipulation of the quadrupole and the ToF data sets. In the quadrupole the removal of spikes and the smoothing of data may be necessary, where as in the ToF data sets, neither may be required (due to the fact that the quadrupole only samples one m/z at a time). |
− | * | + | * For AMS and ACSM data an additional 1-Dimensional numeric wave of the same dimension as the rows of the data matrix will be generated that contains the minimum error, the uncertainty of measuring of 1 ion per time period. This wave is recommended to be used in the PET data preparation section. |
− | * | + | * Buttons in the analysis software facilitate the export of data. Typically, the species exported is organics (Org) or high resolution organics (HROrg), although users have had success in combining this matrix with other signal (i.e. nitrate, etc. or in analyzing binned raw spectra (un-speciated). |
− | ==== In | + | ===== In ToF AMS Analysis Software (SQUIRREL/PIKA) ===== |
− | * Be sure to use v1.41 or later of the [http://cires.colorado.edu/jimenez-group/QAMSResources/Software/index.html#Anal Q-AMS Analysis Software ("James' Program")]. Corrections have been made since earlier versions to the error calculation routines! | + | * Buttons are provided for the export of data to PET. |
+ | |||
+ | * It is important to know that the CE and RIE are *NOT* applied to the Org and Org error matices and HROrg and HROrg error matrices if you choose unit so ug/m3. The general convention is that whenever Squirrel or Pika outputs anything with an m/z dimension the units are NO3-equivalent. By using nitrate equivalent mass, we are scaling all ions equally (i.e. not scaling differently according to which species each ion contributes to). This means that the mass spectra in ugm3 units are identical to those in Hz units except for a single scaling factor. The thinking behind this convention is that whenever we have an m/z dimension, such as with an average mass spectra, we can sum the unit resolution at any nominal mass to compare to the raw spectra. | ||
+ | |||
+ | ===== In the Q-AMS Analysis Software (James') ===== | ||
+ | |||
+ | * Updated Q-AMS analysis software will have buttons and other controls to easily generate * Be sure to use v1.41 or later of the [http://cires.colorado.edu/jimenez-group/QAMSResources/Software/index.html#Anal Q-AMS Analysis Software ("James' Program")]. Corrections have been made since earlier versions to the error calculation routines! | ||
* Download from [http://www.asrc.cestm.albany.edu/qz/ Qi Zhang's website] ''Extract Waves&matrices v 1.1.ipf'' (note that v1.2 is for Squirrel/HR data) and include it in your experiment with James' software. | * Download from [http://www.asrc.cestm.albany.edu/qz/ Qi Zhang's website] ''Extract Waves&matrices v 1.1.ipf'' (note that v1.2 is for Squirrel/HR data) and include it in your experiment with James' software. | ||
Line 208: | Line 284: | ||
* You may want to load the saved waves into a new experiment to run PMF. | * You may want to load the saved waves into a new experiment to run PMF. | ||
** You'll also need to include your time series wave (t_series) and a wave of the m/z's in the matrix (amus). | ** You'll also need to include your time series wave (t_series) and a wave of the m/z's in the matrix (amus). | ||
+ | <!-- [[#Deleting NaNs/zeros for all Instruments|Continue to the Deleting NaNs/zeros for all Instruments section]] --> | ||
− | + | ====== Recommended Practice: Removing Spikes (generally, for Quadrupole data) ====== | |
− | + | * "Spikes" in the time series of an ''m/z'' can occur in Q-AMS data from large but infrequent particles during the scanning of the quadrupole. If such spikes have a common source with a factor that can be retrieved by PMF, they may increase the variation of that factor profile and additional factors may be found that represent this variation, but not a physically-meaningful, separate component. The "excess signal" from these spikes can be subtracted from the spikes and the average mass spectrum of the spikes examined. See [http://cires.colorado.edu/jimenez/ams-papers.html#ZhangEST05 Zhang, Q. et al., ES&T, 2005]. | |
− | ===== Recommended Practice: Removing Spikes ===== | ||
− | "Spikes" in the time series of an ''m/z'' can occur in Q-AMS data from large but infrequent particles during the scanning of the quadrupole. If such spikes have a common source with a factor that can be retrieved by PMF, they may increase the variation of that factor profile and additional factors may be found that represent this variation, but not a physically-meaningful, separate component. The "excess signal" from these spikes can be subtracted from the spikes and the average mass spectrum of the spikes examined. See [http://cires.colorado.edu/jimenez/ams-papers.html#ZhangEST05 Zhang, Q. et al., ES&T, 2005]. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
+ | * Note that if you remove "excess signal" in the method of Zhang et al. 2005 and leave the error values for these points unchanged, you are automatically "downweighting" these points in PMF. This is appropriate because the replacement value for the original spike is not known as well as the values for points without spikes. | ||
+ | ====== Optional Practice: Smoothing ====== | ||
+ | * Smoothing can be used to reduce high-frequency noise in the data that could also be fit as additional factors. If you smooth the data, be sure to propagate this smoothing in the error matrix ''(not added to the wiki yet)''. | ||
+ | <!-- [[#Deleting NaNs/zeros for all Instruments|Continue to the Deleting NaNs/zeros section]] --> | ||
+ | <!-- | ||
==== In SQUIRREL ==== | ==== In SQUIRREL ==== | ||
Line 228: | Line 301: | ||
* If the error matrix has not yet been calculated, this must be done first. In the Corrections tab, Errors sub-tab, check the calc MS errors checkbox. Make sure you don't intend to do any other calculations (such as creating a new AB correction factor) in the corrections tab and then press the Do Corrections button. This will generate the MSSDiff_p_err data set. | * If the error matrix has not yet been calculated, this must be done first. In the Corrections tab, Errors sub-tab, check the calc MS errors checkbox. Make sure you don't intend to do any other calculations (such as creating a new AB correction factor) in the corrections tab and then press the Do Corrections button. This will generate the MSSDiff_p_err data set. | ||
− | * Squirrel | + | * Squirrel versions 1.46 - 1.48 |
In these versions the creation and export of the waves needed for PMF analysis is performed in several steps. | In these versions the creation and export of the waves needed for PMF analysis is performed in several steps. | ||
** The squirrel interface for generating the Org and Org error matricies in Squirrel have changed slightly before and after version 1.46 - 1.48. In all versions one goes to the MS tab, average mass spectra section, enters (only) Org as a species. In versions 1.46 and higher one needs to check the 'Calc, plot 1sigma err' checkbox and optionally use the 'Prompt for max m/z' checkbox and then press 'Calc Time Series Spectra' button (v. 1.47+). | ** The squirrel interface for generating the Org and Org error matricies in Squirrel have changed slightly before and after version 1.46 - 1.48. In all versions one goes to the MS tab, average mass spectra section, enters (only) Org as a species. In versions 1.46 and higher one needs to check the 'Calc, plot 1sigma err' checkbox and optionally use the 'Prompt for max m/z' checkbox and then press 'Calc Time Series Spectra' button (v. 1.47+). | ||
Line 266: | Line 339: | ||
In Pika 1.09F there is a handy checkbox that when checked will automatically generate an itx containing all the waves necessary for the PMF (with the exception of any data preparation steps such as the minimum error, Nan removal, etc.) | In Pika 1.09F there is a handy checkbox that when checked will automatically generate an itx containing all the waves necessary for the PMF (with the exception of any data preparation steps such as the minimum error, Nan removal, etc.) | ||
− | + | --> | |
+ | |||
+ | ==== Tofware for ToF CIMS and ACSM data ==== | ||
+ | |||
+ | Controls in Tofware allow the export of all necessary Igor waves into one external file. | ||
=== For AMS Organics + Other Signals including AMS Inorganics === | === For AMS Organics + Other Signals including AMS Inorganics === | ||
− | Setting up the signal and error matricies for org+inorg is not terribly difficult. The interpretation is much more complicated and nuanced than for a PMF analysis on only the organics. | + | Setting up the signal and error matricies for org+inorg is not terribly difficult. The interpretation is much more complicated and nuanced than for a PMF analysis on only the organics. One can look through 2009 - 2019 [https://cires1.colorado.edu/jimenez-group/wiki/index.php/AMSUsrMtgs AMS Users Meeting talks] by searching for "PMF" to review many options that have been presented and subsequently published. |
− | + | <!-- | |
Adding other signals to the organic matrix and error matrix is done most simply by a cut and paste operation. In Igor make 2 tables:one containing the organic matrix, the other containing the error matrix. You will cut and paste columns (columns correspond to the time series in the 2D matricies) to the organic and error matrices. In the case of UMR you will likely have to add new columns to the organic matrix to the right side end, the max m/z, and keep track of which columns indicate which species m/z. This is because, for example, m/z 30 will have independent signals from org and nitrate. | Adding other signals to the organic matrix and error matrix is done most simply by a cut and paste operation. In Igor make 2 tables:one containing the organic matrix, the other containing the error matrix. You will cut and paste columns (columns correspond to the time series in the 2D matricies) to the organic and error matrices. In the case of UMR you will likely have to add new columns to the organic matrix to the right side end, the max m/z, and keep track of which columns indicate which species m/z. This is because, for example, m/z 30 will have independent signals from org and nitrate. | ||
Line 281: | Line 358: | ||
At this point you have two options for your cutting and pasting. There exists an org and a nitrate signal at m/z 30, so our m/z X = columnX-1 system will be messed up. I suggest that you simply copy and paste the nitrate signal at m/z 30 and 46 to the max org m/z signal you have. So if your original org matrix had 200 columns corresponding to m/z 1 - 200 (columns 0 - 199), your new organic+nitrate matrix will have 202 columns. | At this point you have two options for your cutting and pasting. There exists an org and a nitrate signal at m/z 30, so our m/z X = columnX-1 system will be messed up. I suggest that you simply copy and paste the nitrate signal at m/z 30 and 46 to the max org m/z signal you have. So if your original org matrix had 200 columns corresponding to m/z 1 - 200 (columns 0 - 199), your new organic+nitrate matrix will have 202 columns. | ||
− | The PMF tools won't care at all that column 200 (counting from zero) really is nitrate at 30, etc; it will execute fine. However, you will have to get used to looking at these modified 'spectra' in the | + | The PMF tools won't care at all that column 200 (counting from zero) really is nitrate at 30, etc; it will execute fine. However, you will have to get used to looking at these modified 'spectra' in the PET panels. When you get ready to publish this, you will have to modify/separate out 'by hand' the org and nitrate parts and 'fold in', if you like, the UMR derived portions of your nitrate signal (i.e. 0.004*frag_nitrate[46] at m/z 48 for nitrate). |
==== For HR: ==== | ==== For HR: ==== | ||
Line 299: | Line 376: | ||
** A small but tedious detail to be aware of is the H2O contributions for sulfate and organics. If you do an organic+sulfate+xx PMF analysis I'd just remove all signals of OH and H2O (and O?). We estimate these HR ion contributions from the major signals (CO2, etc) anyway. | ** A small but tedious detail to be aware of is the H2O contributions for sulfate and organics. If you do an organic+sulfate+xx PMF analysis I'd just remove all signals of OH and H2O (and O?). We estimate these HR ion contributions from the major signals (CO2, etc) anyway. | ||
− | ==== For non-AMS Users ==== | + | ==== For non-AMS/ACSM Users ==== |
You will need to create 5 waves to run your PMF analysis in Igor: | You will need to create 5 waves to run your PMF analysis in Igor: | ||
Line 307: | Line 384: | ||
* a '''numeric''' wave that has index values for your species (1, 2, 3, ...) | * a '''numeric''' wave that has index values for your species (1, 2, 3, ...) | ||
* a time wave | * a time wave | ||
− | + | --> | |
+ | <!-- | ||
This section describes how you might make those waves, assuming your data is in Excel and is loaded to Igor with one wave for each species and another wave for each species' error values. Depending on the initial format of your data, some steps may not apply to you. | This section describes how you might make those waves, assuming your data is in Excel and is loaded to Igor with one wave for each species and another wave for each species' error values. Depending on the initial format of your data, some steps may not apply to you. | ||
Line 361: | Line 439: | ||
make/N=(numpnts(year))/D timeseries = date2secs(year,month,day) + hour*60*60 + minute*60 + second | make/N=(numpnts(year))/D timeseries = date2secs(year,month,day) + hour*60*60 + minute*60 + second | ||
+ | --> | ||
+ | |||
+ | === Further Data Preparation Before Calling PMF ('''Prep''' tab in the PET panel) === | ||
+ | * The following steps are recommended for AMS datasets and follow the practices laid out in [http://www.atmos-chem-phys.net/9/2891/2009/acp-9-2891-2009.html| Ulbrich et al., ACP, 2009] (with more detailed references in each section below). Note that '''the only mandatory step''' is step 0 (importing data into the experiment and selecting it) and step 4, (Deleting NaNs/zeros). More extensive documentation on use of the remove/delete nan functions is given in the header comments in the ipf pmf_ErrPrep_AMS_v*.ipf. | ||
+ | |||
+ | ''Before running these functions, you may want to rename your error matrix so that it has fewer than 12 characters. Each data prep step lengthens the wavename and the code will complain if the name gets too long.'' | ||
+ | |||
+ | ==== Data Prep Step 0. Select Initial data ==== | ||
+ | * From the top Igor menu Data - Load Waves load in the Igor file that you exported from the analysis software - Tofware, Squirrel, etc. It is often good practice to create a data folder to contain these newly imported data waves. | ||
− | + | * Select the data folder containing the PMF input and subsequent DATA, ERROR, etc waves. | |
− | + | ==== Recommended Practice: Data Prep Step 1. Apply a Minimum Error ==== | |
+ | * Ions arrive at the mass spectrometer detector with a [http://en.wikipedia.org/wiki/Poisson_distribution| Poisson distribution]. The error for a counted number of ions is sqrt(counted number of ions). The smallest number of ions we can count in one run is, of course, zero ions, but perhaps there was one and it was missed. The error for counting zero is sqrt(0), but an error of 1 would be more appropriate in this case. Hence a minimum error threshold of 1 ion is set. | ||
− | '' | + | * Within this step the user must choose the mass spec detector type. For ToF detectors, there is an m/z dependence to account for the unequal 'pushing' of the ions into the detector. Small m/z ions move faster than large m/z at an approximate rate of square root of the mass. This phenomena is sometimes referred to as an 'm/z duty cycle' correction. The application of a minimum error must undo this duty cycle correction if it was applied to the data to reflect true ions counted. In practice then, when one selects the AMS-ToF for the MS type, a sqrt(28)/sqrt(m/z) factor is temporarily multiplied to the error matrix. For AMS-quad data and for Tofware-processed CIMS data, this undoing of the correction factor is not needed. |
− | + | * The one ion wave needs to be in the same units as the data and error matrices. If the data and error matrices are in units of Hz and the data acquisition (for AMS the actual time measuring i.e. in the 'MS open' mode) for each run is 1 second, then the one ion wave is simply a wave containing all 1s. If each run had an MS open mode lasting one minute, the wave would consist of 1/60. If the data and error matrices are in typical AMS units of ug/m3 then one needs to convert from Hz using the flow rate, ionization efficiency, AB correction factor, etc. This will have been done for you in the PMF export step in Squirrel, Pika, ACSM Tofware code. | |
− | + | * By pressing the "Apply minimum error" button the function produces the following waves: | |
+ | # The error matrix with minimum error applied called nameOfWave(errMx)+"_min" (e.g., Orgerr becomes Orgerr_min) | ||
+ | <!-- | ||
The minimum error is applied in three steps: | The minimum error is applied in three steps: | ||
Line 381: | Line 471: | ||
# The error matrix with minimum error applied called nameOfWave(errMx)+"_min" (e.g., OrgMSerr becomes OrgMSerr_min) | # The error matrix with minimum error applied called nameOfWave(errMx)+"_min" (e.g., OrgMSerr becomes OrgMSerr_min) | ||
# A matrix of the fractional increase of the errors called nameOfWave(errMx)+"_adjErrMask" where the value of each point is (new/old)-1. | # A matrix of the fractional increase of the errors called nameOfWave(errMx)+"_adjErrMask" where the value of each point is (new/old)-1. | ||
+ | --> | ||
+ | ''See also'' discussion of Ulbrich et al., ACPD 2008, [http://www.cosis.net/copernicus/EGU/acpd/8/S2726/acpd-8-S2726.pdf?PHPSESSID=511a6d01a82e0953a68adf517ffd08f8 P. Paatero comment (p. S5730)] and [http://www.cosis.net/copernicus/EGU/acpd/8/S11954/acpd-8-S11954.pdf?PHPSESSID=511a6d01a82e0953a68adf517ffd08f8 Author response (p. S11960)] | ||
− | + | ==== Data Prep Step 2. Removing Spikes (data dependent) ==== | |
+ | |||
+ | * By pressing the Remove spikes (optional) button new data and error matrices will be generated with '''spikes''' removed. | ||
− | ==== Propagation of Smoothing ( | + | ==== Data Prep Step 3. Propagation of Smoothing (generally not relevant for ToF data) ==== |
− | Any smoothing of the data matrix must be propagated in the error matrix. The function '''pmf_err_propogateSmooth''' propagates box or Gaussian smoothing. | + | *Any smoothing of the data matrix must be propagated in the error matrix. The function '''pmf_err_propogateSmooth''' propagates box or Gaussian smoothing. |
Some notes about specifying the smoothing that was performed for the data: | Some notes about specifying the smoothing that was performed for the data: | ||
Line 394: | Line 488: | ||
#* For Gaussian smoothing, the number of points refers to the number of adjacent points used. I.e., smoothing that includes 1 adjacent point on each side is 1-point smoothing. | #* For Gaussian smoothing, the number of points refers to the number of adjacent points used. I.e., smoothing that includes 1 adjacent point on each side is 1-point smoothing. | ||
− | + | * By pressing the "Propogate smoothing to the error mx" button the function produces a wave with the propagated error called nameOfWave(errMx)+"Prop" (e.g., OrgMSerr_Min becomes OrgMSerr_minProp). | |
− | ==== | + | ==== Data Prep Step 4. Removing Nans, 0 columns ==== |
+ | *This step is required for all data. | ||
− | The | + | * The data matrix produced in the previous steps may have NaNs in all rows from bad runs and 0 values in columns with good runs that have no fragments of interest (i.e. Argon, or m/z 40 for AMS data). All of these rows and columns need to be removed before running PMF. |
+ | |||
+ | * The function produces a wave with the propagated error called nameOfWave(errMx)+"_noNans" (e.g., OrgMSerr_Min becomes OrgMSerr_min_noNans). | ||
+ | <!-- | ||
+ | This is a two-step process. | ||
'''1.''' First, change the columns with 0's to NaNs. | '''1.''' First, change the columns with 0's to NaNs. | ||
Line 488: | Line 587: | ||
; pmf_ams_RemoveBlockSize_NaNsWv(NaNsWv, BlockSizeToRemove) | ; pmf_ams_RemoveBlockSize_NaNsWv(NaNsWv, BlockSizeToRemove) | ||
: Creates ''NaNsWv_''type''_trunc'' and deletes the records for original blocks of NaNs of size BlockSizeToRemove. If you did V-W switching and are only running the V-mode data in PMF, your NaNsWv_tseries will have an entry for every other row of your original matrix noting that 1 NaN row was removed. You may also have periods with longer contiguous runs of NaNs (e.g., instrument was down, filter runs, etc.). For publication plots, you should insert NaNs into your final factor waves so that periods of missing data are not crossed by a line connecting the good data before and after the real gaps. For NaNs removed due to menu switching, however, you may not want to insert all of these NaNs again. (In the example case, each solved value would have a NaN on each size of it and you couldn't plot with Lines Between Points. If you uncheck "Gaps" to connect these points, you'll also have the undesirable lines across periods of missing data.) Running '''pmf_ams_removeBlockSize_NaNsWv(NaNsWv_tseries, 1)''' would create ''NaNsList_tseries_trunc'' which would not include a record of all the removed W runs and then could be used to insert NaNs representing only missing data periods in the final waves. | : Creates ''NaNsWv_''type''_trunc'' and deletes the records for original blocks of NaNs of size BlockSizeToRemove. If you did V-W switching and are only running the V-mode data in PMF, your NaNsWv_tseries will have an entry for every other row of your original matrix noting that 1 NaN row was removed. You may also have periods with longer contiguous runs of NaNs (e.g., instrument was down, filter runs, etc.). For publication plots, you should insert NaNs into your final factor waves so that periods of missing data are not crossed by a line connecting the good data before and after the real gaps. For NaNs removed due to menu switching, however, you may not want to insert all of these NaNs again. (In the example case, each solved value would have a NaN on each size of it and you couldn't plot with Lines Between Points. If you uncheck "Gaps" to connect these points, you'll also have the undesirable lines across periods of missing data.) Running '''pmf_ams_removeBlockSize_NaNsWv(NaNsWv_tseries, 1)''' would create ''NaNsList_tseries_trunc'' which would not include a record of all the removed W runs and then could be used to insert NaNs representing only missing data periods in the final waves. | ||
+ | --> | ||
+ | |||
+ | ==== Data Prep Steps 5. and 6. Recommended Practice: Downweight "Weak" Variables (''m/z'''s) ==== | ||
− | + | * Any m/zs that have low signal-to-noise ratio (SNR) may, in fact, have more noise than signal. If these m/zs contribute enough Q, PMF tries to fit the noisy data. In this way, the inclusion of such m/zs can be detrimental to the PMF analysis. If the error associated with these ''m/z'' 's is increased, the Q-contribution (residual/error) is decreased, "downweighting" these points' contribution to the fit. m/zs with SNR<0.2 are considered "bad" by Paatero and Hopke (2003) and should be removed or strongly downweighted (factor of ~10). ''m/z'' 's with 0.2<SNR<2 are considered "weak" and should be downweighted (factor of 2-3). | |
− | + | * When a user presses the "Calculate, view SNR panel" A separate panel appears that shows time series, errors and SNR calculations. This tool is handy to examine ions that are deemed to be "bad" or "weak". This panel is informational only and is used in step 6. | |
+ | * When a user presses the "Downweight weak, bad mzs" button the code will generate new error matrices that have the bad and weak m/z columns adjusted. The function generates a new error matrix called nameofWave(errMx)+"Wk" (e.g., noNaNs_orgMSerr_minProp would become noNaNs_orgMSerr_minPropWk). | ||
+ | |||
+ | <!-- | ||
The calculation of SNR and downweighting of "weak" ''m/z'''s is carried out in three steps: | The calculation of SNR and downweighting of "weak" ''m/z'''s is carried out in three steps: | ||
− | # Calculate SNR of each | + | # Calculate SNR of each m/z using function '''pmf_err_SNRwv''' using the data matrix, the version of the error matrix generated in the previous step, and the model error that will be used in the panel. The function generates a wave of the SNR for each m/z called nameofwave(DataMx) + "_SNRwv". |
# Check the graph produced for "bad" ''m/z'''s. These are not removed in the next function. To remove these columns, you'll need to rerun the ''DeleteNaNs'' step after making them into NaNs or changing them in the mask wave. (This is better than just deleting the columns by had because they'll be added to the ''NaNsList_amus'' and will be reinserted if you '''insertNaNs''' later. | # Check the graph produced for "bad" ''m/z'''s. These are not removed in the next function. To remove these columns, you'll need to rerun the ''DeleteNaNs'' step after making them into NaNs or changing them in the mask wave. (This is better than just deleting the columns by had because they'll be added to the ''NaNsList_amus'' and will be reinserted if you '''insertNaNs''' later. | ||
− | # Downweight "weak" | + | # Downweight "weak" m/zs with function '''pmf_err_DwntWeakColumns''' using the SNRwv generated in the previous step, the error matrix used to calculate the SNRwv, and the multiplicative value used to downweight the weak ''m/z'' 's (Paatero and Hopke recommend 2-3). |
− | |||
+ | --> | ||
+ | ''See also'' Paatero, P., and Hopke, P. K.: Discarding or downweighting high-noise variables in factor analytic models, Anal. Chim. Acta, 490, 277-289, 10.1016/s0003-2670(02)01643-4, 2003. [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6TF4-47RJM08-2&_user=918210&_coverDate=08/25/2003&_rdoc=24&_fmt=high&_orig=browse&_srch=doc-info(%23toc%235216%232003%23995099998%23448216%23FLA%23display%23Volume)&_cdi=5216&_sort=d&_docanchor=&_ct=33&_acct=C000047944&_version=1&_urlVersion=0&_userid=918210&md5=884956903b4a3f2da627f57207589050 Abstract] | ||
− | '' | + | ==== Data Prep Step 7. Recommended Practice: Downweight Peaks Related to ''m/z'' 44 in Frag Table (for AMS and ACSM data only) ==== |
− | + | * In the default fragmentation table, the information in ''m/z'' 44 is repeated 6 or 7 times (in ''m/z'' 's 16, 17, 18, 19, 20, (28 in the Aiken et al. 2008 revision), and 44) with different proportionalities. PMF fits correlations, regardless of the magnitudes of the signals. Repeating the information of ''m/z'' 44 several times implies that it's really (x6 or 7) important, which it isn't! It is possible to downweight the columns of these ''m/z'' 's so that in total they only contribute the ''m/z'' 44 signal once. (It would be possible to remove the repeated information and replace those columns after running PMF, but we think that downweighting them is logistically simpler.) | |
− | + | * There are two buttons, one for UMR AMS or ACSM data and another for HR AMS or ACSM data. Once you press the button the List of duplicate ions will be filled in. If you are analyzing HR data and using the default HR organics frag waves, the equivalent set of HR ions of (the mz44list) correspond to O, HO, H2O, CO and CO2. If you are analyzing organic UMR data the default list will include 16, 17, 18, 19, 20, 28, 44. Then press the '''Downweight duplicated columns''' button. | |
+ | * The function generates a new error matrix called nameofWave(errMx)+"44" (e.g., noNaNs_orgMSerr_minPropWk would become noNaNs_orgMSerr_minPropWk44). | ||
+ | <!-- | ||
Downweighting the ''m/z'' 's related to ''m/z'' 44 is accomplished in two steps: | Downweighting the ''m/z'' 's related to ''m/z'' 44 is accomplished in two steps: | ||
# Make a wave that contains the ''m/z'' 's related to m/z 44 (in any order), e.g. | # Make a wave that contains the ''m/z'' 's related to m/z 44 (in any order), e.g. | ||
Line 515: | Line 623: | ||
:: make/N=7 mz44peaksWv = {16, 17, 18, 19, 20, 28, 44} | :: make/N=7 mz44peaksWv = {16, 17, 18, 19, 20, 28, 44} | ||
# Use function '''pmf_err_dnwt44peaks''' with the error matrix generated in the previous step, the ''mz44peaksWv'', and the ''noNaNs_amus'' wave. | # Use function '''pmf_err_dnwt44peaks''' with the error matrix generated in the previous step, the ''mz44peaksWv'', and the ''noNaNs_amus'' wave. | ||
+ | --> | ||
+ | |||
+ | ==== Data Prep Conclusion: Indicate that all Data Prep Steps have been Completed ==== | ||
+ | |||
+ | * Press the gold "Select prepared waves for PMF" button. The second tab of the PET panel, "Run", will become active. | ||
+ | |||
+ | == Perform PMF Analysis == | ||
− | + | * At this point you are ready to indicate where the PMF executable file resides, where the prepared data and error matrices reside within the Igor experiment and what PMF solutions you are wanting to investigate. | |
− | + | Press the "Create path to PMF executable and batch files" button. A window will pop up asking you where the PMF executable file resides. The path will be displayed once a user has selected a folder. If this path text is red, something is amiss and the code couldn't find the PMF executable. | |
− | + | If you pressed the gold button "Select prepared waves for PMF" in the previous data prep tab, entries in the data section will be filled in for you. | |
+ | <!-- | ||
=== Additional File for PMF Executable Directory === | === Additional File for PMF Executable Directory === | ||
Line 546: | Line 662: | ||
#*Chose the folder (must be root: or a directory in root:) | #*Chose the folder (must be root: or a directory in root:) | ||
#*Choose the Data and Error matrices (use noNaNs_ versions) | #*Choose the Data and Error matrices (use noNaNs_ versions) | ||
− | #*Choose model error (PMF increases the errors provided by newError = oldError + modelError*dataValue) | + | --> |
+ | #*Choose model error | ||
+ | #**It is very rare for users to change this model error value from the default of 0. (PMF increases the errors provided by newError = oldError + modelError*dataValue) | ||
# Choose the type of PMF analysis | # Choose the type of PMF analysis | ||
#* Exploration will run PMF for a range of number of factors and FPEAKs or SEEDs. This is the typical use for exploring a dataset and comparing many solutions. | #* Exploration will run PMF for a range of number of factors and FPEAKs or SEEDs. This is the typical use for exploring a dataset and comparing many solutions. | ||
#* Bootstrapping explores the uncertainty of '''one''' solution (i.e., ''one'' number of factors at ''one'' fpeak for ''one'' seed). This is usually a final step run only on the solution you have selected from the exploratory analysis. | #* Bootstrapping explores the uncertainty of '''one''' solution (i.e., ''one'' number of factors at ''one'' fpeak for ''one'' seed). This is usually a final step run only on the solution you have selected from the exploratory analysis. | ||
# Choose a range for number of factors. | # Choose a range for number of factors. | ||
− | #* When checking to make sure that everything runs properly, you may want to run just one case (Min p = 2, Max p = 2). | + | #* When checking to make sure that everything runs properly, you may want to run just one case (Min p = 2, Max p = 2; p is the number of solved factors). |
#* Recommended Practice: Run cases with 1 factor to have a context for the meaning of the 2-factor solution. | #* Recommended Practice: Run cases with 1 factor to have a context for the meaning of the 2-factor solution. | ||
#* In the Bootstrapping mode, only the "min p" is read; the "max p" is ignored. | #* In the Bootstrapping mode, only the "min p" is read; the "max p" is ignored. | ||
Line 566: | Line 684: | ||
# Select checkboxes | # Select checkboxes | ||
#* '''Run PMF in background''' Each execution of PMF (see [[#Exploration Mode Summary| Exploration Mode Summary]]) creates a black DOS window that pops up. If the box is not checked, this window "grabs the focus" and makes itself the top window. This makes it hard to use the computer for anything else. If the box is checked, the window will not grab focus, but Igor and your computer's CPU will be busy. | #* '''Run PMF in background''' Each execution of PMF (see [[#Exploration Mode Summary| Exploration Mode Summary]]) creates a black DOS window that pops up. If the box is not checked, this window "grabs the focus" and makes itself the top window. This makes it hard to use the computer for anything else. If the box is checked, the window will not grab focus, but Igor and your computer's CPU will be busy. | ||
− | + | #* '''Save experiment before and after PMF has finished''' It is generally a good idea to save the experiment before calling PMF so that all the data prep steps and user selections will be saved before attempting to call the PMF executable. Once the code has finished all its calls to the PMF executable it is a good idea to save the experiment because all the PNF results are now saved as waves within the experiment. | |
==== What the Software Does When You Press the Button... ==== | ==== What the Software Does When You Press the Button... ==== | ||
===== Exploration Mode Summary ===== | ===== Exploration Mode Summary ===== | ||
− | The software will execute PMF once for every combination of number of factors and FPEAK/seed. So if you run 1- | + | The software will execute PMF once for every combination of number of factors and FPEAK/seed. So if you run 1-6 factors and 5 FPEAKs, PMF will run 6x5=30 times. Each run starts a new black DOS window that will close when the run is completed. The duration of each run is printed in the history at the end of each run. In general, runs which solve for more factors and runs with FPEAK farther from 0 take longer. The code runs all of the FPEAKS or seeds for one number of factors, then advances to the next number of factors (e.g., run 1 factor with each of 5 FPEAK values, then 2 factors with each of 5 FPEAK values, etc.). |
+ | |||
+ | When the entirety of sets of calls to the PMF executable has completed many diagnostic values are calculated. The panel from where you selected "Begin PMF analysis with these choices" will be killed and a new panel "View PMF Analysis Results" will be created with PMF inputs filled in for you. Select options for possible additional calculations. For example for AMS/ACSM HR Organic data, calibrated elemental ratios can be generated for each factor. | ||
====== A little more detail ====== | ====== A little more detail ====== | ||
Line 577: | Line 697: | ||
C:\delete_log.bat C:\runPMF.bat | C:\delete_log.bat C:\runPMF.bat | ||
− | and writes | + | and writes the DataMatrix and ErrorMatrix as MATRIX.DAT and STD_DEV.DAT, respectively to the folder with your PMF Executable. The software also writes a file to that folder called STD_DEV_PROP.DAT, which has the same number of points as the DataMatrix and in which every element is equal to the ModelError. These files will be read by the PMF executable. |
− | |||
The software then enters a pair of nested loops in which the following steps occur: | The software then enters a pair of nested loops in which the following steps occur: | ||
Line 588: | Line 707: | ||
imupmf.ini | imupmf.ini | ||
:::which is used as the control file for PMF. | :::which is used as the control file for PMF. | ||
− | + | * The '''convergence criteria''' for completing the PMF calculations can be set in the 3rd tab of the PMR_PerformCalcPanel "PMF defaults & advanced options". | |
− | // | + | //Default settings for PMF iteration convergence proportional to Qexp |
constant ITERATION_CHI2_PROP_QEXP = 0 | constant ITERATION_CHI2_PROP_QEXP = 0 | ||
constant FIRST_ITER_LEVEL_PROP_CONST = 2e-6 | constant FIRST_ITER_LEVEL_PROP_CONST = 2e-6 | ||
constant SECOND_ITER_LEVEL_PROP_CONST = 2e-6 | constant SECOND_ITER_LEVEL_PROP_CONST = 2e-6 | ||
constant THIRD_ITER_LEVEL_PROP_CONST = 1e-6 | constant THIRD_ITER_LEVEL_PROP_CONST = 1e-6 | ||
− | |||
:::: You can read more about convergence criteria and large datasets in the PMF Users Manual Part 1 (pg. 10) (see [[#Other Resources|Other Resources]]). | :::: You can read more about convergence criteria and large datasets in the PMF Users Manual Part 1 (pg. 10) (see [[#Other Resources|Other Resources]]). | ||
:*Delete the old PMF2.LOG file by running delete_log.bat | :*Delete the old PMF2.LOG file by running delete_log.bat | ||
Line 601: | Line 719: | ||
:*Load PMF output (including log file and factors) | :*Load PMF output (including log file and factors) | ||
− | At the completion of the loops, the software calculates some statistics from the output and then creates a panel to select | + | At the completion of the loops, the software calculates some statistics from the output and then creates a panel to select PMF results for viewing. |
− | ===== Bootstrapping | + | ===== Bootstrapping ===== |
The bootstrapping mode is developed after the method described in the EPA PMF v3.0 Users Manual Sect. 6.4 (see [[#Other Resources|Other Resources]]). The bootstrapping method is used to estimate the uncertainty in both the factor mass spectra and time series. This is achieved by running PMF on the full dataset once (''from one PMF solution (i.e., combination of number of factors and fpeak) at a time'') and then making a series of variations (the number is specified by the user when selecting the bootstrapping mode) in which a subset of the original rows (mass spectra) are randomly replaced by other rows from the original matrix and running PMF on each of these. | The bootstrapping mode is developed after the method described in the EPA PMF v3.0 Users Manual Sect. 6.4 (see [[#Other Resources|Other Resources]]). The bootstrapping method is used to estimate the uncertainty in both the factor mass spectra and time series. This is achieved by running PMF on the full dataset once (''from one PMF solution (i.e., combination of number of factors and fpeak) at a time'') and then making a series of variations (the number is specified by the user when selecting the bootstrapping mode) in which a subset of the original rows (mass spectra) are randomly replaced by other rows from the original matrix and running PMF on each of these. | ||
Line 627: | Line 745: | ||
Note that the criteria for including bootstrapped cases in calculations of the average bootstrapped factors is different in our code than in EPA PMF. Bootstrapped cases in which two or more factors are mapped to the same original factor are not included in the averages in our code. In these instances, the whole bootstrapping case is rejected before calculating averages. In EPA PMF, all bootstrapped factors mapped to the same original factor are included in the average. | Note that the criteria for including bootstrapped cases in calculations of the average bootstrapped factors is different in our code than in EPA PMF. Bootstrapped cases in which two or more factors are mapped to the same original factor are not included in the averages in our code. In these instances, the whole bootstrapping case is rejected before calculating averages. In EPA PMF, all bootstrapped factors mapped to the same original factor are included in the average. | ||
− | ''Important:'' At the present time | + | ''Important:'' At the present time the plots produced by the bootstrapping code expect to find the waves '''noNaNs_amus''' (for plotting profiles) and '''noNaNs_t_series''' (for plotting time series). If these are not the names of your waves that describe the columns and rows, respectively) of the input matrix, you should duplicate your row and column description waves to have these names. |
− | ==== | + | ===== When PMF seems to work for some solutions but not others ===== |
+ | In the "Perform PMF calculations" panel, "PMF defaults and advanced options" tab, in versions 3.08D and higher there is a checkbox named "Save the log file for every iteration". If this is checked, then after every call to PMF the PET code will copy the PMF2.LOG file and rename it PMF2_x_y.LOG where x and y are the current number of factors and fpeak. One can then look through these files to see what problems may have arisen for a specific case. | ||
− | You can run two PMF analyses (in two separate experiments) on the same computer simultaneously if you have | + | ==== Running Two Simultaneous Analyses on Multi-Processor Computers ==== |
+ | |||
+ | You can run two or more PMF analyses (in two separate experiments) on the same computer simultaneously if you have multiple processors. Each analysis will run at the same speed as on a single-processor computer (or when one analysis is run on the dual processor computer). The PMF executable is not "multi-processor aware," meaning that it can not utilize both processors simultaneously for one PMF run. | ||
To run two simultaneous analyses with the PET, you'll need '''two''' directories on your computer with the PMF .exe, .key, and mypfmt.ini files. The directory names must end in "1" and "2," respectively. For example, you could have | To run two simultaneous analyses with the PET, you'll need '''two''' directories on your computer with the PMF .exe, .key, and mypfmt.ini files. The directory names must end in "1" and "2," respectively. For example, you could have | ||
− | ; <nowiki>C:\PMF\PMF Executable1</nowiki> | + | ; <nowiki>C:\Users\*\Documents\PMF\PMF Executable1</nowiki> |
− | : | + | : pmf2wopt.exe |
: pmf2key.key | : pmf2key.key | ||
: mypmft.ini | : mypmft.ini | ||
− | ; <nowiki>C:\PMF\PMF Executable2</nowiki> | + | ; <nowiki>C:\Users\*\Documents\PMF\PMF Executable2</nowiki> |
− | : | + | : pmf2wopt.exe |
: pmf2key.key | : pmf2key.key | ||
: mypmft.ini | : mypmft.ini | ||
'''Each experiment running PMF must use a separate Executable directory.''' While a PMF analysis associated with a PMF Executable directory is running, a file called "PMFrunning.txt" exists in that directory. If you try to run your second PMF analysis using the same Executable directory, the PET will give you an error. You can choose a different Executable directory and press the button to start the analysis. | '''Each experiment running PMF must use a separate Executable directory.''' While a PMF analysis associated with a PMF Executable directory is running, a file called "PMFrunning.txt" exists in that directory. If you try to run your second PMF analysis using the same Executable directory, the PET will give you an error. You can choose a different Executable directory and press the button to start the analysis. | ||
+ | |||
+ | ==== Convergence ==== | ||
+ | |||
+ | PMF uses a variety of metrics to assess the progress of finding a solution. Some of these metrics are based on parameters that were initially optimized for AMS data, and hence, may not be appropriate for other data sets. The last tab in the PMF_PerformCalcs panel has a section entitled Q convergence criteria which lists some defaults; users may modify these values to achieve convergence. The log text file, PMF2.LOG generates information about the progress for finding a solution. As an example the following is an excerpt that was written to this file: | ||
+ | |||
+ | The log text file, PMF2.LOG generates information about the progress for finding a solution. Below is an excerpt from a file where a PMF solution didn't converge. | ||
+ | |||
+ | 299 rank1 step chi2= 6.22372E+06 Penalty= 2.5296E+04 Flags GF | ||
+ | 300 rank1 step chi2= 6.22364E+06 Penalty= 2.5295E+04 Flags GF | ||
+ | ************************************************************ 1 | ||
+ | */ Iteration interrupted because maximum step count | ||
+ | *\ has been exceeded in the robust mode. NO CONVERGENCE? | ||
+ | ************************************************************ 1 | ||
+ | 300 iter-steps in all. Final chi2= 6223643.5000 in the robust mode. | ||
+ | |||
+ | As a practical matter, the last version of the PMF solution is saved within PET and users can view this last result for this PMF run. However, because it did not converge graphs in the main PMF panel will be grayed out and the Q/Qexp value will be blank. | ||
== View PMF Analysis Results *Step 2* == | == View PMF Analysis Results *Step 2* == | ||
− | + | Once PET has finished calling PMF and has calculated some diagnostics the "Perform PMF" panel will close and a "View Results" panel will appear with the previous selections of folders and waves preselected. For AMS/ACSM high resolution organic data additional calculations regarding elemental summing & ratios can be selected. Press the gold button "View PMF analysis" to proceed with generating the large PMF results panel. | |
+ | <!-- | ||
This is a clickable picture of the main PMF panel. Clicking on a region will take you down the page to more information on the options possible for that control or graph. | This is a clickable picture of the main PMF panel. Clicking on a region will take you down the page to more information on the options possible for that control or graph. | ||
<imagemap> | <imagemap> | ||
Line 671: | Line 809: | ||
desc bottom-left | desc bottom-left | ||
</imagemap> | </imagemap> | ||
+ | --> | ||
=== Controls === | === Controls === | ||
Line 810: | Line 949: | ||
|[[image:RR_plot_2.JPG]] <!---p-1---> | |[[image:RR_plot_2.JPG]] <!---p-1---> | ||
|} | |} | ||
+ | |||
+ | === AMS loadings and the RIE, CE and CDCE factors === | ||
+ | When exporting species like Org and HROrg from all versions of Squirrel and Pika, users should be aware that the RIE or the CE or CDCE factors are not applied to the 2-dimensional data. | ||
+ | The reasons for this rule are somewhat historical. Basically, whenever a result in Squirrel or Pika retains an MS dimension, as in the export of the 2D wave for PMF, which has dimensions of measurement time and UMR or HR mass values, the RIE and CE (or CDCE or HR CDCE values) are not applied. | ||
+ | |||
+ | Users may wish to compare factors and loading from PMF results to species time series loadings with the RIE or CE applied. Below is a guide for applying default RIE and HR CDCE to source factors within a PET experiment; similar operations can be used for CE or UMR CDCE. Be aware that by performing this operation, the user assumes that the RIE and CE (or CDCE or HR CDCE) is the same for all factors. This may not reflect reality. For example RIE calibrations and CE (or CDCE or HR CDCE) have been generated for bulk ambient organic aerosol; the RIE and CE for BBOA and COA, for example, may not be identical. | ||
+ | |||
+ | There are 3 main steps to incorporating the RIE and CE (or CDCE or HR CDCE) to PMF factors. | ||
+ | |||
+ | 1. Generate a 1D wave matching the time series wave containing a multiplicative factor of the RIE and CE or CDCE and import it to the PMF experiment. | ||
+ | |||
+ | 2. Remove points in this 1D wave that matches the nans removed in the data prep step of PMF. | ||
+ | |||
+ | 3. Divide this wave to your time series factor waves in the PMF experiment. | ||
+ | |||
+ | |||
+ | More details and example of commands are below. | ||
+ | |||
+ | Step 1. Within your Pika experiment, the HR CDCE is root:HRCE_fphase; for UMR data it is root:CE_fphase. The RIE values can be found in the UMR or HR batch tables. Here are commands you can use and modify as needed. | ||
+ | |||
+ | Duplicate/o root:HRCE_fphase root:HRRIECEfactor // Use a similar command for UMR. This wave should be the same length as the t_series wave | ||
+ | |||
+ | root:HRRIECEfactor *= 1.4 // 1.4 is the RIE of HROrg in this data. We will divide the PMF factors by this HRRIECEfactor wave. | ||
+ | |||
+ | Save/T/M="\r\n" HRRIECEfactor as "HRRIECEfactor.itx" // Save this wave as separate itx file. | ||
+ | |||
+ | In this example I saved this file in the same place as the exported PMF text file (that you had loaded into your PMR experiment at the beginning of that analysis). You can also load this wave into your PMF experiment via the browse experiment command without saving it as an external text file. | ||
+ | |||
+ | Step 2. There are a few ways to accomplish this step. Within your PMF experiment and data folder containing the original data (before noNan_* versions of the data was created, create a time series wave to flag when rows (time points) were nan. | ||
+ | |||
+ | Load the HRRIECEfactor wave from step 1 and multiply this wave by it. | ||
+ | |||
+ | LoadWave/T "HRRIECEfactor.itx" | ||
+ | |||
+ | MatrixOp/o HRtimeseries = sumRows(HROrg) // HRtimeseries will be n x 1 matrix with nans in the correct places. | ||
+ | |||
+ | HRtimeseries /=HRtimeseries // make all the values in this wave nans or 1s. | ||
+ | |||
+ | HRRIECEfactor*=HRtimeseries // nan the correct time points | ||
+ | |||
+ | RemoveNaNs(HRRIECEfactor) // The Remove Points ipf containing this RemoveNaNs function should automatically be loaded into the PMF experiment via the line #include <remove points> in the PET scatter ipf. | ||
+ | |||
+ | Step 3. In the main results PMF panel, current factor tseries graph, the y waves are named root:pmf_plot_globals:TSeriesFactor1, root:pmf_plot_globals:TSeriesFactor2, etc. for any factor solution. You may choose to simply copy and divide these waves directly (all assuming you are in the same data folder as in step 1): | ||
+ | |||
+ | duplicate/o root:pmf_plot_globals:TSeriesFactor1 TSeriesFactor1RIECDCE | ||
+ | |||
+ | duplicate/o root:pmf_plot_globals:TSeriesFactor2 TSeriesFactor2RIECDCE //... | ||
+ | |||
+ | |||
+ | TSeriesFactor1RIECDCE/=HRRIECEfactor | ||
+ | |||
+ | TSeriesFactor2RIECDCE/=HRRIECEfactor //... | ||
+ | |||
+ | |||
+ | Know that any time you reselect the factor solution (fpeak, number of factors) in the main results panel, you will need to redo this duplication and division. | ||
=== AMS HR factors colored by families hack === | === AMS HR factors colored by families hack === | ||
− | As of PMF version 2.08D, the grouping of HR ions info CH, CHO1 etc families and elemental ratios of factors is built into thePMF "View Results" panel. | + | Versions 3.0 and higher of PET have incorporated data and code to create elemental ratios for high resolution AMS organic data. Information below is only useful for older (2.x versions of PET). |
+ | |||
+ | As of PMF version 2.08D, the grouping of HR ions info CH, CHO1 etc families and elemental ratios of factors is built into thePMF "View Results" panel. For historical purposes, a 'hack' for performing these tasks for use with older PMF versions is described below. | ||
During the examination of factors of High Resolution AMS spectra, it is often helpful to plot the HR spectra summed into chemical 'families', i.e. familyCH which consists of all chemical fragments consisting only of C and H atoms, in a unit resolution dimension. As of November 2013 the following method can be employed. | During the examination of factors of High Resolution AMS spectra, it is often helpful to plot the HR spectra summed into chemical 'families', i.e. familyCH which consists of all chemical fragments consisting only of C and H atoms, in a unit resolution dimension. As of November 2013 the following method can be employed. | ||
Line 833: | Line 1,029: | ||
== Compare PMF Results with External Factors *Step 3* == | == Compare PMF Results with External Factors *Step 3* == | ||
− | Caution: This part of the code is a bit fussy! Please contact [mailto: | + | Caution: This part of the code is a bit fussy! Please contact [mailto:support@aerodyne.com?Subject=Trouble%20with%20Scatter%20Panel Donna] if you have tried the tips here and have trouble getting this panel to work. |
=== Setting up the External Data === | === Setting up the External Data === | ||
Line 842: | Line 1,038: | ||
# '''The tricky part of using this panel is setting up your mass spectra and time series correctly.''' | # '''The tricky part of using this panel is setting up your mass spectra and time series correctly.''' | ||
#* Each wave must have '''either''' the same number of points as in the corresponding dimension of your ''noNaNs_ data matrix'' '''or''' the same number of points as in the corresponding dimension of your ''original matrix''. | #* Each wave must have '''either''' the same number of points as in the corresponding dimension of your ''noNaNs_ data matrix'' '''or''' the same number of points as in the corresponding dimension of your ''original matrix''. | ||
− | #** For example, if your original matrix is 3246 rows and 300 columns and your noNaNs_ matrix is 3200 rows and 268 columns, your "external" mass spectra can have 300 or 268 points; "external" time series can have 3246 or 3200 points. | + | #** For example, if your original matrix is 3246 rows and 300 columns and your noNaNs_ matrix is 3200 rows and 268 columns, your "external" mass spectra can have 300 or 268 points; "external" time series can have 3246 or 3200 points. |
+ | #** As another example, if your original matrix is 100 rows and 320 columns and your "external" mass spectra has 300 points then you may extend the external mass spectra to 320 points, inserting nan values in the last m/z 301 - m/z 320 rows. | ||
#* For mass spectral comparisons, download "full" spectra (usually 300 points) from the [http://cires.colorado.edu/jimenez-group/AMSsd/ AMS Spectral Database] instead of using the shortened ones provided in the 9th Users Meeting template. You should inspect the length of all waves from the Database to make sure that every one has the correct number of points for your work. | #* For mass spectral comparisons, download "full" spectra (usually 300 points) from the [http://cires.colorado.edu/jimenez-group/AMSsd/ AMS Spectral Database] instead of using the shortened ones provided in the 9th Users Meeting template. You should inspect the length of all waves from the Database to make sure that every one has the correct number of points for your work. | ||
# '''IMPORTANT NOTES''' | # '''IMPORTANT NOTES''' | ||
Line 887: | Line 1,084: | ||
= Considerations for Choosing a Solution = | = Considerations for Choosing a Solution = | ||
+ | * This topic is beyond the scope of discussion on this wiki. | ||
= PMF Evaluation Tool Software = | = PMF Evaluation Tool Software = | ||
Line 894: | Line 1,092: | ||
!Date | !Date | ||
!Release Notes | !Release Notes | ||
− | + | !PET ipf Files for Upgrading | |
− | !PET ipf | ||
! | ! | ||
− | !PET Template | + | !PET Igor Template Experiment |
! | ! | ||
− | ! | + | !PMF Executable Package |
|- | |- | ||
− | |3. | + | |3.08E |
− | + | |24 Oct 2024 | |
− | |[http://cires.colorado.edu/jimenez-group/PMFResources/ | + | |[http://cires.colorado.edu/jimenez-group/PMFResources/PMF_v3_00_ReadMe.txt Release Notes v3.00] |
− | | ''a required file --> | + | <!-- | ''a required file '' --> |
− | |[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/ | + | |[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/PMF_v3_08E.zip Zipped PMF file containing these 7 ipfs]: |
− | + | PMF_SetupPanel_v3_08E.ipf<br> | |
− | + | PMF_ErrPrep_AMS_v3_08E.ipf<br> | |
− | + | PMF_Execution_v3_08E.ipf<br> | |
− | + | PMF_ViewResults_v3_08E.ipf<br> | |
− | + | PMF_scatter_v3_08E.ipf<br> | |
+ | PMF_EACode_v3_08E.ipf<br> | ||
+ | SNRAnalysis_v1_02A.ipf<br> | ||
|''or'' | |''or'' | ||
− | |[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/ | + | |[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/IgorPMF_Template_v3_08E.pxt Igor PMF template experiment v3_08E] |
|rowspan="6"|''and'' | |rowspan="6"|''and'' | ||
− | |rowspan="6"|[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/ | + | |rowspan="6"|[http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/PMF_executable_package.zi PMF_executable_package.zip] |
+ | |||
− | <!-- | + | <!-- |
|- | |- | ||
|2.06 BETA | |2.06 BETA | ||
Line 994: | Line 1,194: | ||
--> | --> | ||
|} | |} | ||
+ | Quick review: Igor code called PET (PMF Evaluation Tool) assists in a PMF analysis. Anyone with credentials can download PET; there is no need to request an individual account. When you click on a link to download, a prompt should appear asking for credentials. If you need credentials to download items on this wiki email [mailto:support@aerodyne.com?Subject=Igor-PMF%20Credentials Donna]. The PET code v3.08E has been tested on these versions of Igor: 6.37, 7.08, 8.04, 9.0.6 | ||
+ | A PMF executable file and a license file is required for PET to interface with. The executable file can be downloaded using the PMF_executable_package link above. Originally the PMF executable and license files could be purchased by sending an email to its creator Pentti Paatero (retired U. of Helsinki). The Swiss company Datalystica is now the sole official seller of the multi-linear engine (ME-2) solver package. This ME-2 package will contain a license that is used for the PMF executable. Once this ME-2 package is downloaded, copy and rename the ME2key.key file to pmf2key.key and place this in the same folder as the PMF executable files. Please go to [http://datalystica.com/me-2-solver/ https://datalystica.com/me-2-solver/] to purchase ME-2. | ||
+ | |||
+ | The Igor PET code is not needed to run PMF; PET provides a handy interface to generate and view PMF results. | ||
+ | |||
+ | <!-- | ||
==BareBones Starter Kit== | ==BareBones Starter Kit== | ||
Download the [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/BareBones_PMF_Starter_Kit.zip BareBones_PMF_Starter_Kit.zip] described for [[#Setting up PMF on Your Computer | initial installation of PMF on a computer]]. | Download the [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/BareBones_PMF_Starter_Kit.zip BareBones_PMF_Starter_Kit.zip] described for [[#Setting up PMF on Your Computer | initial installation of PMF on a computer]]. | ||
+ | |||
+ | ==Bug in code previous to version 2.08 (spring 2017) == | ||
+ | The details: the bug is in step 6 of the Prep tab of the PMF_PerformCalc_panel in PET. This bug occurs when the user presses the "Downweight weak, bad m/zs" button and chooses to 'Remove' 'bad' variables (default setting) and 'bad' variables exist. The bug is serious in that during this step some columns in the data matrix *may have been overwritten*. When this happens PMF results could have been degraded. Yet it is very unlikely that any scientific conclusions will be upended. | ||
+ | |||
+ | Regarding the PET + Igor versions when this happened: | ||
+ | - In PET versions previous to 2.05 (2012) and earlier it is unclear how users handled 'bad' variables. It is conceivable that if PET users incorporated a proto-type of later code that handled 'bad' variables then this bug would exist. At least by December, 2012, PET 2.06Beta, the bug existed. | ||
+ | - In Igor 6.x and PET versions 2.05 up to version 3, this bug will not generate an obvious error (even though this bug/error occurred!). | ||
+ | - If you are using Igor 7 OR are using PET versions 2.08-2.09Z or higher, you will get a Wavemetrics generated 'out-of-index' error and the code will stop executing. You would not be able to continue. | ||
+ | |||
+ | The bug is that the code will incorrectly overwrite the right-most columns of the data matrix (corresponding to the highest m/z if the matrix columns are in increasing m/z order (default for AMS Org, HROrg, etc). The number of right-most columns that are incorrectly overwritten is dependent on the SNR (signal to noise ratio) and the column location of 'bad' variables. The more 'bad' variables were flagged, the larger the number of columns that were incorrectly overwritten. The behavior is an 'out-of-index' error; in Igor 6 or PET versions < 3.00 the overwritten (right-most) columns begin to become (incorrectly) identical. If you do not have 'bad' variables or have chosen to downweight instead of remove them, the data matrix is correct and your analysis will not be affected. | ||
+ | |||
+ | Because the right-most columns generally correspond to signal with relatively little variability, the net effect on PMF results is likely minimal. If you combined any other signal (i.e. inorganic) by appending columns to the right of the main organic matrix, it is likely these right-most (i.e. inorganic) signals are compromised. | ||
+ | |||
+ | Note that it is only the data matrix that is potentially compromised. The error matrix was 'correctly' generated. However, if the data matrix is compromised, then the problem is that the right-most, last (bad) columns are paired with semi-random errors and PMF is trying to make sense of it. | ||
+ | |||
+ | Who should check their PMF results? | ||
+ | - Anyone who analyzed data in Igor 6.x or using PET's code, versions before 2.08, and performed the automatic removal of bad variables. | ||
+ | - If you used PET 2.08+ OR used Igor 7+ to perform the error preparation steps, you are fine. | ||
+ | |||
+ | How can you tell if your previously analyzed PMF results were compromised? Several methods are available: | ||
+ | (A) In the history window search for the words "bad/weak columns" | ||
+ | For example | ||
+ | // Removed 3 bad/weak columns. | ||
+ | in the history window indicates that in step 6, 3 columns were removed. | ||
+ | In this case, the right-most 3 columns in the data matrix are possibly incorrect. | ||
+ | The impact of your results could be proportional to the number of columns removed divided by the total number of columns, or probably less than that, if those are high m/z with low SNR. | ||
+ | |||
+ | OR | ||
+ | |||
+ | (B) In the main PMF evaluation panel, select the largest species (for HR ions) or largest UMR m/z that was analyzed. | ||
+ | Notice the time series graph of this species (or m/z) titled "Reconst., measured signal for current species", and the top-most, purple trace. | ||
+ | Now select the second largest specie or m/z. Did the purple trace change? If it did not change, this is an indication that the data matrix is compromised. | ||
+ | |||
+ | OR | ||
+ | |||
+ | (C) Make a table of the no-nans data matrix, the one used as input to the PMF (selected in PET tab “Run”). Scroll to the right-most columns and see if columns begin to be identical. If so, the data matrix is compromised. | ||
+ | |||
+ | |||
+ | What should I do if my PMF results were affected? | ||
+ | It is likely that your scientific conclusions are OKnot faulty. But the best thing to do is to redo the PMF analysis with PET version 2.08 or higher or in Igor 7. In PET, the first tab, you can use the yellow "Clear error prep work' button and start over without having to reload in the original data and error matricies. | ||
+ | |||
+ | PET version 3.0 has been uploaded to the web site and fixes the bug. | ||
+ | |||
+ | --> | ||
= Other Resources = | = Other Resources = | ||
+ | <!-- | ||
* FTP sites with PMF code, documentation | * FTP sites with PMF code, documentation | ||
** [http://www.helsinki.fi/~paatero/PMF/ P. Paatero's list of PMF url's] (working on 22 Mar 2012) | ** [http://www.helsinki.fi/~paatero/PMF/ P. Paatero's list of PMF url's] (working on 22 Mar 2012) | ||
*** The PMF order form (PMFORDER.pdf) can be found from Paatero's site [http://www.helsinki.fi/~paatero/PMF/pmf2.zip in this zip file] | *** The PMF order form (PMFORDER.pdf) can be found from Paatero's site [http://www.helsinki.fi/~paatero/PMF/pmf2.zip in this zip file] | ||
** [ftp://ftp.clarkson.edu/pub/hopkepk/pmf/ P. Hopke's mirror of Paatero's site] (*not* working on 22 Mar 2012) | ** [ftp://ftp.clarkson.edu/pub/hopkepk/pmf/ P. Hopke's mirror of Paatero's site] (*not* working on 22 Mar 2012) | ||
+ | * Sample [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/SampleData.zip data set] for users of the PET where data prep steps have been already performed. Users can unzip the file and load the 4 waves into a PET Igor template file. | ||
+ | --> | ||
* Ingrid's presentation of the PET at the [http://cires.colorado.edu/jimenez-group/UsrMtgs/index.html#ninth 9th AMS Users Meeting] in three parts (1. Overview and PMF Execution; 2. Viewing PMF Results; 3. Using the Scatter Plot Panel) | * Ingrid's presentation of the PET at the [http://cires.colorado.edu/jimenez-group/UsrMtgs/index.html#ninth 9th AMS Users Meeting] in three parts (1. Overview and PMF Execution; 2. Viewing PMF Results; 3. Using the Scatter Plot Panel) | ||
** The [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/IgorPMF_Tutorial_Template_v2_0.pxt tutorial data] used with that presentation | ** The [http://cires.colorado.edu/jimenez-group/PMFResources/Downloads/IgorPMF_Tutorial_Template_v2_0.pxt tutorial data] used with that presentation | ||
− | * | + | |
− | * For HR PMF users [http://cires.colorado.edu/jimenez-group/ToFAMSResources/ToFSoftware/Downloads/PIKA/HR_PMFsupplemental_v1_0.ipf a function to use with PMF factors that will color them according to families]. | + | * Manjula's [http://cires.colorado.edu/jimenez-group/PMFResources/Tutorial/PMFbgd_AMSUsrMtg_2021_todistribute.pdf overview presentation of PMF at the 21st AMS Users Meeting] [http://cires1.colorado.edu/jimenez-group/wiki/index.php/AMSUsrMtgs#21th_AMS_Users_Meeting.2C_The_First_Virtual_Meeting.2C_2021_Jan_19-22_10:00_-_14:00_Eastern_Standard_Time_.28EST.29 (virtual, 2021)] |
− | * For HR PMF users [http://cires.colorado.edu/jimenez-group/ToFAMSResources/ToFSoftware/index.html#Analysis4 the download site for EA mass spectra code that can be used with a single HR mass spectra factor]. | + | * Donna's [http://cires.colorado.edu/jimenez-group/PMFResources/Tutorial/PMFexport20201UsersMtg_Final.pdf presentation of extending AMS data sets beyond Org or HROrg at 21st AMS Users Meeting] [http://cires1.colorado.edu/jimenez-group/wiki/index.php/AMSUsrMtgs#21th_AMS_Users_Meeting.2C_The_First_Virtual_Meeting.2C_2021_Jan_19-22_10:00_-_14:00_Eastern_Standard_Time_.28EST.29 (virtual, 2021)] |
− | + | ||
+ | <!-- | ||
+ | * For HR PMF users of older versions of PET (2.04-2.07) [http://cires.colorado.edu/jimenez-group/ToFAMSResources/ToFSoftware/Downloads/PIKA/HR_PMFsupplemental_v1_0.ipf a function to use with PMF factors that will color them according to families]. | ||
+ | * For HR PMF users of older versions of PET (2.04-2.07) [http://cires.colorado.edu/jimenez-group/ToFAMSResources/ToFSoftware/index.html#Analysis4 the download site for EA mass spectra code that can be used with a single HR mass spectra factor]. | ||
* [http://www.epa.gov/heasd/products/pmf/pmf.html EPA PMF] and its documentation | * [http://www.epa.gov/heasd/products/pmf/pmf.html EPA PMF] and its documentation | ||
− | + | --> | |
== Some PMF Method Papers == | == Some PMF Method Papers == | ||
Latest revision as of 08:27, 30 October 2024
A shortcut to this page is http://tinyurl.com/PMF-guide
The PMF Evaluation Tool (PET) was described in Ulbrich et al., ACP 2009. Please cite the tool with this work in publications in which you have used the PET. The PET consists of several Igor procedure files (ipfs). This wiki serves as the help and documentation for the software. To run PMF with the panel, the PMF executable and associated files, accessed separately, are required (see Section 2, Installing PMF with Igor).
The ipfs were written by Ingrid Ulbrich (formerly of the Jimenez Group, University of Colorado, Boulder) and Donna Sueper (Aerodyne Research and Jimenez Group, University of Colorado, Boulder) and Greg Brinkman (Hannigan Group, University of Colorado, Boulder). Questions about this code can be addressed to Donna.
PMF (Positive Matrix Factorization) was developed by Dr. P. Paatero (retired, Dept. of Physics, University of Helsinki). Links to Paatero's PMF documentation and many PMF method papers can be found in Section 9, Other Resources.
Originally, the PMF executable and license files could be purchased by sending an email to Paatero. Now one can freely download the pmf executable files below, but a license file is needed for it to run. Please email Donna for credentials to download the executables. The Swiss company Datalystica is now the sole official seller of the multi-linear engine (ME-2) solver package that contains a license that is used for the PMF executable. Once the ME-2 package is downloaded, copy and rename the ME2key.key file to pmf2key.key and place this in the same folder as the PMF executable files. Please go to https://datalystica.com/me-2-solver/ to purchase ME-2. One license file can be used for all in the same research group.
This Igor toolkit was intended for use in analyzing AMS data, but there are only few assumptions in the toolkit relating to AMS-type data. Some information on ways to create the necessary waves and matrices from non-AMS data are found in Section 3.2.4 below.
Contents
- 1 A Message to Contributors
- 2 Installing PMF with Igor
- 3 Data Preparation *Step 1*
- 3.1 Exporting the Data by Instrument, Analysis Software Type into PET
- 3.2 For AMS Organics + Other Signals including AMS Inorganics
- 3.3 Further Data Preparation Before Calling PMF (Prep tab in the PET panel)
- 3.3.1 Data Prep Step 0. Select Initial data
- 3.3.2 Recommended Practice: Data Prep Step 1. Apply a Minimum Error
- 3.3.3 Data Prep Step 2. Removing Spikes (data dependent)
- 3.3.4 Data Prep Step 3. Propagation of Smoothing (generally not relevant for ToF data)
- 3.3.5 Data Prep Step 4. Removing Nans, 0 columns
- 3.3.6 Data Prep Steps 5. and 6. Recommended Practice: Downweight "Weak" Variables (m/z's)
- 3.3.7 Data Prep Step 7. Recommended Practice: Downweight Peaks Related to m/z 44 in Frag Table (for AMS and ACSM data only)
- 3.3.8 Data Prep Conclusion: Indicate that all Data Prep Steps have been Completed
- 4 Perform PMF Analysis
- 5 View PMF Analysis Results *Step 2*
- 6 Compare PMF Results with External Factors *Step 3*
- 7 Considerations for Choosing a Solution
- 8 PMF Evaluation Tool Software
- 9 Other Resources
A Message to Contributors
We want to encourage active participation by all users in the evolution of the information contained within this wiki and welcome the addition of content that is beneficial to the community as a whole. However, please DO NOT delete any content from this page!! Significant time, effort, and deliberation has gone into the information contained in this page. Rather than deleting content, please feel free to voice your concerns by posting a comment to the discussion page where others can contribute (please be sure to include a topic to be referenced in responses).
Installing PMF with Igor
This section has been modified with new information on 23 October 2024 by Donna
PMF and Operating Systems
The PMF executable is compiled only for Windows/DOS; it has been used with Windows 11, 10 and older operating systems Windows XP and Windows Vista. Using OneDrive to access the PMF executable file within PET has been problematic for some users. If possible, it is recommended that users access the PMF executable and the Igor template experiment PET from a local drive.
For users with Macs, two options have proven successful. One method is to execute PMF on a Windows computer and then analyze the experiment on a Macintosh. The other method is to execute PMF under a Windows emulator on a Mac. Note that in this latter case, you may need to have the PMF executable somewhere in the c:\ directory; this isn't necessary when running Windows on a PC.
Setting up PMF and PET on Your Computer the First Time
Download the PMF executable package
The file pmf2wopt.exe is the PMF solver. It is downloadable via the PMF executable package below for free; however a username and password is required. Email Donna for these credentials. An older executable file named pmf2wtst.exe also comes with this package but is not typically used. The PMF executable package will also contain a initialization/configuration file named mypmft.ini. This pmf initialization file contains parameters that is used by PET but is not explicitly needed for the executable to run. For every call to the PMF executable PET will generate a 'working' initialization file based on mypmft.ini. This file, named imupmf.ini, is specific to the PET package.
PMF executable file step:
Step 1: Download the zipped PMF executable package and save locally on the computer from where it will run.
Purchase a group PMF license
Along with the PMF executable file a user will need a license file named pmf2key.key. This pmf2key.key file is checked when the PMF executable begins and ensures that a license is valid. If this key file is missing the PMF executable will abort without any analysis taking place. The license, or key file, is purchased through Datalystica, a Swiss company that obtained the rights from the originator of the PMF code, Penti Paatero (University of Helsinki, retired). For PMF usage, one need not purchase SoFi from Datalystica, only ME-2 (multi-linear engine 2), a linear solver for which PMF is a part. This ME-2 package will contain not only executable files such as me2gf*.exe, but will come with a key file named me2key.key. This same key file for ME-2 executable files will also work for PMF executable files, but it must be copied and renamed pmf2key.key and placed in the same folder as the PMF executable files. Every single end user within an organization is not required to obtain a license; users within the same research group can share a license. In Paatero's words "If you drink coffee together in a real room, you are a group." Datalystica does not support PET and PMF but Datalystica employees may be able to answer questions regarding ME-2. The me2key.key and pmf2key.key license file is small and will likely have characters #PMF2KEY#PMF3KEY#ME-2KEY#3127#MSDOS somewhere in the first line.
PMF license file steps:
Step 1: Purchase and download ME-2 solver from Datalystica
Step 2: Copy the file named me2key.key into the PMF executable folder and rename as pmf2key.key
Download PET Igor template experiment or zipped file of ipfs (ipf = Igor Procedure File)
The Igor interface for viewing and evaluating PMF solutions is called PET (PMF Evaluation Tool).
PET step: Step 1: Download the PET Igor template file (IgorPMF_Template_v*.pxt) and save locally on the computer. It is not recommended that the Igor template experiment reside in the same folder as the PMF executable package.
From the top most Igor menu there is a section named "PMF". From this menu there are these options:
- "Perform PMF analysis *Step 1*"
- "View PMF analysis results *Step 2*"
- "Small Pop-Plots Panel for *Step 2*",
- "Compare PMF results with external factors *Step 3*"
Selecting "Perform PMF analysis" will bring up a panel from which you can preform data preparation steps and select data and conditions for running PMF.
Selecting "View PMF analysis results" will bring up a panel from which you can select waves that were previously input into PMF. Once choices have been selected the large PET panel with multiple graphs showing results and diagnostics will appear. Within one PET experiment, it is possible to run PMF either on multiple data sets or under different conditions. This panel allows one to select the PMF results from possible different data sets or conditions.
Selecting "Small Pop-Plots Panel" will generate a mini version of the big PMF results panel. This option was initially intended for users with small screen sizes.
Selecting "Compare PMF results with external factors *Step 3*" will generate a panel from which one can compare PMF results with other data that was not included in the PMF analysis. For example, if one had some gas phase or meteorological measurements with the same time stamps as the input data, this option will show a panel of correlation plots of PMF factors with this 'external' data.
Set up options
It is recommended that users keep the PET Igor template experiment either within a Wavemetrics folder/subfolder or adjacent to the PMF executable package folder. Once PET data preparation steps have been completed and the gold "Being PMF..." button is pushed, the user will be prompted to save the template experiment with a unique name.
One may have many copies of the PMF executable packages on one computer. A file organization suggestion is to set up multiple copies like so: C:\Users\*\Documents\PMF\PMF Executable1, C:\Users\*\Documents\PMF\PMF Executable2 , etc, each folder with a copy of the pmf2key.key file. One can have multiple versions of Igor open at once; this allows one to run multiple copies of PMF (from different Igor experiments) at the same time.
Upgrade PET version options
One can upgrade versions of the PET software without having to redo any PMF analysis steps. Once can download the latest PET ipfs (Igor Procedure Files) below and load the latest versions of the ipfs into an existing PET experiment and then kill all the old ipfs. You may want to kill and recreate PET panels such as the "View Results" and "Setup" panels to take advantage of new features.
If PMF does not Run Properly within PET
0. Make sure that the path to the executable is correct. If red text appears in the section 'PMF Executable File Path' then something is wrong. Look in the folder you created with the PMF2wtst.exe or pmf2wopt.exe file. If necessary go to the tab entitled "PMF defaults and advanced options' and reselect the executable file; then go to the run tab, 'PMF Executable File Path' and reselect the folder with your chosen .exe file.
1. Look in the PMF executable folder that contains pmf2wopt.exe (and/or pmf2wtst.exe) and check for the existence of the files
Matrix.dat and StdDev.dat
- If these files exist, Igor was able to access the pmf executable and key file. Continue with Step 2.
- If these files do not exist, Igor was not able to access the correct folder. Check the existence and spelling of the license file pmf2key.key
2. Look in the PMF executable folder that contains PMF2*.exe for the existence of the file
PMF2.LOG
- If this file does not exist, PMF was not run in this folder. Go to step 3.
- If this file does exist, PMF attempted to run in this folder. Open the file PMF2.LOG (it is a text file).
- Glance down the contents of the file and look for many lines of sequential numbered output, such as
1 rank1 step chi2= 9282.6 Penalty= 1.5287E+04 Flags GF 2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF
- If these lines are in the file, PMF ran successfully on your computer.
- If the lines of sequential numerical output are not in the file, you should find the following lines within this file (note that every line is NOT included here, but the lines selected here are in the order they appear in the file):
2a)
##PMF2 .ini file for: IMUPMF.INI --- PMF Evaluation Panel v2.3 Successfully read task initialization file imupmf.ini titled: ##PMF2 .ini file for: IMUPMF.INI --- PMF Evaluation Panel v2.3
- If these lines appear in the file, PMF found the .ini file. Continue with Step 2b.
- If these lines do not appear, you should see a message about not finding an appropriate .ini file.
- Make sure that this folder contains the file imupmf.ini, provided with the PMF executable package.
- If the file imupmf.ini already exists in the same folder as the PMF2*.exe file, continue with Step 3.
2b)
Successfully opened input file 30 with name MATRIX.DAT Successfully opened input file 31 with name STD_DEV.DAT
- If these lines appear in the file, everything should have run correctly. Look at the rest of the PMF2.LOG file to see whether other errors are reported. If you still encounter difficulty, contact Donna for assistance and attach the PMF2.LOG file to your email.
- If these lines do not appear in the file, you will see a message about PMF not being able to access one of these files. Check that the files are not being used by other programs, delete the file PMF2.LOG, and press the second button on the panel again to see whether PMF runs successfully.
3. Look in the C:\ directory of your computer (C:...\My Documents with the full version) for the existence of the file
runpmf.bat
- If this file exists, Igor was able to write to your C:\ drive. Continue with Step 3a.
- If this file does not appear, Igor was not able to write this file to your C:\ drive. This might be due to high security settings on your computer. You should create a text file with this name (NOT runpmf.bat.txt) and choose to edit it (not open it) with a text editor (e.g., Notepad, WordPad, Emacs, etc.). The file should contain the lines
cd C:\Documents and Settings\*\Desktop\ pmf2wopt imupmf
- NOTE that you must change the path in this example to the path to the folder where you have put the file PMF2*.exe!
3a) Execute runpmf.bat by double-clicking on its icon. You should see the black DOS window pop up, scroll output, and close again.
- If this happens, PMF has run successfully from the batch. In the Igor experiment, press the second button on the panel to see whether PMF runs successfully.
- If the black window does not show scrolled output, continue with Step 3b.
3b) You now need to execute PMF from the command line.
- To access the DOS command line, launch the Command Prompt from your Windows Start menu, located in
Start -> Programs -> Accessories -> Command Prompt
- Change from the path displayed in the prompt to the folder where you put the PMF2*.exe file. Use the command
cd
- to change directories (e.g.)
cd Desktop\PMF
- You can change one directory at a time, or several at a time as shown in the example. To move up 1 directory, use
cd ..
- Be sure that the folder contains the three files
pmf2wopt.exe and/or PMF2wtst.exe imupmf.ini pmf2key.key
- by listing the contents of the current directory, using the command
dir
- Then type this command at the prompt to run PMF:
pmf2wopt imupmf
or
pmf2wtst imupmf
- You should see scrolling output in the command window. The window will not disappear and you can scroll back through the output (which is also saved in the file PMF2.LOG).
- If the output contains many lines of sequential numbered output, such as
1 rank1 step chi2= 9282.6 Penalty= 1.5287E+04 Flags GF 2 rank1 step chi2= 7411.7 Penalty= 1.4084E+04 Flags GF
- PMF ran successfully here. Make sure to close this file after you are finished examining it so that the next time PMF runs it won't complain about writing to this file.
- If the output does not contain these lines, go back to Step 2 of this section to examine errors that might be reported by PMF in the file PMF2.LOG.
Data Preparation *Step 1*
Exporting the Data by Instrument, Analysis Software Type into PET
- PMF analysis requires two 2-dimensional matrices of the same dimensions, a data matrix and a matrix corresponding to uncertainties, which is typically the calculated errors. Both of these matrices must not have any nans (Not a Number) values. In the data preparation section of PET there is a step for the automated removal of rows containing all nans and columns which contain all zeros or nans. The data matrix may contain negative values (PMF solutions will always be >=0) and the uncertainty matrix must have all values greater than zero. In the data preparation section of PET there is a step for the automated setting of a minimum non-zero error values.
The Igor PET PMF tool needs additional 1-dimensional input waves to facilitate the plotting and interpretation of PMF solutions. Required are a 1-dimenensional numeric wave corresponding to index values for the rows, which typically a indicates measurement time in increasing order and 1-dimenensional numeric wave corresponding to index values for the columns, which is typically an m/q in increasing order. For data sets where the columns correspond to high resolution data, i.e. each column indicates a chemical formula such as C2H3O, a 1-dimensional text wave will also be required.
ToF AMS, Q-AMS & ToF ACSM, Q-ACSM
- Regardless as to whether the data has been generated from a quadrupole or ToF, users should select to use MS airbeam correction option for generating the matrices. This is because the error matrices will be generated using the airbeam correction factor if it exists. In the various analysis software packages users may generate the data in units of ug/m3 or Hz. An Igor text file can be generate to facilitate the exporting of all relevant input information for PET.
- Some steps are slightly different between manipulation of the quadrupole and the ToF data sets. In the quadrupole the removal of spikes and the smoothing of data may be necessary, where as in the ToF data sets, neither may be required (due to the fact that the quadrupole only samples one m/z at a time).
- For AMS and ACSM data an additional 1-Dimensional numeric wave of the same dimension as the rows of the data matrix will be generated that contains the minimum error, the uncertainty of measuring of 1 ion per time period. This wave is recommended to be used in the PET data preparation section.
- Buttons in the analysis software facilitate the export of data. Typically, the species exported is organics (Org) or high resolution organics (HROrg), although users have had success in combining this matrix with other signal (i.e. nitrate, etc. or in analyzing binned raw spectra (un-speciated).
In ToF AMS Analysis Software (SQUIRREL/PIKA)
- Buttons are provided for the export of data to PET.
- It is important to know that the CE and RIE are *NOT* applied to the Org and Org error matices and HROrg and HROrg error matrices if you choose unit so ug/m3. The general convention is that whenever Squirrel or Pika outputs anything with an m/z dimension the units are NO3-equivalent. By using nitrate equivalent mass, we are scaling all ions equally (i.e. not scaling differently according to which species each ion contributes to). This means that the mass spectra in ugm3 units are identical to those in Hz units except for a single scaling factor. The thinking behind this convention is that whenever we have an m/z dimension, such as with an average mass spectra, we can sum the unit resolution at any nominal mass to compare to the raw spectra.
In the Q-AMS Analysis Software (James')
- Updated Q-AMS analysis software will have buttons and other controls to easily generate * Be sure to use v1.41 or later of the Q-AMS Analysis Software ("James' Program"). Corrections have been made since earlier versions to the error calculation routines!
- Download from Qi Zhang's website Extract Waves&matrices v 1.1.ipf (note that v1.2 is for Squirrel/HR data) and include it in your experiment with James' software.
- Call org_mats, which will calculate a data matrix (organics_MS) and error matrix (organics_MS_err) in root: in your expeiment, and save these matrices along with the timeseries for organics, sulfate, nitrate, ammonium, and chloride in a file called "WavesMatricesForOrganicAnalysis.itx" (Igor should prompt you for the folder where you want to save the data). All of the data are saved in ug/m3 with all corrections (CE, RIE) applied.
- You may want to load the saved waves into a new experiment to run PMF.
- You'll also need to include your time series wave (t_series) and a wave of the m/z's in the matrix (amus).
Recommended Practice: Removing Spikes (generally, for Quadrupole data)
- "Spikes" in the time series of an m/z can occur in Q-AMS data from large but infrequent particles during the scanning of the quadrupole. If such spikes have a common source with a factor that can be retrieved by PMF, they may increase the variation of that factor profile and additional factors may be found that represent this variation, but not a physically-meaningful, separate component. The "excess signal" from these spikes can be subtracted from the spikes and the average mass spectrum of the spikes examined. See Zhang, Q. et al., ES&T, 2005.
- Note that if you remove "excess signal" in the method of Zhang et al. 2005 and leave the error values for these points unchanged, you are automatically "downweighting" these points in PMF. This is appropriate because the replacement value for the original spike is not known as well as the values for points without spikes.
Optional Practice: Smoothing
- Smoothing can be used to reduce high-frequency noise in the data that could also be fit as additional factors. If you smooth the data, be sure to propagate this smoothing in the error matrix (not added to the wiki yet).
Tofware for ToF CIMS and ACSM data
Controls in Tofware allow the export of all necessary Igor waves into one external file.
For AMS Organics + Other Signals including AMS Inorganics
Setting up the signal and error matricies for org+inorg is not terribly difficult. The interpretation is much more complicated and nuanced than for a PMF analysis on only the organics. One can look through 2009 - 2019 AMS Users Meeting talks by searching for "PMF" to review many options that have been presented and subsequently published.
Further Data Preparation Before Calling PMF (Prep tab in the PET panel)
- The following steps are recommended for AMS datasets and follow the practices laid out in Ulbrich et al., ACP, 2009 (with more detailed references in each section below). Note that the only mandatory step is step 0 (importing data into the experiment and selecting it) and step 4, (Deleting NaNs/zeros). More extensive documentation on use of the remove/delete nan functions is given in the header comments in the ipf pmf_ErrPrep_AMS_v*.ipf.
Before running these functions, you may want to rename your error matrix so that it has fewer than 12 characters. Each data prep step lengthens the wavename and the code will complain if the name gets too long.
Data Prep Step 0. Select Initial data
- From the top Igor menu Data - Load Waves load in the Igor file that you exported from the analysis software - Tofware, Squirrel, etc. It is often good practice to create a data folder to contain these newly imported data waves.
- Select the data folder containing the PMF input and subsequent DATA, ERROR, etc waves.
Recommended Practice: Data Prep Step 1. Apply a Minimum Error
- Ions arrive at the mass spectrometer detector with a Poisson distribution. The error for a counted number of ions is sqrt(counted number of ions). The smallest number of ions we can count in one run is, of course, zero ions, but perhaps there was one and it was missed. The error for counting zero is sqrt(0), but an error of 1 would be more appropriate in this case. Hence a minimum error threshold of 1 ion is set.
- Within this step the user must choose the mass spec detector type. For ToF detectors, there is an m/z dependence to account for the unequal 'pushing' of the ions into the detector. Small m/z ions move faster than large m/z at an approximate rate of square root of the mass. This phenomena is sometimes referred to as an 'm/z duty cycle' correction. The application of a minimum error must undo this duty cycle correction if it was applied to the data to reflect true ions counted. In practice then, when one selects the AMS-ToF for the MS type, a sqrt(28)/sqrt(m/z) factor is temporarily multiplied to the error matrix. For AMS-quad data and for Tofware-processed CIMS data, this undoing of the correction factor is not needed.
- The one ion wave needs to be in the same units as the data and error matrices. If the data and error matrices are in units of Hz and the data acquisition (for AMS the actual time measuring i.e. in the 'MS open' mode) for each run is 1 second, then the one ion wave is simply a wave containing all 1s. If each run had an MS open mode lasting one minute, the wave would consist of 1/60. If the data and error matrices are in typical AMS units of ug/m3 then one needs to convert from Hz using the flow rate, ionization efficiency, AB correction factor, etc. This will have been done for you in the PMF export step in Squirrel, Pika, ACSM Tofware code.
- By pressing the "Apply minimum error" button the function produces the following waves:
- The error matrix with minimum error applied called nameOfWave(errMx)+"_min" (e.g., Orgerr becomes Orgerr_min)
See also discussion of Ulbrich et al., ACPD 2008, P. Paatero comment (p. S5730) and Author response (p. S11960)
Data Prep Step 2. Removing Spikes (data dependent)
- By pressing the Remove spikes (optional) button new data and error matrices will be generated with spikes removed.
Data Prep Step 3. Propagation of Smoothing (generally not relevant for ToF data)
- Any smoothing of the data matrix must be propagated in the error matrix. The function pmf_err_propogateSmooth propagates box or Gaussian smoothing.
Some notes about specifying the smoothing that was performed for the data:
- Allowed types of smoothing are "box" and "Gaussian".
- The type of smoothing that is done in the AMS software is selected in the Misc tab.
- The number of points used in box and Gaussian smoothing is used as defined in Igor.
- For box smoothing, the number of points refers to the size of the box. I.e., smoothing that includes 1 adjacent point on each size is 3-point smoothing.
- For Gaussian smoothing, the number of points refers to the number of adjacent points used. I.e., smoothing that includes 1 adjacent point on each side is 1-point smoothing.
- By pressing the "Propogate smoothing to the error mx" button the function produces a wave with the propagated error called nameOfWave(errMx)+"Prop" (e.g., OrgMSerr_Min becomes OrgMSerr_minProp).
Data Prep Step 4. Removing Nans, 0 columns
- This step is required for all data.
- The data matrix produced in the previous steps may have NaNs in all rows from bad runs and 0 values in columns with good runs that have no fragments of interest (i.e. Argon, or m/z 40 for AMS data). All of these rows and columns need to be removed before running PMF.
- The function produces a wave with the propagated error called nameOfWave(errMx)+"_noNans" (e.g., OrgMSerr_Min becomes OrgMSerr_min_noNans).
Data Prep Steps 5. and 6. Recommended Practice: Downweight "Weak" Variables (m/z's)
- Any m/zs that have low signal-to-noise ratio (SNR) may, in fact, have more noise than signal. If these m/zs contribute enough Q, PMF tries to fit the noisy data. In this way, the inclusion of such m/zs can be detrimental to the PMF analysis. If the error associated with these m/z 's is increased, the Q-contribution (residual/error) is decreased, "downweighting" these points' contribution to the fit. m/zs with SNR<0.2 are considered "bad" by Paatero and Hopke (2003) and should be removed or strongly downweighted (factor of ~10). m/z 's with 0.2<SNR<2 are considered "weak" and should be downweighted (factor of 2-3).
- When a user presses the "Calculate, view SNR panel" A separate panel appears that shows time series, errors and SNR calculations. This tool is handy to examine ions that are deemed to be "bad" or "weak". This panel is informational only and is used in step 6.
- When a user presses the "Downweight weak, bad mzs" button the code will generate new error matrices that have the bad and weak m/z columns adjusted. The function generates a new error matrix called nameofWave(errMx)+"Wk" (e.g., noNaNs_orgMSerr_minProp would become noNaNs_orgMSerr_minPropWk).
See also Paatero, P., and Hopke, P. K.: Discarding or downweighting high-noise variables in factor analytic models, Anal. Chim. Acta, 490, 277-289, 10.1016/s0003-2670(02)01643-4, 2003. Abstract
Data Prep Step 7. Recommended Practice: Downweight Peaks Related to m/z 44 in Frag Table (for AMS and ACSM data only)
- In the default fragmentation table, the information in m/z 44 is repeated 6 or 7 times (in m/z 's 16, 17, 18, 19, 20, (28 in the Aiken et al. 2008 revision), and 44) with different proportionalities. PMF fits correlations, regardless of the magnitudes of the signals. Repeating the information of m/z 44 several times implies that it's really (x6 or 7) important, which it isn't! It is possible to downweight the columns of these m/z 's so that in total they only contribute the m/z 44 signal once. (It would be possible to remove the repeated information and replace those columns after running PMF, but we think that downweighting them is logistically simpler.)
- There are two buttons, one for UMR AMS or ACSM data and another for HR AMS or ACSM data. Once you press the button the List of duplicate ions will be filled in. If you are analyzing HR data and using the default HR organics frag waves, the equivalent set of HR ions of (the mz44list) correspond to O, HO, H2O, CO and CO2. If you are analyzing organic UMR data the default list will include 16, 17, 18, 19, 20, 28, 44. Then press the Downweight duplicated columns button.
- The function generates a new error matrix called nameofWave(errMx)+"44" (e.g., noNaNs_orgMSerr_minPropWk would become noNaNs_orgMSerr_minPropWk44).
Data Prep Conclusion: Indicate that all Data Prep Steps have been Completed
- Press the gold "Select prepared waves for PMF" button. The second tab of the PET panel, "Run", will become active.
Perform PMF Analysis
- At this point you are ready to indicate where the PMF executable file resides, where the prepared data and error matrices reside within the Igor experiment and what PMF solutions you are wanting to investigate.
Press the "Create path to PMF executable and batch files" button. A window will pop up asking you where the PMF executable file resides. The path will be displayed once a user has selected a folder. If this path text is red, something is amiss and the code couldn't find the PMF executable.
If you pressed the gold button "Select prepared waves for PMF" in the previous data prep tab, entries in the data section will be filled in for you.
- Choose model error
- It is very rare for users to change this model error value from the default of 0. (PMF increases the errors provided by newError = oldError + modelError*dataValue)
- Choose model error
- Choose the type of PMF analysis
- Exploration will run PMF for a range of number of factors and FPEAKs or SEEDs. This is the typical use for exploring a dataset and comparing many solutions.
- Bootstrapping explores the uncertainty of one solution (i.e., one number of factors at one fpeak for one seed). This is usually a final step run only on the solution you have selected from the exploratory analysis.
- Choose a range for number of factors.
- When checking to make sure that everything runs properly, you may want to run just one case (Min p = 2, Max p = 2; p is the number of solved factors).
- Recommended Practice: Run cases with 1 factor to have a context for the meaning of the 2-factor solution.
- In the Bootstrapping mode, only the "min p" is read; the "max p" is ignored.
- Choose FPEAK or SEED values
- "FPEAK" is a tool used to explore rotations of the solutions of a given number of factors. Note that FPEAK does not explore all possible rotations of a solution. FPEAK = 0 does not apply any rotational forcing. Non-zero values of FPEAK create near-zero values in the factor profiles (mass spectra) or time series. More information about FPEAK can be found in the PMF Users Manual Part 1 (pp. 9,12,14,21) and Part 2 (p. 24), and in several papers by P. Paatero (see Other Resources).
- A good first set of FPEAK values is -1.0 to +1.0 with a delta value of 0.1 or 0.2. For a full analysis, a wide enough range of FPEAKs to achieve Q/Qexp of at least 1% above the minimum value is recommended.
- In Exploration mode when varying the fpeaks, the Seed is set in headers in the PMF_Execution...ipf file: constant DEFAULT_SEED = 0
- In Bootstrapping mode, only the "min fpeak" is read; the "max fpeak" is ignored.
- "SEED" is a tool used to choose different random starts (initial values) for the PMF algorithm. Using different seeds may lead to solutions in different local minima (Q/Qexp) in the solution space. One set of solutions may have more physical meaning than another, or multiple sets may make physical sense. It is impossible to test all start values, but testing many seeds may give an indication of local minima for your dataset. More information about seeds can be found in the PMF Users Manual Part 1 (p. 11) and Part 2 (p. 16).
- Run seeds from 0 to your preferred maximum with a delta value of 1.
- In Exploration mode when varying the Seed, the fpeak is set in the headers of the PMF_Execution...ipf file: constant DEFAULT_FPEAK = 0
- Exploring Seeds with a non-zero fpeak should be done with caution, as the "push" of a magnitude of fpeak for one area of local minima may be different than the "push" of the same magnitude of fpeak for a different area of local minima.
- In Bootstrapping mode, the Seed is set in headers in the PMF_Execution...ipf file: constant DEFAULT_SEED = 0
- "FPEAK" is a tool used to explore rotations of the solutions of a given number of factors. Note that FPEAK does not explore all possible rotations of a solution. FPEAK = 0 does not apply any rotational forcing. Non-zero values of FPEAK create near-zero values in the factor profiles (mass spectra) or time series. More information about FPEAK can be found in the PMF Users Manual Part 1 (pp. 9,12,14,21) and Part 2 (p. 24), and in several papers by P. Paatero (see Other Resources).
- Select checkboxes
- Run PMF in background Each execution of PMF (see Exploration Mode Summary) creates a black DOS window that pops up. If the box is not checked, this window "grabs the focus" and makes itself the top window. This makes it hard to use the computer for anything else. If the box is checked, the window will not grab focus, but Igor and your computer's CPU will be busy.
- Save experiment before and after PMF has finished It is generally a good idea to save the experiment before calling PMF so that all the data prep steps and user selections will be saved before attempting to call the PMF executable. Once the code has finished all its calls to the PMF executable it is a good idea to save the experiment because all the PNF results are now saved as waves within the experiment.
What the Software Does When You Press the Button...
Exploration Mode Summary
The software will execute PMF once for every combination of number of factors and FPEAK/seed. So if you run 1-6 factors and 5 FPEAKs, PMF will run 6x5=30 times. Each run starts a new black DOS window that will close when the run is completed. The duration of each run is printed in the history at the end of each run. In general, runs which solve for more factors and runs with FPEAK farther from 0 take longer. The code runs all of the FPEAKS or seeds for one number of factors, then advances to the next number of factors (e.g., run 1 factor with each of 5 FPEAK values, then 2 factors with each of 5 FPEAK values, etc.).
When the entirety of sets of calls to the PMF executable has completed many diagnostic values are calculated. The panel from where you selected "Begin PMF analysis with these choices" will be killed and a new panel "View PMF Analysis Results" will be created with PMF inputs filled in for you. Select options for possible additional calculations. For example for AMS/ACSM HR Organic data, calibrated elemental ratios can be generated for each factor.
A little more detail
The software writes the files
C:\delete_log.bat C:\runPMF.bat
and writes the DataMatrix and ErrorMatrix as MATRIX.DAT and STD_DEV.DAT, respectively to the folder with your PMF Executable. The software also writes a file to that folder called STD_DEV_PROP.DAT, which has the same number of points as the DataMatrix and in which every element is equal to the ModelError. These files will be read by the PMF executable.
The software then enters a pair of nested loops in which the following steps occur:
- for each number of factors
- for each FPEAK or SEED
- use the file
- for each FPEAK or SEED
mypmft.ini
- as a template to create the file
imupmf.ini
- which is used as the control file for PMF.
- The convergence criteria for completing the PMF calculations can be set in the 3rd tab of the PMR_PerformCalcPanel "PMF defaults & advanced options".
//Default settings for PMF iteration convergence proportional to Qexp constant ITERATION_CHI2_PROP_QEXP = 0 constant FIRST_ITER_LEVEL_PROP_CONST = 2e-6 constant SECOND_ITER_LEVEL_PROP_CONST = 2e-6 constant THIRD_ITER_LEVEL_PROP_CONST = 1e-6
- You can read more about convergence criteria and large datasets in the PMF Users Manual Part 1 (pg. 10) (see Other Resources).
- Delete the old PMF2.LOG file by running delete_log.bat
- Execute PMF by running run_PMF.bat.
- Wait for PMF to complete its run.
- Load PMF output (including log file and factors)
At the completion of the loops, the software calculates some statistics from the output and then creates a panel to select PMF results for viewing.
Bootstrapping
The bootstrapping mode is developed after the method described in the EPA PMF v3.0 Users Manual Sect. 6.4 (see Other Resources). The bootstrapping method is used to estimate the uncertainty in both the factor mass spectra and time series. This is achieved by running PMF on the full dataset once (from one PMF solution (i.e., combination of number of factors and fpeak) at a time) and then making a series of variations (the number is specified by the user when selecting the bootstrapping mode) in which a subset of the original rows (mass spectra) are randomly replaced by other rows from the original matrix and running PMF on each of these.
For each new PMF case ("bootstrapped case"), the resultant factors are compared to those from the original dataset and assigned as "matching" the original factor with which it has the highest correlation. Bootstrapped cases in which each bootstrapped factor was matched to exactly one of the original factors (i.e., there is a one-to-one mapping between original factors and those from the individual boot-strapped cases) are retained for calculation of the average mass spectrum and time series of bootstrapped factors. Plots of the original factors and the average bootstrap factors with 1-sigma variation bars are produced automatically.
EPA PMF Users Manual recommends doing 100 bootstrap runs for final results.
A little more detail
All output from the bootstrapping runs is saved in folder root:pmf_bootstrap:.
Row (mass spectrum) replacement is performed by using the StatsResample function in Igor to select rows for replacement. The row values are then sorted in increasing order as a convenience.
- The 2d wave RowsToBeReplaced records the rows to be used in each bootstrapped case. Each column represents one bootstrap case. Each column lists the rows of the original matrix included in that bootstrapped case.
- The 2d wave ReplacementHistogram counts the number of times that each original matrix row was used in a bootstrap case. Each column represents one bootstrap case. Summing the rows of this matrix gives the total number of times that each original matrix row was was used in the bootstrapping cases.
The assignment of bootstrapped factors to the factors from the original case is made by Pearson R correlation. Factors are assigned only on the basis of mass spectral comparison, and each factor is assigned to one of the original factors.
- Note that this is different than in EPA PMF. Our code does not have a criterion for the lowest allowable correlation between bootstrapped and original factors. In EPA PMF, factors that fall below this limit are "unmapped"; no factors are "unmapped" in our code.
- Note that if the original case has factors that are very similar to each other, the assignment of the bootstrapped factors may be incorrect or ambiguous. No current work has been done to give guidance as to what "very similar" means. No sanity checks are made in the code for this type of situation.
- The 3d wave FactorProfile_Rval stores the correlation between the boostrapped and original factors. Rows represent the factors from the original case, columns represent the factors from the bootrapped cases, and layers represent each bootstrapped case.
- The 2d wave FactorProfileSort contains the number of the original factor to which each boostrapped factor (row) has been matched in the bootstrapped case (column). Columns which contain (e.g., in a case with three factors) "0, 1, 2" have factors which were uniquely matched to the columns in the original case; these will be included in the averages of mapped factors. Columns which contain duplicate entries (e.g., "0, 1, 0") have multiple factors that were matched to the same original factor; these cases will not be included in the averages of the mapped factors.
Note that the criteria for including bootstrapped cases in calculations of the average bootstrapped factors is different in our code than in EPA PMF. Bootstrapped cases in which two or more factors are mapped to the same original factor are not included in the averages in our code. In these instances, the whole bootstrapping case is rejected before calculating averages. In EPA PMF, all bootstrapped factors mapped to the same original factor are included in the average.
Important: At the present time the plots produced by the bootstrapping code expect to find the waves noNaNs_amus (for plotting profiles) and noNaNs_t_series (for plotting time series). If these are not the names of your waves that describe the columns and rows, respectively) of the input matrix, you should duplicate your row and column description waves to have these names.
When PMF seems to work for some solutions but not others
In the "Perform PMF calculations" panel, "PMF defaults and advanced options" tab, in versions 3.08D and higher there is a checkbox named "Save the log file for every iteration". If this is checked, then after every call to PMF the PET code will copy the PMF2.LOG file and rename it PMF2_x_y.LOG where x and y are the current number of factors and fpeak. One can then look through these files to see what problems may have arisen for a specific case.
Running Two Simultaneous Analyses on Multi-Processor Computers
You can run two or more PMF analyses (in two separate experiments) on the same computer simultaneously if you have multiple processors. Each analysis will run at the same speed as on a single-processor computer (or when one analysis is run on the dual processor computer). The PMF executable is not "multi-processor aware," meaning that it can not utilize both processors simultaneously for one PMF run.
To run two simultaneous analyses with the PET, you'll need two directories on your computer with the PMF .exe, .key, and mypfmt.ini files. The directory names must end in "1" and "2," respectively. For example, you could have
- C:\Users\*\Documents\PMF\PMF Executable1
- pmf2wopt.exe
- pmf2key.key
- mypmft.ini
- C:\Users\*\Documents\PMF\PMF Executable2
- pmf2wopt.exe
- pmf2key.key
- mypmft.ini
Each experiment running PMF must use a separate Executable directory. While a PMF analysis associated with a PMF Executable directory is running, a file called "PMFrunning.txt" exists in that directory. If you try to run your second PMF analysis using the same Executable directory, the PET will give you an error. You can choose a different Executable directory and press the button to start the analysis.
Convergence
PMF uses a variety of metrics to assess the progress of finding a solution. Some of these metrics are based on parameters that were initially optimized for AMS data, and hence, may not be appropriate for other data sets. The last tab in the PMF_PerformCalcs panel has a section entitled Q convergence criteria which lists some defaults; users may modify these values to achieve convergence. The log text file, PMF2.LOG generates information about the progress for finding a solution. As an example the following is an excerpt that was written to this file:
The log text file, PMF2.LOG generates information about the progress for finding a solution. Below is an excerpt from a file where a PMF solution didn't converge.
299 rank1 step chi2= 6.22372E+06 Penalty= 2.5296E+04 Flags GF 300 rank1 step chi2= 6.22364E+06 Penalty= 2.5295E+04 Flags GF
************************************************************ 1 */ Iteration interrupted because maximum step count *\ has been exceeded in the robust mode. NO CONVERGENCE? ************************************************************ 1 300 iter-steps in all. Final chi2= 6223643.5000 in the robust mode.
As a practical matter, the last version of the PMF solution is saved within PET and users can view this last result for this PMF run. However, because it did not converge graphs in the main PMF panel will be grayed out and the Q/Qexp value will be blank.
View PMF Analysis Results *Step 2*
Once PET has finished calling PMF and has calculated some diagnostics the "Perform PMF" panel will close and a "View Results" panel will appear with the previous selections of folders and waves preselected. For AMS/ACSM high resolution organic data additional calculations regarding elemental summing & ratios can be selected. Press the gold button "View PMF analysis" to proceed with generating the large PMF results panel.
Controls
Factor Space and Fpeak Slider
Save Solution Space Waves
Select Factor
Select species
Select Multiple Factors or Fpeak/Seed
Mass Fraction
Mass Fraction/Variance
RotMx
Q Plots
Q versus Number of Factors
Q versus fpeak/seed
Factor Graphs
Time Series
Profiles (usually Mass Spectra)
Reconstruction and Residuals
Total Reconstruction
Total Residuals
Current Species Reconstruction and Residual
Scaled Residuals
All Species Scaled Residuals
Current Species Scaled Residuals
"RR" plot
AMS loadings and the RIE, CE and CDCE factors
When exporting species like Org and HROrg from all versions of Squirrel and Pika, users should be aware that the RIE or the CE or CDCE factors are not applied to the 2-dimensional data. The reasons for this rule are somewhat historical. Basically, whenever a result in Squirrel or Pika retains an MS dimension, as in the export of the 2D wave for PMF, which has dimensions of measurement time and UMR or HR mass values, the RIE and CE (or CDCE or HR CDCE values) are not applied.
Users may wish to compare factors and loading from PMF results to species time series loadings with the RIE or CE applied. Below is a guide for applying default RIE and HR CDCE to source factors within a PET experiment; similar operations can be used for CE or UMR CDCE. Be aware that by performing this operation, the user assumes that the RIE and CE (or CDCE or HR CDCE) is the same for all factors. This may not reflect reality. For example RIE calibrations and CE (or CDCE or HR CDCE) have been generated for bulk ambient organic aerosol; the RIE and CE for BBOA and COA, for example, may not be identical.
There are 3 main steps to incorporating the RIE and CE (or CDCE or HR CDCE) to PMF factors.
1. Generate a 1D wave matching the time series wave containing a multiplicative factor of the RIE and CE or CDCE and import it to the PMF experiment.
2. Remove points in this 1D wave that matches the nans removed in the data prep step of PMF.
3. Divide this wave to your time series factor waves in the PMF experiment.
More details and example of commands are below.
Step 1. Within your Pika experiment, the HR CDCE is root:HRCE_fphase; for UMR data it is root:CE_fphase. The RIE values can be found in the UMR or HR batch tables. Here are commands you can use and modify as needed.
Duplicate/o root:HRCE_fphase root:HRRIECEfactor // Use a similar command for UMR. This wave should be the same length as the t_series wave
root:HRRIECEfactor *= 1.4 // 1.4 is the RIE of HROrg in this data. We will divide the PMF factors by this HRRIECEfactor wave.
Save/T/M="\r\n" HRRIECEfactor as "HRRIECEfactor.itx" // Save this wave as separate itx file.
In this example I saved this file in the same place as the exported PMF text file (that you had loaded into your PMR experiment at the beginning of that analysis). You can also load this wave into your PMF experiment via the browse experiment command without saving it as an external text file.
Step 2. There are a few ways to accomplish this step. Within your PMF experiment and data folder containing the original data (before noNan_* versions of the data was created, create a time series wave to flag when rows (time points) were nan.
Load the HRRIECEfactor wave from step 1 and multiply this wave by it.
LoadWave/T "HRRIECEfactor.itx"
MatrixOp/o HRtimeseries = sumRows(HROrg) // HRtimeseries will be n x 1 matrix with nans in the correct places.
HRtimeseries /=HRtimeseries // make all the values in this wave nans or 1s.
HRRIECEfactor*=HRtimeseries // nan the correct time points
RemoveNaNs(HRRIECEfactor) // The Remove Points ipf containing this RemoveNaNs function should automatically be loaded into the PMF experiment via the line #include <remove points> in the PET scatter ipf.
Step 3. In the main results PMF panel, current factor tseries graph, the y waves are named root:pmf_plot_globals:TSeriesFactor1, root:pmf_plot_globals:TSeriesFactor2, etc. for any factor solution. You may choose to simply copy and divide these waves directly (all assuming you are in the same data folder as in step 1):
duplicate/o root:pmf_plot_globals:TSeriesFactor1 TSeriesFactor1RIECDCE
duplicate/o root:pmf_plot_globals:TSeriesFactor2 TSeriesFactor2RIECDCE //...
TSeriesFactor1RIECDCE/=HRRIECEfactor
TSeriesFactor2RIECDCE/=HRRIECEfactor //...
Know that any time you reselect the factor solution (fpeak, number of factors) in the main results panel, you will need to redo this duplication and division.
AMS HR factors colored by families hack
Versions 3.0 and higher of PET have incorporated data and code to create elemental ratios for high resolution AMS organic data. Information below is only useful for older (2.x versions of PET).
As of PMF version 2.08D, the grouping of HR ions info CH, CHO1 etc families and elemental ratios of factors is built into thePMF "View Results" panel. For historical purposes, a 'hack' for performing these tasks for use with older PMF versions is described below.
During the examination of factors of High Resolution AMS spectra, it is often helpful to plot the HR spectra summed into chemical 'families', i.e. familyCH which consists of all chemical fragments consisting only of C and H atoms, in a unit resolution dimension. As of November 2013 the following method can be employed.
(1) Add the ipf named HR_PMFsupplemental_v1_1.ipf to your PMF experiment.
(2) From the pika experiment from which the HR spectra was generated, copy the entire folder named HR into the PMF experiment. This can easily be done via the Data Browser - Browse Experiment feature in Igor. Specifically, in your PMF experiment, press the Browse Experiment button, select the pika experiment, and then in the right hand side of the window click and drag the HR data folder over to the left hand side, so that the cursor points to the root folder. The PMF experiment should now have the waves root:HR:HR_family_B copied into it. Not all the waves in this folder will be used, but it is easiest to simply copy them all.
(3) Execute the following line, or analagous, to calculate and plot the HR family grouped spectrum for a currently chosen PMF factor of interest. BreakHRSpectraIntoUMRFamilies(root:pmf_plot_globals:profilefactor1, root:noNaNs_ExactMassWaveNoIso, root:noNaNs_ExactMassTextNoIso, 1)
In the line of code above, the first parameter indicates which currently displayed HR factor is to be calculated. The second and third parameters indicate the HR mass value and the Mass Text identifier waves of the HR spectrum. These waves should be exactly the same as those you chose in Step 2 of the PMF tool, Description of COLUMNS of Data matrix (numbers) and (text). The last parameter is a simple true/false flag for indicating whether or not to plot and color the results.
Subsequent executions of the line, i.e. BreakHRSpectraIntoUMRFamilies(root:pmf_plot_globals:profilefactor2, root:noNaNs_ExactMassWaveNoIso, root:noNaNs_ExactMassTextNoIso, 1) will generate a new HR colored spectrum for the second factor (of the currently selected PMF solution).
Compare PMF Results with External Factors *Step 3*
Caution: This part of the code is a bit fussy! Please contact Donna if you have tried the tips here and have trouble getting this panel to work.
Setting up the External Data
- Check that the waves NaNsWv_amus" and "NaNsWv_tseries" (these were created when you deleted NaNs from your original matrix) are in the datafolder where your PMF output is being saved. If they're not already there, you'll need to find them and copy them to this location.
- Versions 2.02 and lower used the strings "NaNsList_amus" and "NaNsList_t_series" to record the location of deleted NaNs. If you are upgrading experiments, make sure that these strings are in the folder where your PMF output is being saved; "NaNsWv_amus" and "NaNsWv_tseries" will be created and used for future calculations.
- You'll need separate folders (in root: ; they cannot be in a lower directory) for mass spectra and time series you want to use for comparison to the factors.
- The tricky part of using this panel is setting up your mass spectra and time series correctly.
- Each wave must have either the same number of points as in the corresponding dimension of your noNaNs_ data matrix or the same number of points as in the corresponding dimension of your original matrix.
- For example, if your original matrix is 3246 rows and 300 columns and your noNaNs_ matrix is 3200 rows and 268 columns, your "external" mass spectra can have 300 or 268 points; "external" time series can have 3246 or 3200 points.
- As another example, if your original matrix is 100 rows and 320 columns and your "external" mass spectra has 300 points then you may extend the external mass spectra to 320 points, inserting nan values in the last m/z 301 - m/z 320 rows.
- For mass spectral comparisons, download "full" spectra (usually 300 points) from the AMS Spectral Database instead of using the shortened ones provided in the 9th Users Meeting template. You should inspect the length of all waves from the Database to make sure that every one has the correct number of points for your work.
- Each wave must have either the same number of points as in the corresponding dimension of your noNaNs_ data matrix or the same number of points as in the corresponding dimension of your original matrix.
- IMPORTANT NOTES
- The reason for the restriction on the number of points in "external" comparison waves is the following: After you select the datafolders for the external mass spectra and time series, the code makes a folder inside each of these folders called "noNaNs". Each wave in the external datafolder is copied to the new directory. Then the code checks whether the waves have the same number of points as the same dimension in the matrix used to run PMF; if so, no change is made to the wave. If not, the code _assumes_ that the wave has the dimension of the original matrix (which it doesn't know about) and therefore uses the string "NaNsList_amus" or "NaNsList_t_series" to delete the rows that it believes were NaN in the original dataset. (It's ok if these waves still have NaNs (e.g., missing points in data from another instrument when the AMS data was good); only the points where both the factor and external waves have valid data are included in the correlation calculation.)
- Because of some internal coding restrictions, time series waves for comparison cannot include the string "series" in their name; such waves will not be created in the noNaNs folder.
- Each of the mass spectral and time series waves must be 1-dimensional. This means that you cannot use the waves from root:pmf_plot_globals: called TseriesFactor1, TseriesFactor2, etc. since these are 2-dimensional waves.
- Old versions of the mass spectra in the AMS Spectral Database have m/z = point number, meaning that point 0 = m/z 0. This is not usually the way that AMS matrices are saved from James' software or from Squirrel, so you may need to delete 1 point from the beginning of each spectrum that you use from the Database. This is important to check, or else you correlate the wrong m/z's with each other.
Choosing the Folders with External Data
- The first time you want to calculate factors you can do so by pressing the "External Data Panel" button on the main panel or by choosing from the PMF pulldown menu "Compare PMF results with External Factors *Step 3*".
- Note that after you have accessed the "External Data Locations" panel, pressing the "External Data Panel" button on the main panel will not bring you back to the selection panel again. To choose different folders or force recalculations you must access the selection panel from the PMF pulldown menu.
- Select your external data folders.
- Other choices from the pulldown menus include
- "No external data of this type": The PET will not attempt to calculate correlations between factors and external data of this type.
- "Update List": If after calculating factor correlations you wish to add new external waves of this type, choose this option to add the correlations for the new waves do your existing list of correlations.
- Other choices from the pulldown menus include
- Press the button to proceed.
What the PET Does (and how to fix things if something goes wrong)
- In each external data folder, a folder called "noNaNs" is created as described in the IMPORTANT Note 1 above (item 3 of Setting Up the External Data).
- Each of these new waves is compared to every factor wave. (This can take a while if you have a lot of waves for comparison.)
- If the factor and external waves have different lengths, the function aborts and tells you that the comparison function was called with waves of different lengths. Unfortunately, it doesn't give helpful information about which wave had the wrong length (we'll try to look into that and improve that error message).
- If this happens, you should look in the "noNaNs" folder in the appropriate external data folder and check whether all of the waves have the correct length. Waves with incorrect numbers of points in this folder may be the result of incorrect wave lengths in the "external data" folder. Try to fix all of the problem waves and then run the function for calculating the scatter comparison again by choosing step 3 from the PMF pull-down menu. Recall that this is the only way to force recalculation of the comparison of the factors!
- If forcing recalculation didn't fix the problem, delete the "noNaNs" folder in the appropriate external data folder and then recalculate again.
- If this happens, you should look in the "noNaNs" folder in the appropriate external data folder and check whether all of the waves have the correct length. Waves with incorrect numbers of points in this folder may be the result of incorrect wave lengths in the "external data" folder. Try to fix all of the problem waves and then run the function for calculating the scatter comparison again by choosing step 3 from the PMF pull-down menu. Recall that this is the only way to force recalculation of the comparison of the factors!
- If the factor and external waves have different lengths, the function aborts and tells you that the comparison function was called with waves of different lengths. Unfortunately, it doesn't give helpful information about which wave had the wrong length (we'll try to look into that and improve that error message).
- The correlation values between the factor waves and the external data waves are stored in waves in the folder with PMF output called
- "RcorrMx4d_Profiles" and "RcorrMx4d_Tseries" (with Pearson R)
- "RcorrMx4d_Profiles_pear_mzGrt44" (with Pearson R, only for m/z > 44)
- It is also possible use the function scat_calc_RCorrMx4d_UC() to calculate
- "RcorrMx4d_Profiles_UC" and "RcorrMx4d_Tseries_UC" (with the Uncentered Correlation, as reported in [ Ulbrich et al., ACP, 2009]
- When the calculation is complete, the Scatter Panel is created.
Other Potential Problems and Solutions
- Pulldown menus don't have lists. Each "noNaNs" folder should also contain a text wave called "TseriesWvsNms" for time series or "FactorWvsNms" for profiles. This wave is used for the pulldown menus in the panel. If this wave is missing, the pulldown menus may not work. There should also be a string of wave names in the folder called "TseriesWvsNmsList" or "FactorWvsNmsList"; if so, you can create the text wave by gen_list2txtWv(listStr, wvNm).
Some Other Notes
- Order of the factors. Factors are numbered 1 to N and match the factors in the main panel, counting from the bottom of the factor plots.
- Colors. Factor 1 is black, Factor 2 is red, Factor 3 is green, Factor 4 is blue, etc. Factors in this panel have the same color as they did in the main panel. In the overlay plots, the factor is its usual color and the external species is orange.
- Size of Factor Space and Current Fpeak value sliders. The sliders in this panel and in the main panel control both panels simultaneously. Graph updates are slower with both panels. Be patient and don't click on anything until everything has updated.
More Features
- Assign Groups to External Data. This feature allows you to reorder the external data waves and assign them to groups. Groups then define the colors used in the R bar plots in the panel and in the Comprehensive External Data Correlation Plot (below).
- Comprehensive External Data Correlation Plot. This plot display the "R vs External Factor" plots for all factors at once.
Considerations for Choosing a Solution
- This topic is beyond the scope of discussion on this wiki.
PMF Evaluation Tool Software
Version | Date | Release Notes | PET ipf Files for Upgrading | PET Igor Template Experiment | PMF Executable Package | ||
---|---|---|---|---|---|---|---|
3.08E | 24 Oct 2024 | Release Notes v3.00 | Zipped PMF file containing these 7 ipfs:
PMF_SetupPanel_v3_08E.ipf |
or | Igor PMF template experiment v3_08E | and | PMF_executable_package.zip
|
Quick review: Igor code called PET (PMF Evaluation Tool) assists in a PMF analysis. Anyone with credentials can download PET; there is no need to request an individual account. When you click on a link to download, a prompt should appear asking for credentials. If you need credentials to download items on this wiki email Donna. The PET code v3.08E has been tested on these versions of Igor: 6.37, 7.08, 8.04, 9.0.6
A PMF executable file and a license file is required for PET to interface with. The executable file can be downloaded using the PMF_executable_package link above. Originally the PMF executable and license files could be purchased by sending an email to its creator Pentti Paatero (retired U. of Helsinki). The Swiss company Datalystica is now the sole official seller of the multi-linear engine (ME-2) solver package. This ME-2 package will contain a license that is used for the PMF executable. Once this ME-2 package is downloaded, copy and rename the ME2key.key file to pmf2key.key and place this in the same folder as the PMF executable files. Please go to https://datalystica.com/me-2-solver/ to purchase ME-2.
The Igor PET code is not needed to run PMF; PET provides a handy interface to generate and view PMF results.
Other Resources
- Ingrid's presentation of the PET at the 9th AMS Users Meeting in three parts (1. Overview and PMF Execution; 2. Viewing PMF Results; 3. Using the Scatter Plot Panel)
- The tutorial data used with that presentation
- Manjula's overview presentation of PMF at the 21st AMS Users Meeting (virtual, 2021)
- Donna's presentation of extending AMS data sets beyond Org or HROrg at 21st AMS Users Meeting (virtual, 2021)
Some PMF Method Papers
- Paatero, P., and Tapper, U.: Analysis of different modes of factor analysis as least squares fit problems, Chemom. Intell. Lab. Syst., 18, 183-194, 1993. Abstract
- Paatero, P., and Tapper, U.: Positive Matrix Factorization: a non-negative factor model with optimal utilization of error estimates of data values, Environmetrics, 5, 111-126, 1994. Abstract
- Paatero, P.: Least squares formulation of robust non-negative factor analysis, Chemom. Intell. Lab. Syst., 37, 23-35, 1997. Abstract
- Paatero, P., Hopke, P. K., Song, X. H., and Ramadan, Z.: Understanding and controlling rotations in factor analytic models, Chemom. Intell. Lab. Syst., 60, 253-264, 2002. Abstract
- Paatero, P., and Hopke, P. K.: Discarding or downweighting high-noise variables in factor analytic models, Anal. Chim. Acta, 490, 277-289, 10.1016/s0003-2670(02)01643-4, 2003. Abstract
- Paatero, P., Hopke, P. K., Begum, B. A., and Biswas, S. K.: A graphical diagnostic method for assessing the rotation in factor analytical models of atmospheric pollution, Atmos. Environ., 39, 193-201, 10.1016/j.atmosenv.2004.08.018, 2005. Abstract
- Paatero, P., and Hopke, P. K.: Rotational Tools for Factor Analytic Models, J. Chemom., 23, 91-100, 10.1002/cem.1197, 2009. Abstract