Difference between revisions of "Analysis Software"

From Jimenez Group Wiki
Jump to: navigation, search
(ICARTT Software)
m (Mie Code)
 
(208 intermediate revisions by 10 users not shown)
Line 4: Line 4:
 
*If eventually your group makes improvements, we'd like to have those shared back and ideally keep a master version.
 
*If eventually your group makes improvements, we'd like to have those shared back and ideally keep a master version.
  
All files on this page require proper credentials for download. Please contact us if you would like access to the files or to report any bugs with the software: [mailto:michael.lechner@colorado.edu,jlj.colorado@gmail.com,douglas.day@colorado.edu?Subject=Analysis%20Software%20Request password request]
+
All files on this page require proper credentials for download. Please contact us if you would like access to the files or to report any bugs with the software: [mailto:jlj.colorado@gmail.com,douglas.day@colorado.edu?Subject=Analysis%20Software%20Request password request]
  
==ICARTT==
+
 
===ICARTT Software===
+
==ICARTT ==
 +
===ICARTT Igor Software===
 
{|style="color:black; background-color:#ffffcc; width: 1000px" border="2" cellpadding="20"
 
{|style="color:black; background-color:#ffffcc; width: 1000px" border="2" cellpadding="20"
 
|-
 
|-
Line 13: Line 14:
 
!Version
 
!Version
 
!Notes
 
!Notes
!Link
+
!Links
 
|-
 
|-
|ICARTT Loader and File Creation Tool||2013||Donna Sueper and Ken Aiken <br /> [http://esrl.noaa.gov/csd/groups/csd7/measurements/ICARTT_data_mgt15.pdf ICARTT Documentation]||[http://cires.colorado.edu/jimenez-group/software/2013GeneralMacros.ipf 2013 General Macros (.ipf)] 01.17.2013 <br /> [http://cires.colorado.edu/jimenez-group/software/2013IcarttFormat.ipf 2013 ICARTT Format (.ipf)] 04.16.2013
+
|ICARTT Loader and File Creation Tool||2023  3.4.2||K Zarzana, D Sueper and K Aikin <br>
 +
 
 +
[https://earthdata.nasa.gov/esdis/eso/standards-and-references/icartt-file-format NASA ICARTT Standard]<br>
 +
[http://cires.colorado.edu/jimenez-group/software/ICARTT/ICARTT_GUI_Readme_v3xx.pdf HOWTO for this Igor Panel]<br/>
 +
 
 +
|[http://cires.colorado.edu/jimenez-group/software/ICARTT/Icartt_GUI.ipf ICARTT GUI v3 (.ipf)] Aug 10 2023<br />
 +
[http://cires.colorado.edu/jimenez-group/software/ICARTT/Standard_Names_Igor_Tool.zip ICARTT Std Name Lookup Tool v1 (.zip)] Aug 8 2023<br />
 +
[http://cires.colorado.edu/jimenez-group/software/ICARTT/Merge_all_ICARTT_files_in_same_folder.ipf Merge_all_ICARTT_files_in_same_folder (.ipf)] August 2019<br />
 +
<br />
 +
<!--- Password protected links, removed for SOAS to avoid password hassles
 +
 
 +
[https://earthdata.nasa.gov/esdis/eso/standards-and-references/icartt-file-format]<br>
 +
[http://cires.colorado.edu/jimenez-group/software/restricted/GenTools/2017GeneralMacros.ipf 2017 General Macros (.ipf)] 02.13.2013 <br /> [http://cires.colorado.edu/jimenez-group/software/2013IcarttFormat.ipf 2013 ICARTT Format (.ipf)] 04.16.2013
 
<br /><br />
 
<br /><br />
 +
 +
-->
 +
 +
<!--
 +
 +
[http://cires.colorado.edu/jimenez-group/software/2013GeneralMacros_v1-01.ipf 2013 General Macros v.1.01 (.ipf)] 01.17.2013<br />
 
[http://cires.colorado.edu/jimenez-group/software/2013IcarttFormat_v2-01.ipf 2013 ICARTT Format v.2.01 (.ipf)] 02.05.2013
 
[http://cires.colorado.edu/jimenez-group/software/2013IcarttFormat_v2-01.ipf 2013 ICARTT Format v.2.01 (.ipf)] 02.05.2013
 +
-->
 +
 
|}
 
|}
  
===ICARTT Software Notes===
+
===ICARTT Igor Software Notes===
*These functions create and load data files of the format indicated in the 2008 ICARTT Data Management Plan document, which was roughly based on the Haines-Hipskind NASA Ames Format. Functions and macros herein are not appropriate for all data generated by ICARTT participants; assumptions are outlined below.
+
*These functions create and load data files in the ICARTT format, a widely used, text based format for atmospheric data, particularly for aircraft data. The latest ICARTT specification is linked above. ICARTT is based on the Haines-Hipskind NASA Ames Format, and has undergone several revisions since its inception in 2004. Functions and macros herein are not appropriate for all data generated by ICARTT users; assumptions are outlined below.
*Functions in this file use functions in the 2008 GeneralMacros (latest version) igor procedure file. Users should have both files open in an igor experiment.
+
*This code can write and read data in a 1D timeseries format, which is the by and large most used ICARTT format ("Fileformat 1001", see below). The ICARTT specification also includes several additional file formats for posting multidimensional data (Fileformats 2110 and 2310), but these are currently not supported by this software.
*Whenever a user loads or creates a ICARTT format file, an igor data folder called ICARTTFileFormat gets created in the experiment and most of the work is done here. If something goes awry, the user can reset the data folder to root (most everyone's default)by inserting the text below (less the //) into the command line SetDataFolder root:
+
*There are 2 main versions of the ICARTT specification, v1.1 and v2, the latter being in use since 2017. The specification linked above refers to v2.0. Importantly, the code provided here is able to read and write most standard-compliant v1 and v2 ICARTTs.
*Some parameters are calculated; some have simple default values.  They are:
+
*Whenever a user loads or creates a ICARTT format file, an igor data folder called ICARTTFileFormat gets created in the experiment and most of the work is done here. If something goes awry, the user can reset the data folder to root (most everyone's default) by inserting the text below (less the //) into the command line SetDataFolder root:
**NumHeaderLines - gets calculated
+
*Some parameters are calculated; some have simple default values.  The later are:
 
**FileFormatType = 1001 // this value is assumed for reading and creating files.
 
**FileFormatType = 1001 // this value is assumed for reading and creating files.
 
**FileVolumeNumber = 1 // this value is assumed for reading and creating files.
 
**FileVolumeNumber = 1 // this value is assumed for reading and creating files.
 
**TotalVolumeNumber = 1 // this value is assumed for reading and creating files.
 
**TotalVolumeNumber = 1 // this value is assumed for reading and creating files.
**UTCDataStart - set from selected time wave
+
**ScaleFactor = 1 // this value is assumed for creating files, but reading files with non-unity scaling factors is supported
**NumVariables - gets set when user selects waves
+
**MissingDataIndicator = -9999 // NOTE - This is adjusted depending on the significant digits of the datastream
**ScaleFactor = 1 // this value is assumed for reading and creating files.
 
**MissingDataIndicator = -9999 // NOTE - The same missing value is used for all parameters when creating a file.
 
**NumSpecialCommentLines - gets calculated
 
**NumNormalCommentLines - gets calculated
 
**Normal Comments are the concatenation of several other parameters
 
 
*None of the original waves in the experiment are modified in any way. Waves selected to be saved are duplicated in the ICARTTFileFormat data folder. The UTC timewave (the independent parameter) and any dependent parameters formatted with a "t" (from the create panel) will have the starting date subtracted from it (in the created file) and will be formatted with one digit after the decimal.
 
*None of the original waves in the experiment are modified in any way. Waves selected to be saved are duplicated in the ICARTTFileFormat data folder. The UTC timewave (the independent parameter) and any dependent parameters formatted with a "t" (from the create panel) will have the starting date subtracted from it (in the created file) and will be formatted with one digit after the decimal.
 
*When loading an icartt file into an experiment, the missing value as reported in the header will be used to convert missing values in the incoming waves to nans.  The independent variable (time) will have the UTC date (as indicated in the header) added to it.
 
*When loading an icartt file into an experiment, the missing value as reported in the header will be used to convert missing values in the incoming waves to nans.  The independent variable (time) will have the UTC date (as indicated in the header) added to it.
Line 41: Line 57:
 
*The name of the wave in the experiment will be the name given to the dependent and independent parameters, with the exception of matricies, where column names much as mymatrix0, mymatrix1, etc are used.
 
*The name of the wave in the experiment will be the name given to the dependent and independent parameters, with the exception of matricies, where column names much as mymatrix0, mymatrix1, etc are used.
 
*If anything is named in the experiment in a 'liberal' fashion (i.e. wave names with spaces), all bets are off.
 
*If anything is named in the experiment in a 'liberal' fashion (i.e. wave names with spaces), all bets are off.
*Users should be aware that there are opearting system differences for end of line designations. Mac uses carriage return (CR), Unix uses line feed (LF), and Windows use carriage return + line feed (CRLF) For icarrtt, the preferred end of line designation is CRLF (which correspondes to \r\n in the code).
+
*Users should be aware that there are operating system differences for end of line designations. Mac uses carriage return (CR), Unix uses line feed (LF), and Windows use carriage return + line feed (CRLF) For icartt, the preferred end of line designation is CRLF (which correspondes to \r\n in the code).
 +
*NASA has been working since 2016 on a general naming specification for atmospheric variables, documented [https://www.earthdata.nasa.gov/esdis/esco/standards-and-practices/acvsnc here]. An Igor GUI to browse and find suitable standard names for your application is provided in the table above.
 +
 
 +
===Other Igor ICARTT Software===
 +
 
 +
*[http://cires1.colorado.edu/jimenez-group/software/ICARTT/ICARTT_AMS_FormattingTool_V2_0.ipf ICARTT_AMS_FormattingTool_V2_0.ipf]. This tool was written by James Allan (U Manachester) and uses one ipf.
 +
*Christoph Knote's Python package for reading and writing ICARTT formatted files – fully documented and tested. [https://pypi.org/project/icartt/ On pypi].
 +
 
 +
===ICARTT Matlab Software===
 +
The following software was written by Glenn Wolfe (NASA GDFL), who also maintains it, please contact him for support. It provides a generic reader and a specific/example writer procedure:
 +
* [https://github.com/AirChem/FlightTools Matlab ICARTT repository at Github]
 +
 
 +
===ICARTT R Software===
 +
The following ICARTT reader was written by Steve Wofsy (Harvard), who also maintains it, please contact him for support.
 +
* [http://cires1.colorado.edu/jimenez-group/software/ICARTT/read_1001_icartt_new.r ICARTT Reader in R]
 +
 
 +
===ICARTT Python Software===
 +
Provided by Christoph Knote (being updated as of 2022)
 +
 
 +
*Project page at Pypi [https://pypi.org/project/icartt/ Python Package]
 +
*Project page at U Augsburg [https://mbees.med.uni-augsburg.de/gitlab/mbees/icartt_pypackage ICARTT Python Package and Discussion]
 +
 
 +
Provided by: Barron Henderson barronh@gmail.com (21-Aug-2013)
 +
*ICARTT reader / netcdf converter for python
 +
*Part of PseudoNetCDF package found [https://github.com/barronh/pseudonetcdf here]
 +
 
 +
&#35; In a python script<br />
 +
from PseudoNetCDF.icarttfiles import ffi1001<br />
 +
f = ffi1001('pathtofile')<br />
 +
var = f.variables['HO2_pptv']<br />
 +
print var[:].mean(), var.units<br />
 +
<br />
 +
&#35; From command line: dump CDF (like ncdump)<br />
 +
python -m PseudoNetCDF.pncdump -f ffi1001 pathtofile<br />
 +
<br />
 +
&#35; From command line: make a NetCDF file (like ncdump file1 | ncgen file2)<br />
 +
python -m PseudoNetCDF.pncgen -f ffi1001 pathtofile <br />pathtonetcdfoutput.nc<br />
 +
 
 +
==How to read NetCDF v3 into Igor==
 +
 
 +
*Provided by Pedro Campuzano-Jost, with credit to Pengfei Yu.
 +
*This applies to working with instrument/model output that is formatted in netcdf v3 (1997 release).
 +
 
 +
*Reading netcdf3 files in Igor is extremely painful. There is a 2004, 32-bit compatible XOP from a Japanese group buried in the depths of IgorExchange, with no hope of an update, as well as some code on top of that XOP from both the Manchester guys and Donna. But regardless, it requires you to open a 32-bit version of Igor just to load the file, and the handling of 2D waves and wave attributes is...less than ideal.
 +
 
 +
*On the other hand, netcdf v4 is very easy to deal with since the data structure conforms to the hdf5 format (used by Aerodyne and Tofwerk instruments and many other applications). But despite the fact that v4 is 10 years old (v 4.0.0 of the netcdf library was released in 2008), many scripts used by atmospheric chemist still use v3 (including about 80% of global modelers). The fix is actually surprisingly easy:
 +
 
 +
**Download the netcdf 4.6 for windows library/utility pack: http://www.unidata.ucar.edu/downloads/netcdf/ftp/netCDF4.6.1-NC4-64.exe (NCAR, for unknown reasons does not advertise the windows version on their main software page, so if you want to stay up to date check this webpage: https://www.unidata.ucar.edu/software/netcdf/docs/winbin.html)
 +
 
 +
**Install the package and MAKE SURE you add the library to your System/DOS path (installing for all users is probably the best/easiest option)
 +
 
 +
**Open a command prompt and cd yourself into the directory where your netcdf v3 file lives (if this sounds scary, just put your file in your personal folder, above "Documents", and you won't need to cd at all)
 +
 
 +
**Run "nccopy -4 MyNetCDFv3File.nc MyNetCDFv3File.h5"
 +
 
 +
**The new file you created, MyNetCDFv3File.h5 is now a netcdf4/h5 file and can be perused at will with the hdf5browser in Igor (warning, it is likely quite a bit bigger than the old one).
 +
 
 +
==Mie Code==
 +
 
 +
* The Igor IPF below contains the J-Group Scattering/Extinction calculator, based on the code provided in the book by C. F. Bohren and D. R. Huffman, [http://onlinelibrary.wiley.com/book/10.1002/9783527618156 Absorption and Scattering of Light by Small Particles].
 +
* Our group uses this function frequently to independently check the quantification of the AMS, by e.g. comparing the mass scattering efficiency (MSE) calculated from AMS mass divided by dry PM1 scattering, to the MSE calculated from measured size distributions (e.g. from an SMPS) and this Mie code. For an example see Figure 2d in [http://www.atmos-chem-phys.net/8/4027/2008/acp-8-4027-2008.pdf DeCarlo et al., 2008]
 +
* The original version of the Mie subroutine was translated from Fortran into Igor by Charles Brock, NOAA ESRL, and it is provided "as is" (few comments). For interpretation of what that subroutine is doing see the B&H book and also e.g. [http://rime.aos.wisc.edu/MW/models/bhmie.shtml this link] that provided Fortran code with more detailed comments.
 +
* The wrapper code was written by Ed Dunlea, Mike Cubison and Pedro Campuzano Jost.
 +
* Contact Pedro.CampuzanoJost-1@colorado.edu and jose.jimenez@colorado.edu for feedback or bug reports.
 +
* You can download the code from [http://cires1.colorado.edu/jimenez-group/software/SizeDist_Scat_Ext_Mie_JGroup_v2_0.ipf this link].
  
 
==Picarro G2401==
 
==Picarro G2401==
Line 52: Line 132:
 
!Link
 
!Link
 
|-
 
|-
|Picarro G2401 Loader and Data Plotting||v1.0||First release by Michael Lechner ||[http://cires.colorado.edu/jimenez-group/software/Picarro_Panel_v1_0.ipf Picarro Panel (.ipf)]
+
|Picarro G2401 Loader and Data Plotting||v1.0||First release by Michael Lechner ||[http://cires.colorado.edu/jimenez-group/software/restricted/Packages/Picarro_Panel.ipf Picarro Panel (.ipf)]
 
|}
 
|}
  
Line 60: Line 140:
 
**Plots:
 
**Plots:
 
***Picarro measurement time series
 
***Picarro measurement time series
****CH4 / CH4 dry
+
****CO is the fully corrected measurement which corrects for CO2 and H2O spectroscopic interferences, as well as H2O dilution.
****CO
+
****CO2_dry is corrected for H2O spectroscopic interference & dilution effect of H2O
****CO2 / CO2 dry
+
****CO2 is not corrected for anything (probably never use)
 +
****CH4_dry is only corrected for H2O spectroscopic interference & dilution effect of H2O
 +
****CH4 is not corrected for anything (probably never use)
 
****H2O
 
****H2O
 
***Picarro diagnostic time series
 
***Picarro diagnostic time series
Line 86: Line 168:
 
!Link
 
!Link
 
|-
 
|-
|Real Time AMS Igor Display||v10||First release Donna Sueper and Mike Cubison <br /> Modified by Michael Lechner ||[http://cires.colorado.edu/jimenez-group/software/DAQ_QuickView.ipf DAQ_QuickView.ipf]
+
|Real Time AMS Igor Display||v1.0||First release Donna Sueper and Mike Cubison <br /> Modified by Michael Lechner ||[http://cires.colorado.edu/jimenez-group/software/restricted/Packages/DAQ_QuickView.ipf DAQ_QuickView.ipf]
 
|}
 
|}
  
Line 110: Line 192:
 
==SMPS==
 
==SMPS==
 
===SMPS Software===
 
===SMPS Software===
 +
 +
<font color="red">The SMPS Software is generally not shared at this point, except with close collaborators of our group, as we don't have time to support it.</font>
 +
 
{|style="color:black; background-color:#ffffcc; width: 1000px;" border="2" cellpadding="20"
 
{|style="color:black; background-color:#ffffcc; width: 1000px;" border="2" cellpadding="20"
 
|-
 
|-
Line 117: Line 202:
 
!Link
 
!Link
 
|-
 
|-
|SMPS data loader||v4||First release Doug Day (built on MJC's original loader) ||[http://cires.colorado.edu/jimenez-group/software/JG_SMPS_v4.pxt JG_SMPS_v4.pxt]<br /> [http://cires.colorado.edu/jimenez-group/software/JG_SMPS_v4_example.pxp Example experiment]<br /> [http://cires.colorado.edu/jimenez-group/software/101213_SMPS_1_3test.zip Sample AIM exported text files]
+
|SMPS data loader/plotter||v4||First release Doug Day (built on MJC's original loader) ||[http://cires.colorado.edu/jimenez-group/software/restricted/Old_Versions/JG_SMPS_v4.pxt JG_SMPS_v4.pxt]<br /> [http://cires.colorado.edu/jimenez-group/software/restricted/Old_Versions/JG_SMPS_v4_example.pxp Example experiment]<br /> [http://cires.colorado.edu/jimenez-group/software/restricted/Old_versions/101213_SMPS_1_3test.zip Sample AIM exported text files]
 +
|-
 +
|SMPS data loader/plotter||v4.46 <br /> (latest)|| 6 Aug. 2014 <br /> -Revised so that only supporting ipf needed is DS_GenTools.ipf (included in pxt posted here). See version notes in header of main .ipf || [http://cires.colorado.edu/jimenez-group/software/restricted/Packages/JG_SMPS.ipf JG_SMPS.ipf] <br /> [http://cires.colorado.edu/jimenez-group/software/restricted/Packages/JG_SMPS_v4_46.pxt JG_SMPS_v4_46.pxt]
 
|}
 
|}
  
 
===SMPS Software Notes===
 
===SMPS Software Notes===
<b>JG_SMPS_V4 (dd 1/12/11)</b>
+
<b>JG_SMPS_V4.46 (dd 8/06/14)</b>
*Loads and Concatenates SMPS data collected with TSI AIM (built on Mike C's BEARPEX/CalNex JG_SMPS_v3.ipf and uses ToolsMJC_v1a.ipf and Tools_20100221.ipf)
+
*Loads and Concatenates SMPS data collected with TSI AIM (built on Mike C's BEARPEX/CalNex JG_SMPS_v3.ipf and used ToolsMJC_v1a.ipf and Tools_20100221.ipf)
 +
*As of V4.46 uses DS_GenTools.ipf for general functions (previous to then, used JG_GenTools or DougTools.ipf))
 
*Input should be .txt files created as ouput in TSI AIM software (row-formatted) and dNdLogDp and in named in format: 2010_12_13_1522_SMPS.txt  (yyyy_mm_dd_hhmm_SMPS.txt)
 
*Input should be .txt files created as ouput in TSI AIM software (row-formatted) and dNdLogDp and in named in format: 2010_12_13_1522_SMPS.txt  (yyyy_mm_dd_hhmm_SMPS.txt)
*"Load-Concatenate" button promps user to select folderpath for text files to load  
+
*If starting with .ipf (rather than .pxt, .pxp);
and also promps user to input assign "project" name to files loaded (will put  
+
**0) Compile
all data in folder names as such
+
**1) Run function "PopulateSMPSpanel()" (which populates the panel with default input values in "root:Temporary")
*Image Time Series Plots plots either number or mass distributions (density approximated as 1.0 g/cm3)
+
**2)"SMPS Plotting Panel" from "SMPS" menu dropdown
Optional inputs for:
+
**3) If using a mask wave, a) customize "AssignFlags()" function and run; b) load mask and time waves in "Masterlogger folder"
Interpolation across data gaps
+
*"Load-Concatenate" button prompts user to select folderpath for text files to load and also promps user to input assign "project" name to files loaded (will put all data in folder names as such)
Dp smoothing
+
**Choose the version of TSI AIM used to export text files (diagnostic columns vary)
Max percentile of data assigned to hottest color in image (help use full color range)
+
**User can chose number of Dp bins to regrid dNdLogDp to at the loading step only. This is useful for when file of different Dp ranges are loaded together. (when plotting later can chose to use these or the native bins). Warning, uses a linear interpolation, not binning, so if much fewer bins than in original data is chosen,not all data points are used in regridding so statistics may be degraded. Thus best to chose similar Dp number to original data and use raw data for final workups, instrument comparison, etc.
 +
*Image Time Series Plots plots either number or volume distributions
 +
**Optional inputs for:
 +
***Dp smoothing
 +
***Max percentile of data assigned to hottest color in image (help use full color range)
 +
***Interpolation across data gaps (of user-specified duration)
 +
***Choosing whether to use regridded or raw dXdLogDp (native scaling)
 
*Integrated # and Mass Concentration Plot (Number or Mass)
 
*Integrated # and Mass Concentration Plot (Number or Mass)
Optional inputs: Max/Min Dp over which to integrate total concentration  
+
**Optional inputs: Max/Min Dp over which to integrate total concentration
 
   
 
   
 
<b>Additional Notes:</b>
 
<b>Additional Notes:</b>
 
*Best to reselect "Data Folder to Plot" to be sure that datafolder is set to proper project folder (should be reset after all plotting, so only really needed if user changes datafolders). However, must select before first plot.
 
*Best to reselect "Data Folder to Plot" to be sure that datafolder is set to proper project folder (should be reset after all plotting, so only really needed if user changes datafolders). However, must select before first plot.
 
*Panel inputs stored in root:Temporary (DO NOT DELETE this folder) - can delete all other folders
 
*Panel inputs stored in root:Temporary (DO NOT DELETE this folder) - can delete all other folders
 +
*Scaled dXdlogDp waves: Currently, these are not used anywhere in the code. To avoid make large "scaled waves" (interpolated to new/fixed timestamp) see the line near the top of the main ipf: "Variable offset = date2secs(2010,12,13), delta=60 //DD". As written (in the posted version) this would make a scaled size distribution from Dec, 13, 2010 with a point every 60 seconds up until YOUR last scan time. To avoid make this long, unused, mostly empty wave, change the date to the day before you collected your data and maybe change the delta to something bigger, like 300+. Then those waves will be small. Will code with more elegant solution in future version.
 +
*There are several auxillary functions at the bottom that may be useful for QC, exporting data. Use "CutBadDat()" function to "manually" enter bad periods that thereafter will be cut when using panel buttons.
 +
*Version update notes at top of .ipf
  
 
<b>Next Version Steps:</b>
 
<b>Next Version Steps:</b>
*some QC functions like in CalNex example
+
*Several. See top of ipf. Feel free to send suggestions to Doug.
*option for mask waves
 
  
 
===Software Output===
 
===Software Output===
 +
<b>Panel v4.44</b><br />
 +
[[image:JG_SMPS_Panel_v4_44.png|400x650px]]<br />
 
<b>dNdlogDp Size Distribution</b><br />
 
<b>dNdlogDp Size Distribution</b><br />
 
[[image:dn_sizedist.jpg|700x1000px]]<br />
 
[[image:dn_sizedist.jpg|700x1000px]]<br />
Line 152: Line 250:
 
<b>dM Total Concentration</b><br />
 
<b>dM Total Concentration</b><br />
 
[[image:dm_total_conc.jpg|700x1000px]]
 
[[image:dm_total_conc.jpg|700x1000px]]
 +
 +
== KinSim ==
 +
===KinSim Software===
 +
 +
{|style="color:black; background-color:#ffffcc; width: 1000px;" border="2" cellpadding="20"
 +
|-
 +
!Software
 +
!Version
 +
!Notes
 +
!Link
 +
|-
 +
|KinSim||v2|| First release by Harald Stark <br /> 10 Nov. 2014 Modified by Zhe Peng || [http://cires.colorado.edu/jimenez-group/software/KinSim/KinSim_v2.1.ipf KinSim_v2.1.ipf]
 +
|-
 +
|KinSim||v3|| Release by Zhe Peng and Jose-Luis Jimenez <br /> 31 May 2018 Modified by Zhe Peng || [http://cires.colorado.edu/jimenez-group/software/KinSim/KinSim_v3.75.ipf KinSim_v3.75.ipf]
 +
|}
 +
 +
===KinSim Software Notes===
 +
<b>KinSim_v2.1.ipf (ZP 11/10/14)</b>
 +
*Version 1 (by Harald Stark): mechanism generation and integration of rate equations.
 +
*Version 2: functions in v1 and
 +
**P- and T-dependent rate constants
 +
**Advanced input of initial concentrations - compatible to direct input of numbers (for concentrations in molec/cm3), and mixing ratios in "%", "ppth", "ppm", "ppb", "ppt", and "ppq"
 +
**Optimized parameters for the integrator - to handle stiff problems
 +
*Version 2.1: update of help notes and fix of minor bugs
 +
*General usage: compile the procedure then select "Display_help" from the "KinSim" menu. After the dialog box is prompted, select "Basic version" if only the functions in v1 are needed, else select "Version 2". Follow the help notes.
 +
<b>KinSim_v3.75.ipf (ZP 5/31/18)</b>
 +
*Version 3: switch from menu control to user-friedly panel control, substantial function reorganization, and
 +
**Automatic input check (for basic parameters and mechanism)
 +
**Species concentration constrain
 +
**Special treatment for photolysis, emissions, and dilutions (user-defined dilution air composition)
 +
**Times series plots of species concentrations and reaction rates
 +
*Version 3.75: fix of bugs, conversion of some macros to functions, and
 +
**Mechanism export and import
 +
**Preclusion of species from dilution
 +
**Mechanism input in both JPL and IUPAC formats
 +
**Possibility of enabling/disabling reactions in the mechanism
 +
**Fractional product stoichiometric coefficients
 +
**Option of simple rate-coefficient input table
 +
**Translation table to build time-dependence table
 +
**Real-time T & P change
 +
**Gas-particle partitioning calculations
 +
*General usage: see [https://docs.google.com/document/d/1PyADb1YBjtJ4-BQjnOuVn5Am1UeMbrzYS4gYROb1NUw here]
 +
 +
===Input Mechanisms===
 +
* Basic OFR mechanism (v1.00 ZP 9/10/14)
 +
** Including photolytic, Ox, HOx, NOx, and SO2 reactions in OFRs
 +
** A few reactions have special rate constant expressions. Refer to the "Notes" column for more details.
 +
** Link: [http://cires.colorado.edu/jimenez-group/software/KinSim/Mechanisms/Basic_OFR_mechanism.pxp Basic_OFR_mechanism.pxp]
 +
* Mechanism of CH4 full oxidation in OFRs (v1.00 ZP 9/10/14)
 +
** Should be used along with the basic OFR mechanism
 +
** Link: [http://cires.colorado.edu/jimenez-group/software/KinSim/Mechanisms/CH4_OFR-oxidation_mechanism.pxp CH4_OFR-oxidation_mechanism.pxp]
 +
* Mechanism for 2014 NO3-Terpene Chamber Studies (v1.00 DD, 6 Jan, 2015)
 +
** Link: [http://cires1.colorado.edu/jimenez-group/software/KinSim/Mechanisms/NO3_Terpene_mechanism_v1_00_Jan6_15_Doug_Day.pxp NO3_Terpene_mechanism.pxp]
 +
** Should work with latest KinSim versions, however see [http://cires1.colorado.edu/jimenez-group/software/KinSim/Mechanisms/NO3_Terpene_mechanism_v1_00_Jan6_15_Doug_Day_AllFiles.zip here] for ipfs pre-version control, including additional post-processing/plotting functions and example plots images.
 +
* Mechanism for IEPOX-SOA (v1.00 DJ, 17 Nov, 2017)
 +
** IEPOX-SOA mechanism based on [https://www.atmos-chem-phys.net/16/1603/2016/ Marais et al. (2016) ]
 +
** Link: [http://cires.colorado.edu/jimenez-group/software/KinSim/Mechanisms/IEPOX_SOA_mechanism.pxp IEPOX_SOA_mechanism.pxp]
 +
 +
===Other Utilities===
 +
* List/post other utilities here

Latest revision as of 12:35, 11 August 2023

If you would like access to the software on this page we ask that you adhere to a few basic conditions:

  • You don't share it further than your group, but direct any additional requests to us
  • We will answer simple questions and try to add requested features and especially fix bugs, but we can't "support" the software as if it were a commercially available package, we also will not answer basic Igor questions.
  • If eventually your group makes improvements, we'd like to have those shared back and ideally keep a master version.

All files on this page require proper credentials for download. Please contact us if you would like access to the files or to report any bugs with the software: password request


ICARTT

ICARTT Igor Software

Software Version Notes Links
ICARTT Loader and File Creation Tool 2023 3.4.2 K Zarzana, D Sueper and K Aikin

NASA ICARTT Standard
HOWTO for this Igor Panel

ICARTT GUI v3 (.ipf) Aug 10 2023

ICARTT Std Name Lookup Tool v1 (.zip) Aug 8 2023
Merge_all_ICARTT_files_in_same_folder (.ipf) August 2019


ICARTT Igor Software Notes

  • These functions create and load data files in the ICARTT format, a widely used, text based format for atmospheric data, particularly for aircraft data. The latest ICARTT specification is linked above. ICARTT is based on the Haines-Hipskind NASA Ames Format, and has undergone several revisions since its inception in 2004. Functions and macros herein are not appropriate for all data generated by ICARTT users; assumptions are outlined below.
  • This code can write and read data in a 1D timeseries format, which is the by and large most used ICARTT format ("Fileformat 1001", see below). The ICARTT specification also includes several additional file formats for posting multidimensional data (Fileformats 2110 and 2310), but these are currently not supported by this software.
  • There are 2 main versions of the ICARTT specification, v1.1 and v2, the latter being in use since 2017. The specification linked above refers to v2.0. Importantly, the code provided here is able to read and write most standard-compliant v1 and v2 ICARTTs.
  • Whenever a user loads or creates a ICARTT format file, an igor data folder called ICARTTFileFormat gets created in the experiment and most of the work is done here. If something goes awry, the user can reset the data folder to root (most everyone's default) by inserting the text below (less the //) into the command line SetDataFolder root:
  • Some parameters are calculated; some have simple default values. The later are:
    • FileFormatType = 1001 // this value is assumed for reading and creating files.
    • FileVolumeNumber = 1 // this value is assumed for reading and creating files.
    • TotalVolumeNumber = 1 // this value is assumed for reading and creating files.
    • ScaleFactor = 1 // this value is assumed for creating files, but reading files with non-unity scaling factors is supported
    • MissingDataIndicator = -9999 // NOTE - This is adjusted depending on the significant digits of the datastream
  • None of the original waves in the experiment are modified in any way. Waves selected to be saved are duplicated in the ICARTTFileFormat data folder. The UTC timewave (the independent parameter) and any dependent parameters formatted with a "t" (from the create panel) will have the starting date subtracted from it (in the created file) and will be formatted with one digit after the decimal.
  • When loading an icartt file into an experiment, the missing value as reported in the header will be used to convert missing values in the incoming waves to nans. The independent variable (time) will have the UTC date (as indicated in the header) added to it.
  • When loading data from the LoadDataOnly menu item, almost no format checking is done and missing data isn't changed to nans.
  • The name of the wave in the experiment will be the name given to the dependent and independent parameters, with the exception of matricies, where column names much as mymatrix0, mymatrix1, etc are used.
  • If anything is named in the experiment in a 'liberal' fashion (i.e. wave names with spaces), all bets are off.
  • Users should be aware that there are operating system differences for end of line designations. Mac uses carriage return (CR), Unix uses line feed (LF), and Windows use carriage return + line feed (CRLF) For icartt, the preferred end of line designation is CRLF (which correspondes to \r\n in the code).
  • NASA has been working since 2016 on a general naming specification for atmospheric variables, documented here. An Igor GUI to browse and find suitable standard names for your application is provided in the table above.

Other Igor ICARTT Software

  • ICARTT_AMS_FormattingTool_V2_0.ipf. This tool was written by James Allan (U Manachester) and uses one ipf.
  • Christoph Knote's Python package for reading and writing ICARTT formatted files – fully documented and tested. On pypi.

ICARTT Matlab Software

The following software was written by Glenn Wolfe (NASA GDFL), who also maintains it, please contact him for support. It provides a generic reader and a specific/example writer procedure:

ICARTT R Software

The following ICARTT reader was written by Steve Wofsy (Harvard), who also maintains it, please contact him for support.

ICARTT Python Software

Provided by Christoph Knote (being updated as of 2022)

Provided by: Barron Henderson barronh@gmail.com (21-Aug-2013)

  • ICARTT reader / netcdf converter for python
  • Part of PseudoNetCDF package found here

# In a python script
from PseudoNetCDF.icarttfiles import ffi1001
f = ffi1001('pathtofile')
var = f.variables['HO2_pptv']
print var[:].mean(), var.units

# From command line: dump CDF (like ncdump)
python -m PseudoNetCDF.pncdump -f ffi1001 pathtofile

# From command line: make a NetCDF file (like ncdump file1 | ncgen file2)
python -m PseudoNetCDF.pncgen -f ffi1001 pathtofile
pathtonetcdfoutput.nc

How to read NetCDF v3 into Igor

  • Provided by Pedro Campuzano-Jost, with credit to Pengfei Yu.
  • This applies to working with instrument/model output that is formatted in netcdf v3 (1997 release).
  • Reading netcdf3 files in Igor is extremely painful. There is a 2004, 32-bit compatible XOP from a Japanese group buried in the depths of IgorExchange, with no hope of an update, as well as some code on top of that XOP from both the Manchester guys and Donna. But regardless, it requires you to open a 32-bit version of Igor just to load the file, and the handling of 2D waves and wave attributes is...less than ideal.
  • On the other hand, netcdf v4 is very easy to deal with since the data structure conforms to the hdf5 format (used by Aerodyne and Tofwerk instruments and many other applications). But despite the fact that v4 is 10 years old (v 4.0.0 of the netcdf library was released in 2008), many scripts used by atmospheric chemist still use v3 (including about 80% of global modelers). The fix is actually surprisingly easy:
    • Install the package and MAKE SURE you add the library to your System/DOS path (installing for all users is probably the best/easiest option)
    • Open a command prompt and cd yourself into the directory where your netcdf v3 file lives (if this sounds scary, just put your file in your personal folder, above "Documents", and you won't need to cd at all)
    • Run "nccopy -4 MyNetCDFv3File.nc MyNetCDFv3File.h5"
    • The new file you created, MyNetCDFv3File.h5 is now a netcdf4/h5 file and can be perused at will with the hdf5browser in Igor (warning, it is likely quite a bit bigger than the old one).

Mie Code

  • The Igor IPF below contains the J-Group Scattering/Extinction calculator, based on the code provided in the book by C. F. Bohren and D. R. Huffman, Absorption and Scattering of Light by Small Particles.
  • Our group uses this function frequently to independently check the quantification of the AMS, by e.g. comparing the mass scattering efficiency (MSE) calculated from AMS mass divided by dry PM1 scattering, to the MSE calculated from measured size distributions (e.g. from an SMPS) and this Mie code. For an example see Figure 2d in DeCarlo et al., 2008
  • The original version of the Mie subroutine was translated from Fortran into Igor by Charles Brock, NOAA ESRL, and it is provided "as is" (few comments). For interpretation of what that subroutine is doing see the B&H book and also e.g. this link that provided Fortran code with more detailed comments.
  • The wrapper code was written by Ed Dunlea, Mike Cubison and Pedro Campuzano Jost.
  • Contact Pedro.CampuzanoJost-1@colorado.edu and jose.jimenez@colorado.edu for feedback or bug reports.
  • You can download the code from this link.

Picarro G2401

Picarro Software

Software Version Notes Link
Picarro G2401 Loader and Data Plotting v1.0 First release by Michael Lechner Picarro Panel (.ipf)

Picarro Software Notes

  • Picarro Panel v1.0 (MJL 02/05/2013)
    • Recursively loads all .DAT Picarro acquisition files or a specified date range of files in a given directory (best to point the loading directory to DataLog_User and leave the sub directories in their original configuration)
    • Plots:
      • Picarro measurement time series
        • CO is the fully corrected measurement which corrects for CO2 and H2O spectroscopic interferences, as well as H2O dilution.
        • CO2_dry is corrected for H2O spectroscopic interference & dilution effect of H2O
        • CO2 is not corrected for anything (probably never use)
        • CH4_dry is only corrected for H2O spectroscopic interference & dilution effect of H2O
        • CH4 is not corrected for anything (probably never use)
        • H2O
      • Picarro diagnostic time series
        • Das Temperature
        • Etalon Temperature
        • Warm Box Temperature
        • Cavity Temperature
        • Cavity Pressure

Software Output

Measurement Timeseries
Picarro Timeseries.jpg
Diagnostic Timeseries
Picarro Diagnostics.jpg

Real Time AMS Igor Display

Real Time Software

Software Version Notes Link
Real Time AMS Igor Display v1.0 First release Donna Sueper and Mike Cubison
Modified by Michael Lechner
DAQ_QuickView.ipf

Real Time Notes

  • THIS FUNCTION WILL NOT COMPILE WITHOUT APPROPRIATE XOP!
    • An XOP (external operation) is a modular chunk of code that adds features to Igor One needs the file called VDT2.xop or a shortcut to this file to be placed in the directory Wavemetrics:Igor Pro Folder:Igor Extensions: The VDT2.xop is normally found in the folder called Wavemetrics:Igor Pro Folder:More Extensions:Data Acquisition
  • Real Time (RT) and Serial Feed (SF) ipf
    • This code is for use with the ToF AMS data acquisition program. In the daq one has the option of outputting a limited set of parameters to one text file (TimeTrace.txt)many individual text files (RT_xxxxx.txt) and/or serial output (No file name, just a an old fashioned serial RS232 output. Most will only work with only the RT_xxx.txt files
    • The functions in this ipf are intended to run in an Igor experiment concurrently with the DAQ. The RT_xxxx.txt files generated by the DAQ are intended to be read into an Igor experiment on the DAQ computer. Absolutely nothing in this ipf should change anything that happens in the DAQ
  • Usage Instructions:
    • Aircraft / Ground Plot will bring up a plot of predetermined parameters focused on Aircraft / Ground data
    • Delete Real Time Files checkbox when unchecked files are read into igor and then the file is moved to the processed folder. When checked file is not moved to the process folder, but instead is deleted.
    • The "Clear.." button will clear existing waves - will set them to zero points. "Pause.." will resume appending values to existing waves.
    • The "Process MS itx files" checkbox causes the itx mass spectra files to be loaded and appended to a 2D m/z vs time wave (assuming they have been selected to be saved in the DAQ menu). No plotting or further averaging is done in the DAQ_QuickView ipf posted here.
  • The TimeTrace.txt nd the serial feed functions are intended to more specialized purposes.
    • The TimeTrace.txt file is intended to be read into an Igor experiment as one file, and no special code is needed (beyond that of creating an igor-friendly time wave). In this case, just create the Real Time panel and press the begin button.
    • The serial feed functions are intended to work on a computer that has a serial stream connection to the daq. In this case, just create the Serial Feed panel and press the begin button. In either case, a standard graph of values will be plotted. Users may make new plots or change the existing one to suit their needs.

Software Output

Aircraft Data Plot
Real Time Plot.jpg

SMPS

SMPS Software

The SMPS Software is generally not shared at this point, except with close collaborators of our group, as we don't have time to support it.

Software Version Notes Link
SMPS data loader/plotter v4 First release Doug Day (built on MJC's original loader) JG_SMPS_v4.pxt
Example experiment
Sample AIM exported text files
SMPS data loader/plotter v4.46
(latest)
6 Aug. 2014
-Revised so that only supporting ipf needed is DS_GenTools.ipf (included in pxt posted here). See version notes in header of main .ipf
JG_SMPS.ipf
JG_SMPS_v4_46.pxt

SMPS Software Notes

JG_SMPS_V4.46 (dd 8/06/14)

  • Loads and Concatenates SMPS data collected with TSI AIM (built on Mike C's BEARPEX/CalNex JG_SMPS_v3.ipf and used ToolsMJC_v1a.ipf and Tools_20100221.ipf)
  • As of V4.46 uses DS_GenTools.ipf for general functions (previous to then, used JG_GenTools or DougTools.ipf))
  • Input should be .txt files created as ouput in TSI AIM software (row-formatted) and dNdLogDp and in named in format: 2010_12_13_1522_SMPS.txt (yyyy_mm_dd_hhmm_SMPS.txt)
  • If starting with .ipf (rather than .pxt, .pxp);
    • 0) Compile
    • 1) Run function "PopulateSMPSpanel()" (which populates the panel with default input values in "root:Temporary")
    • 2)"SMPS Plotting Panel" from "SMPS" menu dropdown
    • 3) If using a mask wave, a) customize "AssignFlags()" function and run; b) load mask and time waves in "Masterlogger folder"
  • "Load-Concatenate" button prompts user to select folderpath for text files to load and also promps user to input assign "project" name to files loaded (will put all data in folder names as such)
    • Choose the version of TSI AIM used to export text files (diagnostic columns vary)
    • User can chose number of Dp bins to regrid dNdLogDp to at the loading step only. This is useful for when file of different Dp ranges are loaded together. (when plotting later can chose to use these or the native bins). Warning, uses a linear interpolation, not binning, so if much fewer bins than in original data is chosen,not all data points are used in regridding so statistics may be degraded. Thus best to chose similar Dp number to original data and use raw data for final workups, instrument comparison, etc.
  • Image Time Series Plots plots either number or volume distributions
    • Optional inputs for:
      • Dp smoothing
      • Max percentile of data assigned to hottest color in image (help use full color range)
      • Interpolation across data gaps (of user-specified duration)
      • Choosing whether to use regridded or raw dXdLogDp (native scaling)
  • Integrated # and Mass Concentration Plot (Number or Mass)
    • Optional inputs: Max/Min Dp over which to integrate total concentration

Additional Notes:

  • Best to reselect "Data Folder to Plot" to be sure that datafolder is set to proper project folder (should be reset after all plotting, so only really needed if user changes datafolders). However, must select before first plot.
  • Panel inputs stored in root:Temporary (DO NOT DELETE this folder) - can delete all other folders
  • Scaled dXdlogDp waves: Currently, these are not used anywhere in the code. To avoid make large "scaled waves" (interpolated to new/fixed timestamp) see the line near the top of the main ipf: "Variable offset = date2secs(2010,12,13), delta=60 //DD". As written (in the posted version) this would make a scaled size distribution from Dec, 13, 2010 with a point every 60 seconds up until YOUR last scan time. To avoid make this long, unused, mostly empty wave, change the date to the day before you collected your data and maybe change the delta to something bigger, like 300+. Then those waves will be small. Will code with more elegant solution in future version.
  • There are several auxillary functions at the bottom that may be useful for QC, exporting data. Use "CutBadDat()" function to "manually" enter bad periods that thereafter will be cut when using panel buttons.
  • Version update notes at top of .ipf

Next Version Steps:

  • Several. See top of ipf. Feel free to send suggestions to Doug.

Software Output

Panel v4.44
JG SMPS Panel v4 44.png
dNdlogDp Size Distribution
Dn sizedist.jpg
dMdlogDp Size Distribution
Dm sizedist.jpg
dN Total Concentration
Dn total conc.jpg
dM Total Concentration
Dm total conc.jpg

KinSim

KinSim Software

Software Version Notes Link
KinSim v2 First release by Harald Stark
10 Nov. 2014 Modified by Zhe Peng
KinSim_v2.1.ipf
KinSim v3 Release by Zhe Peng and Jose-Luis Jimenez
31 May 2018 Modified by Zhe Peng
KinSim_v3.75.ipf

KinSim Software Notes

KinSim_v2.1.ipf (ZP 11/10/14)

  • Version 1 (by Harald Stark): mechanism generation and integration of rate equations.
  • Version 2: functions in v1 and
    • P- and T-dependent rate constants
    • Advanced input of initial concentrations - compatible to direct input of numbers (for concentrations in molec/cm3), and mixing ratios in "%", "ppth", "ppm", "ppb", "ppt", and "ppq"
    • Optimized parameters for the integrator - to handle stiff problems
  • Version 2.1: update of help notes and fix of minor bugs
  • General usage: compile the procedure then select "Display_help" from the "KinSim" menu. After the dialog box is prompted, select "Basic version" if only the functions in v1 are needed, else select "Version 2". Follow the help notes.

KinSim_v3.75.ipf (ZP 5/31/18)

  • Version 3: switch from menu control to user-friedly panel control, substantial function reorganization, and
    • Automatic input check (for basic parameters and mechanism)
    • Species concentration constrain
    • Special treatment for photolysis, emissions, and dilutions (user-defined dilution air composition)
    • Times series plots of species concentrations and reaction rates
  • Version 3.75: fix of bugs, conversion of some macros to functions, and
    • Mechanism export and import
    • Preclusion of species from dilution
    • Mechanism input in both JPL and IUPAC formats
    • Possibility of enabling/disabling reactions in the mechanism
    • Fractional product stoichiometric coefficients
    • Option of simple rate-coefficient input table
    • Translation table to build time-dependence table
    • Real-time T & P change
    • Gas-particle partitioning calculations
  • General usage: see here

Input Mechanisms

  • Basic OFR mechanism (v1.00 ZP 9/10/14)
    • Including photolytic, Ox, HOx, NOx, and SO2 reactions in OFRs
    • A few reactions have special rate constant expressions. Refer to the "Notes" column for more details.
    • Link: Basic_OFR_mechanism.pxp
  • Mechanism of CH4 full oxidation in OFRs (v1.00 ZP 9/10/14)
  • Mechanism for 2014 NO3-Terpene Chamber Studies (v1.00 DD, 6 Jan, 2015)
    • Link: NO3_Terpene_mechanism.pxp
    • Should work with latest KinSim versions, however see here for ipfs pre-version control, including additional post-processing/plotting functions and example plots images.
  • Mechanism for IEPOX-SOA (v1.00 DJ, 17 Nov, 2017)

Other Utilities

  • List/post other utilities here