Sq pk programming

From Jimenez Group Wiki
Revision as of 17:32, 27 October 2009 by DonnaS (talk | contribs) (→‎Squirrel)
Jump to: navigation, search

APES

To Do

  • Get and incorporate Jose's read me to the wiki
  • Allow users to select HR spectra from non-root data folders
  • Create some documentation on the web!
  • The default APES panel is too long on Macs.

Done

Done in latest Pika beta:

  • Fold the whole thing into Pika?
  • Delphine wanted to spit out the elemental mass spectra without normalizing them all to average total mass.
  • Add the ability to apply the AB correction factor, if it exists.

Done in latest APES release:

  • Ken had an issue when he chose the tag wave instead of the text wave as the chemical mass formula wave. Perhaps there should be a warning that appears when x number of mass fragments are flagged as having issues instead of simply printing them to history. (done in 1.02)
  • Add a new table on the web page for downloading ipf (done in 1.01)
  • Add ability to not require the fit of CO and/or CO2 (done in 1.01)
  • Add the ability to do the analysis on user selected HR spectra, such ans PMF results (done in 1.01)

Pika

Bugs fixed since last release

There was an annoying bug that resulted in not being able to choose > 1 mass fragment each for the left and right-side peak shape candidate. The code was fine in 1.05 and using Igor version < 6.10. But in Igor 6.10 there was a compilation error, and the quick fix for Igor 6.10 caused this behavior. You can download PK_peakShape_1_06A.ipf at [1]. It is also downloadable form the main ToF_AMS sofware download site.

To Do Priority

  • Tim O made a good suggestion for instances where peaks are not fit (resulting in nans). Perhaps one could compare the squirrel signal at an integer m/z, and if the value is sufficiently small, (the raw spectra shows noise), all the peak height values could be set to 0.
  • The peak width/shape panel needs tweaks to be more user friendly. The specific changes have not yet been worked out.
  • The HR_PeakHeights_gr is too small on a Mac. Add code to check for OS type and change window size for this one graph.
  • Doug wants a more clear warning when masses not fit (most likely due to nonmonotonicity in peak shape) in one spectra or many.
  • In the PeakWidth/Shape graph, change the functional form of the power function for peak width and add an "End" button.
  • In HR_peakHeights_gr, change the color of the residuals to be different from the red, orange, brown. Also add squirrel integration regions. Also, add a checkbox to add remove black lines between raw data points.
  • We may want to allow a 'smoothly varying' peak width.
  • Change all peak shape candidates to have a 0 entry in the left- and right-most value so that 'peak-shape type baselines' dont' get extended beyond +- 7 PW values.
  • Automatically generate plot of the Residual Errors vs Counting errors (for one m/z and for chosen species)
  • Begin to automatically create uncertainty values generated from peak shape and/or m/z calibration knowledge.
  • Many parameters panel
    • add small raw spectra insets to left and right
    • add graph showing spectra-like data with window highlighting where you are
    • have run numbers update with cursor move
  • Add ability to duplicate PeakHeights graph and functionality
  • In the m/z calibration panel, currently the width and accuracy vs m/z graphs don't show a summary of all the runs. E.g. the summary of all the peak widths for all the runs with error bars in the peak shape panel very nicely summarizes this information.
    • (Jose) This could be done with a radio button on top of the graph that selects SR/AR (meaning single run / all runs)
  • We need to add camels, dromedaries, and vw "bug" ions to the lists. We need some notation for these ions, let me suggest vw28, vw23, d28, d32, and c28, c32, with the comment column describing what they are. It is important that those are lowercase letters, so people don't confuse these ions with vanadium (V), tungsten (W), and deuterium (D).

To Do (Less Urgent)

  • For Pika, we need to incorporate some sort of algorithm that sets the minimum error to be 1 ion.
  • Jose wants to add other types of errors for folks to plot/use. If the current list of errors is considered to be a,b,c,d, we also want sqrt(a^2+d^2) and sqrt(a^2+b^2+d^2).
  • In average mass spec plots (including the 5-graph panel and the elemental analysis) Doug wants a way to 'move' isotopic child HR ions to their parent ion (i.e. j13C would be moved and grouped with the HR ion C.)
  • From the HR_peakHeights_gr, when adding a brand new mass to the table, add a note in the history. Also add a message when adding and removing masses to fit?
  • m/z calibration and masses - We need to subtract the mass of an electron from all our masses. Note from Donna: I want to leave this decision up to the HR list committee (Puneet, Niall, Delphine)
  • Generate 'Neil's graph' whereby we plot as sticks from zero amount of C and H mass as a function of number of Oxygens. Do a similar thing for the number of C's (the amount of H and Os and a function of number of Cs).
  • In PeakWidth/Shape panel, add ability to not do average PW fits using the std dev as weights
  • In the Graph 5 summaries, include one 'bar' type graph showing the fraction of each family to the total. Also, we may want to redo the addition of HR fragments to force the child isotopes to go with their parents.
  • In HR_PeakHeights graph perhaps color the individual peak shapes to the colors of the families.
  • Pika on PToF spectra (gasp!)
  • There is a need to consolidate the three sets of m/z lists that exist into a single one. We currently have the list of m/z possibly used in fits,(1) ones actually selected for HR sticks (2) list of m/z used for m/z calibration, and (3) the list of isolated ions. Each list is a subset of the previous one, and we can work with a single big list, and a series of masks to select / deselect the various m/z for the different purposes.
  • If user only does the m/z calibration and does not play with the HR panel, the code calculating the HR sticks may go awry (maybe - I need to check).
  • Think about NH4 measure vs predicted and RIE for NH4 (Donna: Jose's July 2008 email).
  • Make clear what limitations there are about the total number of masses that can be fit at any m/z value.

To Discuss

  • Change saving of the HR sicks matrix from masses that were selected to fit to all known masses.

Done

Done in latest beta:

  • Begin to implement the HR frag table. W need to introduce the concept of species. For example CO2 would belong partially to an HR-Organic and partially to the HR_air species. This also involves the use of an HR batch table, In this HR batch table users also their own RIE, CE. Note that this is a similar but slightly different concept than the family groupings.
  • Change default behavior so that we have two HR difference sticks one which is found by fitting the raw diff spectra (called HRDiff), and an OpenMinusClosed which is simply the open HR sticks - closed HR sticks.
    • Change the default behavior from fitting difference spectra to only fitting open and closed. James' point at the user's mtg is that we should always be looking at the HR_OpenSticks - HR_ClosedSticks.
    • Jose, 4-Apr-09: disagree with not fitting the diff, I think we should keep fitting it as for some peaks it is much more useful. However we should also show the open-closed fits (not just sticks) in the PIKA window. User should be able disable fitting the diff, and to use the diff results, even if the default it to use the open-closed results.

Done in latest release:

Squirrel

To Do Priority

  • Tim O made a good suggestion that we optionally kill some waves in the ms_mats and HR_ms_mats folder when we are done with them. This interface will likely look like a checkbox in the misc tab.
  • Add Doug's threshholding diagnostics and correction factor.
  • Incorporate John S's method for improving m/z calibration using PFT (or PFK).
  • Redo the Frag Corrections section to have just one big panel with pop-able graphs.
  • Heringa Maarten brought up the issue about smoothing the AB correction factor when users are doing V/W mode switching. The current code does not smooth across V mode runs when users alternate between v and w mode. Some code was inserted in the squirrel panel 1.46D, but has not been implemented into the panel. There needs to be a new field in the AB corrections section of the panel to prevent smoothing across i.e. V-mode runs when adjacent V mode runs are too far apart in time space.
  • Tim O really want to have PToF min and max size regions for PToF operations.
  • Incorporate the CE algorithms as identified by Tim O, Roya, etc
  • Implement Mike Cubison's suggestion of the use of pcurser in creating new todo waves.
  • Change the field analysis to warn users to check for large CO influence in ab correction; provide a sample pre and post AB correction from Tim to demonstrate
  • Add somewhere in the help/faq something about the averaging of raw spectra and the units conversion (Donna - see James email of 11/2007).
  • Doug wants the diagnostic of Closed/Diff sticks. Anything that sticks out as being very high has a different source (CO2, O, K, C).
  • Rework the entire m/z calibration fitting routine so that we fit all plausible peaks (getting peak centers and widths) at the beginning. Then we only need to select which masses to use for the m/z calibration equation.
  • In the m/z calibration panel we should use the term 'resolving power' instead of 'resolution'.
  • In m/z calibration panel, add vertical red lines for gaussian peak centers. Change black lines to gray. When saving a copy of the m/z calibration settings, write a message to the history window.
  • Manjula had the fantastic idea of including a > and < button when looking at individual m/z calibration peaks (could also be used for individual peak width peaks).
  • In m/z calibration window, include ability to nan parameters only in runs in a todo wave.
  • In the m/z calibration panel, think about the ability to tweak settings in conditions of high loadings (Mike C).
  • In Tim's Ron Brown data set the m/z calibration panel updates so slow (91,000) runs because the right side plots update also.
  • Allow for m/z fitting of PToF data

To Do (Less Urgent)

  • Add some sort of pre-compiler directive that helps users that don't have the hdf5 xop loaded.
  • For exporting 2-d matricies for import into PMF, it may be helpful to create a separate time series wave so that the big matrix that is generated isn't unnecesarily huge.
  • Tim O wants the capability to write the frag matricies for import into the DAQ.
  • Doug really likes to have time series graphs be such that grids are on midnight of every day, and labels have day of week inserted.
  • The save interval section near the get index should perhaps be removed.
  • In some data sets, many small intermediate files get written instead of a few big ones. (Some of Mike C's Arctas)
  • Add a note to the help web site that tells what the blue/red progress bars mean.
  • Leah noticed that when doing a recursion check when the frag waves are big (5000 m/z)her computer slowed waaay down. Revisit this functionality for big frag waves.
  • We may want to help folks who want to combine several squirrel experiments in cases, for example, when users have a 2 month campaign. Create a meta-squirrel?
  • Some want the ability to 'shut off' the automatic coloring that happens with a MS time series graph, for example.
  • Add the ability to do weekday/weekend diurnal time series plots (Folks can do this right now by making todo waves that reflect weekday/weekend periods. Still, it would be a convenience.
  • Deal with the popup menus that deal with long strings by using 'reflection'.
  • Doug wants to rename 'AB reference runs' to something else.
  • Make more consistent the todo wave creation scheme whereby users can input "todowave and not 1xxx and not 2yyy" for the todo wave formula
  • Change PToF code so that users have an option to plot the legend (for size distribution graphs).
  • For PToF size distributions, often the size range extends to smaller values than are useful. Add controls so that user can quickly zoom in and out of interested size ranges.
  • Add the ability to calculate diameter mean and median in PToF data (Manjula).
  • Occasionally after preprocessing, the status bar ends saying “Estimating space requirements” rather than “Done” (Donna can't reproduce this)
  • For DC marker corrections, Manjula wants to be able to enter nitrate, and then have squirrel figure out what m/zs this corresponds to.
  • Re-look at IE/AB calculations (Donna - see Tim O's email of Jan 2008).
  • In the diagnostic plots, perhaps the PToF airbeam wave should be nanned whenever the run doesn't have PToF data.
  • In m/z calibration panel, check the possible bug when interpolating across a todo wave.
  • Review todo wave name length limitations.
  • In the m/z calibration panel, make more clear what the nan param button does.
  • From Tim O:During manual (F3) saves, the mass spectral data (e.g. MSOpen_V, MSSOpen_V, etc.) are saved as simple data arrays with the m/z in the rows direction. During autosaving, even if only for one run, these waves are always saved with the m/z in the column (run number in the row dimesion). It turns out that Squirrel can read and process either, just not at the same time. Thus, while he can process the data separately, he cannot ask Squirrel to load and process autosaved data with nonautosaved data interspersed.
  • Incorporate some peak finding code as per Jesse's request.
  • Update squirrel web site.
  • From Ken: In the baseline fitting panel, when scrolling through the baseline fits for all runs in a todo list, add the ability to pause on a chosen run. It would also be great to select the direction in which the run baselines are scrolled - forward or backward.
  • Create Frag table default fragment text waves (Donna - this would would be good for the manual)
  • Tim O's issue with somehow getting the squirrel_index matrix messed up and getting the stick matrix data set being as wide (columns) as the raw spectra matrices. (Donna - is this replicable?)
  • Review the use of t_series_old
  • Review the use of the todo graph.... add checkboxes to remove todo waves?
  • Add code that calculates the mean/median for size distributions (Manjula)
  • Add an upper limit to when we generate normalization factors (current usage is to have user's create custom size interval).
  • Suggestion from Tim O: Since everyone needs to do size calibrations, can we get the fitting function into the squirrel code for use? Ideally, we would have two functions, one to fit velocity vs dva data (typical process for getting size calibration parameters) and one that takes the parameters and calculates the dva from velocity (i.e. same equation, just solved for dva rather than velocity). I can provide you with both, if you wish. I keep finding myself going back to either retype these in all of the time or copying for older *.pxp files.
  • Achim wants the ability to calculate sticks beyond 1000m/z for SP2. (It is a simple code change; Achim knows how to do it. It is not clear how this could be incorporated into squirrel for other users.)

To Discuss

  • Leah mentioned that in the legend for the average mass spectra plots the speciated total values do not use the RIE for each species. They are just the nitrate equivalent sum. Should we change the total values in the legend to reflect the species RIE values? Perhaps just make it more obvious in the legend? Similarly, when one truncates the sticks to zero, should the values in the legend be the sum of the truncated or the pre-truncated values?
  • Normalize w mode to v mode - One diagnostic:the ratio of v and w sticks with the x axis as m/z.
  • Change frag table of sulfate according to suggestion from Ann
    • Current frag_sulphate -> frag_sulphate_old
    • New frag_sulphate doesn't depend on frag_SO3 and frag_H2SO4 (Ann will wori up this new table)
    • Frag_SO3 and H2SO4 are kept in Squirrel for reference and if someone wants to put them back to use, but not used by default
  • Change frag_organics as per Allison's paper
  • Should we allow users to enter a negative value for the PToF vl parameter? (Liz, John J)
  • Should I add PAH to the default list of species? (i.e. in the MS average graphs)

Completed

  • The names for the average raw mass spec should be changed to not have the "_all" suffix. (done in 1.47I)
  • Add a button to the misc tab for dumping MSSDiff_p to intermediate files (Tim O Ron Brown data). (done in 1.47I)
  • Users may often run into out of memory issues when performing the Export Matricies in the PToF tab, because results will be #runs * #mz * #sizebins big. Perhaps there should be an option for exporting these matricies, but cutting things off a a max m/z value (150ish?) (done in 1.47I)
  • In the preprocess step, make clear the distinction between using DAQ sticks and recalcing sticks.(fixed in 1.47I)
  • Appending one size distribution on top of another (with a different todo wave) does not work.(fixed in 1.47I)
  • In the modify SI table, perhaps include the TofTypeVKey and TofTYpeWKey waves to help folks know that they are available for use.(done in 1.47I)
  • In baseline panel, when averaging many spectra, if run num > 1e6, temporary waves have an E+ in them (Liz Rainer data) (done in 1.47I).
  • In the MS tab, when one chooses to update an average mass spec plot, the legend should update (done in 1.47I).
  • In the MS tab, perhaps move all the average mass spec controls to the left, and the buttons to the right so that users don't hit one when they mean to hit the other.
  • Tim O's issue with HDF file generating errors for not being able to append to hdf files. The code needs to add the checking of errors generated during the routine closing of hdf files (done in 1.47I).
  • In pre-processing section, change the Diff Sticks = Open Sticks - Closed Sticks to that most users don't use it (Leah W) (done in 1.47I)

Time Trace

Not Urgent

  • Make the quick view program instructions more clear.


LS

To Do

Get the firstBeta release out.


EC

To Do

  • Wait for request from Delphine - nothing to do yet.
Retrieved from "http://cires1.colorado.edu/jimenez-group/wiki/index.php?title=Sq_pk_programming&oldid=4724"