Core Minutes 3/25/2008

• Big run: As always, follow progress on the Big Run Checklist. (Richard) current hold-ups are TChain access to interleave files and reprocessing not yet working. [see details below]

  • GRBgrid (Heather) GRBgrid issues have been narrowed down to two areas:
    1. xrootd bugs in both v5.16 and v5.18. For some of the recent runs, the ntuple files being accessed via Interleave were located in xrootd - and those jobs would get hung up. Tom has since moved all the files into AFS space. We are now poised to officially upgrade to ROOT v5.18.00b which should take care of all those xrootd issues.
    2. TChain::AddFile was failing in some cases. The problem does not seem to be related to any specific set of ROOT files - as the problem has always been intermittent. The same files would often be successfully accessed, and then later an exception would be thrown for another run on that same file. Tom has noticed that most (though not all) of the failed runs are occurring on the rh3 machines, rather than on the rhel4s. We are not sure what the significance of this is though! Heather created a stand-alone ROOT application that pounded on the TChain::AddFile calls, and found that in some cases, files that truly exist on AFS, were failing to open due to non-existence. This may be another example of intermittent AFS failures. To test that possibility, Heather re-ran her jobs with the ROOT files all on local disk, and none failed.

    Julie has put in a request to get these GRBgrid runs done asap. A possible course of action is to store the ROOT files on local disk, perhaps temp space, and prefer running on rhel4 machines. Nicola has offered to start up such a set of jobs to see what happens. Richard does caution that there is an estimated 10 GB for these files, so any mechanism that tries to copy these files into temp and removed them, should be handled carefully.

    In the meantime, Heather needs to tidy up the HEAD of Interleave, and examine precisely why an bad allocation exception may be thrown in this case. (Richard) If no quick fix for Interleave is found, we'll force the GRB grid run to go and deal with the 10% failure rate.

  • Pass 6 reprocessing (Leon) [post-meeting report] As of Tuesday
    • has modified RootTupleSvc to be able to update existing merit variables
    • Heather has added an ntupleReaderSvc to RootIo which has basic functionality for reading in a merit ntuple in synchronism with the other root output files
    • Independently, Tracy has modified RootTupleSvc to read as well as write ntuples, and has used it to read in, modify and write out a merit file. (No synchronization as yet, requires the proper linking of RootIo and ntupleWriterSvc.)

    As of Wednesday morning, he's about to try to put together a reprocessing job from the above pieces.

  • Big sky run (Richard) Time is running out on the sky generation. We may punt on it and downsize to something like a final 55 day run to test out Pass6 with our best shot at residual backgrounds.
  • Disk clean-up Charlotte is making a list of all the tasks in nfs, sorted by size. We'll then remove the "to be saved" ones from the list and proceed to the actual deletion. In the meantime, our 150 TB of disk has arrived and there will be a meeting on Thursday of ATLAS, BABAR and us to negotiate an installation schedule. Of course, everyone wants theirs first. [thanks to Richard for this note after the meeting.]

• Science Tools report: Seth went through the Science Tools Update for March 25.

• Data handling: (Dan)

  • Tony the upgraded tomcat servers
  • New Oracle servers: 1 installed; additional back-up server to be installed soon. New server is undergoing stress testing, tuning.
  • Passwords have to be changed (and in future will have to be changed again, every six months. This is very painful since passwords are embedded in the code. Looking into use of Oracle "wallet", somewhat like Escrow.
  • New version of Data Catalog installed last week.
  • New email server is working well, should go at least part way towards alleviating an old bottle-neck.

• Documentation: (Chuck) Updates to ScienceTools tutorials and the ISOC web site are still in progress; should be finished by the end of the week. Coming up soon after: March Newsletter, Confluence how-to-fix pages, and information on Data Handling operations and data access.

• SAS CCB: (Richard) The Board is in the process of defining procedures. Issues on the table now are minor; it should be possible to allow them to go forward without much fuss. Approving the new Oracle server will require something more formal.

  We're looking for technology aids in a couple areas:

  • Would like something better than just the current list of Confluence news items to keep track of requests. We plan to try using Jira to track the requests and use it to set current status. The Jira content would be a link into confluence. Confluence allows us to tabulate Jira items.
  • We need a way to track all packages under control of this CCB.

• GR news: (Heather) GR v13r14 has hit the streets. This tag is very similar to GR v13r13, including just one tweak of CalXtalResponse to address any concerns in the CAL systest plots noticed in v13r13. The v13r14 systests, while still reported as "failing" are quite close. Zach provided a very detailed explanation:

  While upgrading the CalXtalResponse unit test, I noticed some minor inconsistencies in the Cal ideal calibration code, notably an integer truncation of a floating point value.

  Changing this value resulted in slightly modified Cal overall gain in ideal mode. LEX8 adc values (for example) in GR v13r13 are lower than those in v13r12 by an average of 0.0017% (varies slightly due to quantization).

  Another result is that a few crystals will no longer pass zero suppression. I saw this happen to 2 channels out of 1000 events with the vertical_muon 1gev source.

  While the new code is certainly better than the old code, the new output data could only be called 'infinitesimally' more correct than the old data. That is to say, I can tweak the calibration constants so that we get exactly the same answer as before, but still use the improved code.

  This only affects 'ideal' calibration mode - which basically means simulated data only - in fact most 'real' simulations no longer use ideal cal calibrations.

  The other "failed" plot pointed to OBF CalEnergyRaw, which to Tracy's eyes actually does not show much discrepancy between v13r14 and v13r12 at all. The OBF uses its own pedestal and gains defined in their own shareables controlled by FSW. There are two sets: ideal, which we will continue to use for typical MC runs..and another that is used for production MC and data running, where the gains and pedestals match those of the instrument.

  So in conclusion, we seem satisfied with the GR v13r14 systest outcomes..and are ready to move on. Next up is to complete our migration to ROOT v5.18.00b

  Meanwhile, we've made a new tag, GRv13r11p4, for ETE 5 incorporating changes for the new DFI. Yet another DFI release which will include filter statistics is on the horizon. (Richard) thinks he heard JJ say he might out an interim release for Bryson to play with which would expose filter status (the more critical thing for us) as well.

• Skimmer: (David C.) Not much to report. The Skimmer is not yet ready to take CELs as input; he's working on it.

• Event display: (Heather) Tony has been dusting off the web-based event display. He contacted Heather about changes needed to HepRepSvc to make it go. She cobbled something together which apparently served for him; should be officially released.

  (Leon) has been working with Joe Perl on the problem Fred has with large events. It seems not to be in the Fred client since Joe's WIRED client shows the same failure. Joe is thinking of using a different CORBA to see if that helps. Leon sent a query to Riccardo but has gotten no response yet.

• SCons, RM news: (Navid) Have seen build failures having to do with missing environment variable definitions; fixes in progress. There have also been some test program failures for unknown reasons

  New RM pages are currently down. This may be a temporary problem while Karen moves them from out behind the firewall.

  New RM will keep more information in MySQL, will not need as much disk space. However, in the short run we need to cear some space from u9.

  Code to delete exess tags is written; currently being tested.

  Gui for SCons (Joanne) As a first step in understanding the issues, last week she SConsified the little container package rdbGuiRelease on her laptop. With generous amounts of help from Navid, this went quickly. Everything builds and applications run. Next she made a list of requirements for MRStudio or other gui interface. Other MRvcmt users, especially on Windows, are urged to look this over and comment. Finally, in hopes that only some part of the MRStudio back-end would need substantial changes, she started looking at the code. No such luck. There are several problems:

  • Meeting all the requirements on the list would entail touching a substantial fraction of the MRStudio code.
  • The code is approaching impenetrable. We have no local (or even distant but reachable) Ruby expertise. The weakly-typed nature of the language and some unique features of its syntax make it that much more difficult to figure out what's going on.
  • To use the fox graphics library (written in C++) with Ruby one also needs the wrapper library FXRuby. Keeping everything up to date and in sync has become quite a burden. We are currently using an ancient version of fox partly because of this. The build and installation procedures for Ruby applications using fox (Fred, MRvcmt) are complicated in ways unlike procedures for any of our other applications.

  For all these reasons she is now leaning towards writing a brand-new gui in C++. (Jim) Since SCons is written in Python, that would seem to be a natural choice. (Joanne) Yes, in some respects, but Navid has probably already written much of the code needed to invoke SCons functions from C++ for the new RM; in any case I don't believe that is going to be the difficult part of the application, which should ideally be somewhat decoupled from the underlying build tool.

  I would like to stick with the fox graphics library; applications look good and have good performance. There is a Python wrapper for fox analogous to FXruby but it's old, built against an old version of fox, and probably not being actively maintained. Finally, I have a preference for compiled, strongly-typed languages for a moderate-sized and long-lived application like this one.

• Launch: (Richard) We need to keep an eye on progress on key pre-launch projects.

  Launch date is holding steady at May 16th. If we do somehow lose it, launch would probably move all the way to October.

Thanks

to Richard, Heather and Leon for substantial contributions to these minutes.

previous

minutes index

next

J. Bogart, Last Modified: 01-Jun-2010 15:45:58 -0700