Core Mind-Meld
Dec. 11, 2003
Attendees: Joanne Bogart, Toby Burnett, Richard Dubois, Dan Flath, Navid Golpayegani, Heather Kelly, Matt Langston, Sean Robinson, Alex Schlessinger
We began with a discussion concerning the Root to ODBC interface. Bill is interested in using it as his interface to ROOT for his Insightful Miner code. Matt believes this will be made available by Christmas. The remaining work involves creating an Windows installer. Joanne wondered if the functionality to be gained by this would also be attractive to any UNIX users. Toby remarked that MySQL could probably be made to serve a similar function, if necessary.
External Libraries Part I
Richard suggested that we should start creating tarballs of the external
libraries and source code directly from the builds created by the Release
Manager. This bundling of the libraries and source could be part of the
automated procedure.
Pipeline News (aka why we love OPUS)
Alex began with an update on the Database backend. The last couple of
weeks have been spent writing the stored procedures for Oracle. They are
now running and have been tested. PERL functions are being written to call
the stored procedures so that they can be called from a shell script.
Richard asked when will this be ready? SOON
A little digression into how RedHat 8 may soon be unavailable - requiring a
rebuild of OPUS on RedHat 9.
Dan discussed OPUS at length and drew a pretty picture.
glast04 talks to OPUS and is the place where a script is started that starts up an LSF script to start processing. The script on glast04 waits until the LSF script (that runs on the farm) returns. Both scripts end up updating the Oracle database. For each stage of the pipeline there is a script, for example a script to start up Gleam, another to handle ROOT querying of the output files to extract number of events, etc.
Matt asked about using Oracle event notification rather than scripts to communicate. Richard stated that this may be a worthy improvement - it should wait until the initial version of OPUS is out the door.
Richard asked: What does OPUS buy us?
Dan answers: Built in ability for inter-process communication.
Allows for check-pointing, such that if a process reaches a milestone and then
later crashes..the process can be re-started from the point of that last
milestone. Also provides a visualization facility for the processes so you
can view what is going on, and can direct a process to re-start at a particular
point.
Alex feels that OPUS is big and is a bit of a black box...how long will it take us to use it productively? Writing our own scripts might be a better path.
Dan suggested that we should get OPUS going, we have gone this far, the work should be completed. After that, work could begin on an alternative that may be better, but in the meantime we will have something that works. Dan believes this could be ready by the holidays.
Matt checked the OPUS FAQ - found the following entry: Couldn't we just use a shell script to do the same thing? The Answer: Yes You Can and that might even be the low cost solution for a low volume project. But the FAQ entry goes on to caution you that as the pipeline becomes more complex - your lovely little set of scripts may not serve you so well.
Richard cautioned that the pipeline must be replicated at the SSC. If we use the OPUS interface correctly, it should be possible to easily copy the pipeline to the SSC. Joanne was having trouble seeing how this differed much from, say, writing Perl scripts carefully. LSF and Oracle functions have to be isolated in any case since LSF and Oracle might have to be replaced by local equivalents.
Toby chimed in stating.. what about the GRID? That might be the ultimate direction for us, when it is ready.
Windows Release Manager
Code is available and is in CVS that could handle a Windows Release
Manager. SCS disscussion..about what they will and will not allow.
[Minute-taker exits to give permission for her daughter to take part in a wagon
ride] Alex will clean up the windows release manager code after OPUS is up and
running and ass it on to Navid.
TagCollector
Navid promises an updated tagCollector that can peek through the SLAC
firewall by tomorrow.
Next Round of Upgrades
Now that we are beyond DC1 - it's time to consider upgrades of our external
libraries, compilers..etc. Toby suggests the following:
New CMT - no known compatibility problems, so there should not be any
required upgrades to our requirements files.
New Gaudi
New ROOT
We would like to finally move to gcc 3.2 - but that seems to require a move to RedHat 9. Apparently there is some question as to whether there will be a move to RedHat Enterprise in March. Richard volunteered to talk to SCS to ask what their plans are as far as RedHat 9/Enterprise is concerned and will specifically ask if they will do a standard install.
Navid suggested that we consider building static libraries to avoid these RedHat upgrade problems in the future. Our release could be one large executable and perhaps some smaller set of shared libraries - such as the ones required by Gaudi. Toby went on to mention that CMT has the notion of a release area. Navid will take a look at this now while Alex is making the Windows Release Manager pretty.
Going back to external libraries: Matt agreed to work on creating VS 2003 binaries for our external libraries and will huddle with Toby.
Action Items:
F. Lastname Last Modified: 2010-06-01 15:45:20 -0700