From nch@roe.ac.uk Sat Mar 8 16:28:41 2008 Date: Fri, 7 Mar 2008 14:18:30 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 7/03/08 Minutes of WFAU VDFS Science Archive meeting: 7th March 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, PMW, RSC, MSH, LGR, RGM, MAR Apologies: JPE, JMS, AL, BCW DONM: 10am, Friday 14th March 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: AL to set up a WFAU TWiki topic on ESO-SAF interfacing (both technical and political...) Discharged; see Project Management below. ACTION: NCH to look into obtaining/licensing the most recent mxODBC Python DB middleware. Discharged; ETWS has installed (including mxBase) on all linux machines. ACTION: ETWS to get ITSG to mount public server shares on khafre: /mnt/amenhotep and /mnt/thutmose Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Continues; MAR reported started looking into producing colour images (initially as cut-outs). ACTION: NCH to bug MAR about release history/wsa-announce messages concerning the FIRST catalogue and associated crossmatch Continues; MAR will now see to this. Actions carried forward from 15/02/08 meeting: ---------------------------------------------- ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues; MSH noted that optimal settings have been determined so this can now be done by ITSG. Specific points and new actions: -------------------------------- Project management: Much discussion this week concerning potential impact of the Programmatic Review; could type loads on this but life's too short... except to say that from the UKIDSS/UKIRT angle there was a telecon of the survey heads and JAC yesterday at which plans for input to the consultation exercise were discussed. AL met with ESO-SAF folks yesterday in Munich, and has written a TWiki report on the discussions concerning VSA-SAF interfacing: http://apache.roe.ac.uk/twiki/bin/view/WFAU/VisitToESO NCH noted that an offer has been made concerning the operations post. PMW noted the VDMT meeting of next Monday at 2pm; the progress chart was filled in for February and PMW will attend and present the usual updates. WFCAM & VISTA updates: NJC noted that the Vista primary mirror has been provisionally accepted and PMW noted that it is due to fly out to Chile over Easter. Concerning responses from Vista Public Survey PIs, NCH noted that VIKING, VIDEO and VHS have all replied; nothing yet from VHS and VVV (deadline given was end Feb) while of course UltraV plan to do their own thing. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: Nothing new to report. WSA Operations: ETWS noted that 07B WFCAM transfers (12 MB/s) finished earlier in the week. ACTION: ETWS to update archive provenance in advance of QC ETWS also noted: - Installed latest mxBase and mxODBC including the new licenses. - Finished 07B ingestion, all data is now available for QC. - Ingested catalogue data for already registered non-surveys. - Speeded up the production of the monitor page by 50% using the new FLatFileLookup table. - Updated software for easier handling for general operations. - Moved the data from khufu's disks to the new NASbox. The new development disk is now disk05. Hardware MSH noted that new web server horus is now set up. Software: RSC reported: * Created the new dev_schema_1 branch, following synchronising of the existing branches, to implement the new partitioned schema and have created a new TestWSArecal database for testing these changes. Everyone has been e-mailed with the development plan for this branch, and I've been working with NJC to update the template schema and its parser in this branch. * Updated and tested all software branches to reflect system changes: The interface with mxODBC had be tested and updated for the new version, with notes on the changes detailed in the SoftwareUpgrade TWiki page. Moving disk09 to disk05 had a myriad of impacts on the software, with many scripts using hard-links to disk09, and most notably CU16 had to be updated to allow it to run on any server now with the TWiki documentation being updated too. * I've now created a test_all.py script as a crude start to aiding future full-scale software tests, of which several are expected very soon for the 64-bit update etc., and have put notes on the UsingOurDbAPI TWiki page about how to delete test databases through Python. * Working with NJC on testing and implementing a necessary revision to the dither offset calculation used to flag detections close to the image boundary and determine the direction of likely cross-talk artefacts. In anticipation of this change the ppErrBitsStatus and ppErrBits flags have all been reset for these two bits and I've corrected the default values for the PTS. * Also NJC and I have worked together to modify exnumeric and the database schemas to test the application of the photometric distortion correction. A bug was identified and fixed in CreatePixCatList, allowing a test CU4 ingest on a non-survey to be successfully achieved. This week NJC has finished the first version of CU6 - archiving synoptic data and started testing the output. NJC has also changed the template schemas to accomodate the changes necessary for recalibration and for the synoptic archive. NJC has fixed a minor bug in the dither boundary flagging and has added a totalExpTime keyword into MultiframeDetector and modified the stacking code to output the total exposure time for deep stacks. The mosaicing code still needs modifying though. NJC also has been checking the dither offsets for quality bit boundary flagging and modifying the methodology based on the results of this. With RSC NJC has been modifying the magnitudes code to calculate the distortion correction and the first stage of the illumination correction and made more modifications to the schemas to allow this. NJC has also continued some work on the synoptic pipeline, adding indices to make it run faster and for multiple filters. NCH noted that he is checking with SJW with the lower level illumination corrections are required for UKIDSS DR4 (NJC noted advice from CASU is that they should be applied). Survey Data Release: NCH noted now that 07B data is all in, QC can start ACTION: MAR to start 07B QC UKIDSS are busily (?) eyeballing jpegs and it has been requested that all results be in by the end of March. NCH noted that so far Aug, Sep and Oct for the GCS look great. MAR raised the question over disk space for DR4 on the public servers. ACTION: MAR to check disk space for putative DR4 DBs on the public servers. Non-survey Data Release: MAR reported working on some automated QC for non-surveys, and noted that all registered 07B PIs now have access to their data. Astrogrid deployment: MSH noted that new hardware for the AG redeployment in advance of NAM had hit a snag in that there seems to be a problem on a motherboard. NCH asked about the requirements for any SIAP services delivering the legacy SSS imaging data; this was not seen as a high priority, but migration of the data to the newer hardware and implementation of the required low-level pixel access codes are proceeding as background tasks in preparation anyway. LGR noted that his new SDSS pixel prototype analysis service is now implemented and is being tested in advance of presentation at NAM. Miscellaneous: Nothing else this week