From nch@roe.ac.uk Fri May 9 10:39:34 2008 Date: Fri, 2 May 2008 15:57:24 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 2/5/08 Minutes of WFAU VDFS Science Archive meeting: 2nd May 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, PMW, RSC, MAR, LGR, RGM Apologies: JPE, JMS, AL, NJC, BCW, MSH, RPB DONM: 10am, Friday 9th May 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RSC to check out XMM/2XMM. Discharged; it seems our naming convention was confusing us, and all is well after all - RSC noted "we'd just confused ourselves over the name of the database as it was renamed to TWOXMM on the public servers, so now WSA.ExternalSurvey points to the public server copy with the new name. Everything is OK." ACTION: RPB to make a full ingest DB (ahmose.WSA) backup this weekend Discharged; also the weekendly differential backups are scheduled as normal (but ETWS noted that BackupExec was playing silly buggers over the scheduling so somebody should check that the first one executes successfully this weekend). ACTION: RPB to backup master and other system DBs on thutmose prior to upgrade to SQL2005. Discharged ACTION: RPB to upgrade thutmose to SQL2005 on Monday; MSH and MAR to note downtime in the usual ways in the user interfaces. Discharged; turned out to be relatively straightforward. ACTION: RSC and NCH to check NJC's new boundary quality error bit flag algorithm. Discharged; NJC's scheme seems to cover all bases and is much simpler than the old one - a most elegant solution - RSC noted: "Nick's simple solution is a work of genius and we're kicking ourselves for not thinking about it before. Results of new algorithm are here: http://apache.roe.ac.uk/twiki/bin/view/WFAU/QualityBitFlags#Comparison_of_di ther_offset_algo Essentially many more LAS + DXS sources will be flagged as being in underexposed region as well as a few more being flagged as possible cross-talk. All boundary flag and cross-talk flags are reset and will be re-applied to all surveys for DR4 (this should be noted in the release notes), except for the UDS mosaic detections which remain identical to DR3 (the UDS intermediates will be updated for new algorithm as there are new frames anyway)." Actions partly discharged but continuing: ----------------------------------------- ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues; has been set up from the Windows side, but "permission denied" from the linux side when attempting to access /mnt/hatshepsut ACTION: ETWS to investigate /mnt/hatshepsut and liaise with ITSG as necessary ACTION: RSC to create open time/proprietary metadata DBs on public server Partly done; proprietary DB is now available for the interface; automated mirroring via cron jobs of the ingest DB metadata is not yet implemented - RSC noted: "WFCAMOPENTIME - created last Friday, just needs to be visible in interface and announced. NB: only contains non-deprecated 05A/B metadata. WFCAMPROPRIETY - interface will switch over to this when new software is released allowing a daily cron job to run and schema changes begin." Actions carried forward from 25/04/08 meeting: ---------------------------------------------- ACTION: RPB to convene a meeting of interested parties to discuss locations and importance of all data help on WFAU servers to organise a consistent backup and DR strategy. Continues; RSC noted that new data products (e.g. as for DR4) must be considered, and some kind of automated copying into backup staging space might be a good idea. ACTION: MSH to set up usernames for MAR, LGR and NCH on the new AG community Continues Specific points and new actions: -------------------------------- Project management: The team completed the monthly progress chart updates for April. WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 29th April. So the IAC request *is* serious after all - ¡Ay, caramba! Regarding photometric radial distortion correction in UKIDSS releases, just to clarify: this was *not* implemented (as agreed with SJW) in release database UKIDSSDR3PLUS owing to time constraints; it will be in DR4 et seq. The team also noted the hiccup with cfitsio, and wondered whether there may be any trouble at this end ... which version/distribution of cfitsio are we talking about? Networking: ETWS noted that operations have started on 08A, the Feb data has been transferred with a speed of ~13 MB/s. Thanks to PSB and MSH for all their efforts put in to optimise this. WSA Operations: The team discussed the rearrangement of the ingest DB detection table schema in advance of recalibration. ETWS and MAR noted that February 08A ingests and DR4 QC1 (bar LAS eyeball deprecations) should finish in advance of this weekend's incremental backup, so early next week should be fine for starting the rearrangement. So wrt backups, we will have a fall-back position if all goes wonky by restoring last weekend's full backup plus this weekend's diff; RSC noted that it would then be a good idea to keep that backup set as a legacy set not to be recycled. Once the rearrangement is complete, another full backup should be made. Hardware ETWS noted that ITSG are still having trouble getting khafre to work with 64bit Debian4.0; at this stage there's no problem in fiddling with this over the next week or so while the DB rearrangements are taking place but it would be good to have the processing power of khafre available when we are ready to start the recalibration for DR4. Software: ETWS reported: - Continued work on the recal changes to CU4. RSC noted: "SplitDetectionTable has been updated and tested to alter the true default rows in all detection tables to replace non-default cuEventIDs with defaults. Tests of software updates to support the new schema continue, following a successful test for a complete non-survey data release used for recalibration testing. An awkward problem in the UDS mosaic cross-talk flagging procedure was identified due to constituent multiframes being deprecated and their detections deleted in the schema transition - a solution is now in the works. Hopefully a new release_6 software branch based on the new schema can be rolled out next week allowing the WSA to be updated to the new schema during a suitable break in operations and following a full backup." RSC noted that the schema constraints in the dev_1_schema branch needed reparsing. ACTION: ETWS to reparse & check-in the WSA SQL schema constraints. ETWS, RSC and NCH have been working with Annette's non-survey project u/05b7 as a test case for the illumination correction; MJI has kindly agreed to supply an independently calibrated catalogue from a single paw-print as a sanity check of WFAU's implementation. Survey Data Release: Nothing new this week. Non-survey Data Release: ETWS noted that he and MAR created a temporary non-survey which showed up some features normally not so easy to spot in standard operations; this was to get an 08A non-survey dataset ready in response to an urgent request. Astrogrid deployment & Data Analysis services LGR reported that his new analysis tool for generalised IR list-driven photometry in SDSS optical images is nearly ready for testing. NCH suggested that Dan Mortlock (cc Steve Warren) and Richard McMahon be asked for guidance and test cases (and to act as beta-testers if interested) for this new facility. Miscellaneous: RGM noted a constructive LSST meeting last week; a note is on the TWiki: http://apache.roe.ac.uk/twiki/bin/view/WFAU/LSST and RGM suggested that a few of us look over their baseline schema design to provide comments from the VDFS experience (but not until after DR4). ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================