From pmw@roe.ac.uk Mon Dec 11 15:07:44 2006 Date: Fri, 8 Dec 2006 17:08:30 -0000 From: Peredur Williams To: Nigel Hambly , Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Stephen Warren Subject: WFAU Science Archive weekly project meeting minutes, 8/12/06 Minutes of WFAU VDFS Science Archive meeting: 8th December 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: RSC, MAR, JB, NJC, ETWS, LGR, BCW, MSH, PMW Apologies: AL, NCH (at UKIRT), JPE, RGM, JMS, JDT DONM: 10am, Friday 17th December 2006 in the Plate Library (MAR in the chair as NCH is at UKIRT until the 12th and then on hols until 23rd). Actions discharged: ------------------- Project management: PMW noted that we have received no feedback concerning the VDFS FDR in October, and the date of the next VDMT has yet to be confirmed. ACTION: PMW to contact JPE regarding VDFS FDR feedback and the date of the next VDMT. The FDR Report was received from JPE and circulated to group members with thanks for their hard work. RSC has put a copy of the report on the Twiki. PMW e-mailed JPE suggesting the next VDMT meeting be a face-to-face one in January given absences of members and approach of holidays. Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. Specific points and new actions: -------------------------------- Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB reported that he had been in contact with Sam Wilson (EUCS) about sorting out the hardware for the UKLight link and apparently it just needs UKLight to okay a piece of hardware that EUCS is lending them, then Sam can install it, sort the other necessary cables and it's up to UKLight to give us the actual link. ETWS noted that transfer of BestDR5 from Johns Hopkins is almost done, with 50 of the 64 files checksummed OK. WSA Operations: JB reported CU4 had been completed on 06Av1 and all broken files had been fixed. Provenance was redone from scratch due to a number of old problems and has now finished. Quality Bits flags have now been updated for UDS, DXS and LAS. Work on the GPS is in progress. JB is planning a WSA database backup this weekend or next week, which would have some impact on Ahmose performance. Hardware: JB is still awaiting a replacement spare disk for Amenhotep/Ahmose. Another two disks on Thumose broke, he replaced one and put the other back online, after the rebuild had finished the second one broke badly and needs replacement. He had contacted Ian Davidson about this. He is looking at the configuration options for the NAS but has yet to set it up. Software: RSC reported: "I've mostly been working on squashing bugs this week to ensure the data release schedule stays on track. Finding out thorough details of duplicate multiframeIDs in database so that Johann can delete them with confidence. Investigating the extent of incorrect entries in the ProvenanceTable, which has now been completely fixed up. Making CU7 handle non-surveys with single-passbands more gracefully (until a better solution is found), and placing a check that quality bit flags have been set before continuing with source merging. Also, providing Nick with further support in coding the new recalibration scripts. Updated the WSA schema to add the quality bit flag status attribute, ppErrBitsStatus, to the table ProgrammeFrame, to enable detction quality bit flagging to begin. The performance for setting the first four quality bit flags has been fantastic, with the longest run so far being just 1.5 hours (for the GCS, the second largest survey w.r.t. number of detections). I expect GPS to be completed today, so we expect this stage to just take approximately one day in total, sans cross-talk flagging. I've also re-configured the front page of the TWiki slightly, merging the WSA/VSA/VDFS links into a single "VDFS Science Archives" page, emphasising that this is all essentially the same project." ETWS reported development of a general wrapper to parallelise running of CU 1 to 4 and farm them out to our computers had progressed. He had also produced the xml metadata of the external catalogues for Astrogrid and the updated version of the browser pages including the new non-survey catalogues. NJC reported that he had changed the updateFITS and reCalibration codes to take into account the new FITS keyword 'NIGHTNUM' added in by CASU. He had also changed the relevant schema and tested the changes. He had removed updates to Multiframe.photZP since the primary header zeropoint has become obsolete. PreviousMFZP has also been deprecated. The tests had gone well, with updateFITS almost ready to be used in anger. recCalibration needs some more testing. Speedtests haven't been possible since SDSS DR5 is being downloaded at the same time onto the same disks. MAR reported he had adapted output of MultiGetImage per user request for fixed sized cross-hairs on jpegs. MAR has started QC1 for 06A data. Survey Data Release: Reiterating the proto-schedule from last week, minus another week of transfer/ingests: Complete transfer of 06A : Done " ingest " " : done Photometric recal : Still not finalised by the Calibration WG, but following discussions with SJW it has been agreed that QC can proceed without the the final calibration, so this is no longer on the critical path (but will be if not delivered by mid-December at the latest) Quality bit flagging : +1 (can design & test in parallel with previous) QC1 : +1 (06A in progress) CU7 (source merging) : +3 (again totally dominated by GPS, and has to be done from scratch because of q-bits and recal) DXS CUs : not on the critical path, since much can be done in parallel with previous; later CUs run fast on the relatively small amount of data) CU16 : +1 QC2 : +1 Final CUs : +3 ... plus a couple of weeks for Xmas. Non-survey Data Release: CU18 has been run for Non-Surveys and two non-surveys have been built, copied, attached and released; U/05A/100 and U/SERV/1652. These non-surveys were used to test UpdateQualityFlags, CU7, CU18 and CU19. Astrogrid deployment: MSH has arranged to be able to get emergency access to C1 during the holidays if servers have to be reset. Miscellaneous: Nothing else this week. E. and O. E.