From nch@roe.ac.uk Fri May 12 17:07:45 2006 Date: Fri, 12 May 2006 12:16:12 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 12th May 2006 Minutes of WFCAM Science Archive meeting: 12th May 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: JB, MAR, ETWS, NCH, NJC, PMW, RSC, MSH Apologies: AL, JPE, JMS, RGM DONM: 10am, Friday 19th May 2006 Hearty congratulations to RGM & Kay on the delivery of a bouncing baby Alexander William Tisdall Mann (henceforth AWTM if/when appearing in these minutes...) Actions discharged this week: ----------------------------- ACTION: NJC, RSC, NCH, ETWS & MAR to meet Wed 3rd May, 2pm to finalise arbitrary multi-epoch survey data model. Discharged; synoptic survey data model (appropriate for both UKIDSS planetary transit survey and calibration data, and probably certain Vista surveys as well) has been discussed and agreed. NJC and NCH will document. ACTION: NJC to liaise with MJI & JRL concerning obtaining the latest versions of the CASU toolkit codes. Discharged; contact has been established and new versions of various toolkit codes are being integrated and tested. Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 5/05/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES Note that these are on hold until after DR1. Specific points and new actions: -------------------------------- Project management: NCH noted that he has asked JMS and JPE about the autumn UK VDFS review schedule. WFCAM update: WFCAM is now producing (yet more) data... Comments and issues arising from CASU fortnightly minutes: No new minutes as of this morning. Networking: JB reported that UKLight is progressing slowly, an application to submit has been drafted and will be circulated for comments asap. WSA Operations: NCH noted that an(other) ingest glitch had been identified during the week, but that this was not holding preparations for DR1 and would in any case be fixed by the end of the coming weekend. JB reported: "System backups and a full backup of the WSA have been made. Bug fixes, software cleanups and additions have meant that CU3 and CU4 have had to be rerun for various bits of data in 05B and having also dealt with a few broken files I am now just finishing rerunning CU4 for missed December and January data (August through November having already been done). Provenance, though having been run now needs to be rerun and checks that there are no further problems with CU4 ingestions also need to be done but overall 05B is looking in good shape and pretty much finished up to QC1. CU1 has been run for more of the 05A data, we now have up to mid April. The WSA has had it files shrunk and set to only grow at 1%, this house keeping means that the WSA is a little more managable in size for the moment." NCH noted that we have scrubbed the version 2 processed individual microstep frames from UKIDSS in 05A to make room for reprocessed 05A data. ETWS reported that along with some JPEG juggling, this has freed up 1.1 TB of space which should keep us going until the next file storage node is on line. With reference to retransfer/reingest of the reprocessed 05A data, CASU have now made available all of April bar 11th and much of May; transfers will continue over this weekend along with ingest of the already transfered data. Hardware: New high-specification catalogue server thutmose is online; JB will test out by attempting a SDSS DR3 restore with a view to including this in the UKIDSS DR1 release (but no promises at this stage!). No news on the next file storage node khafre. Software: NCH reported continuing work on CU7 procedure and code. Major enhancements have been implemented to QC1 (see below). ETWS reported a number of enhancements to provide more stage zero QC sanity checks on ingested data. NCH emphasised that all curation application codes must make it clear to the operator when something may have gone wrong during processing. RSC reported: "I've upgraded the new Cu16 script to produce neighbour tables from programme detection tables as well as source tables to enable the PtsDetectionNeighbours table to be released with DR1. This was tested successfully. Also, I've updated the Multiframe table in the database with the new "ignore" attribute values, which incidentally lead on to the discovery and subsequent investigation of a dodgy sky frame. Previously, investigated an issue of SkySubScale values being set to integer 1 values. Nick and I have been integrating the latest version of the CASU source extractor, cirdir, into our source code repository, whilst identifying changes. Also we have tested out large file support in the new version of CFITSIO (v3.06) for producing mosaics. When I haven't been working on high priority tasks my time has been spent ensuring the code documentation is up to date, the overall code design is consistent (to aid documentation), and also further developing the overall OO model for the Python code." MAR reported: "Altered acrhive listing to cope with deleted pixel files. Readied for usre release the 3 nonSurvey progs that JB has created DBs on Amenhotep. Just need to update the Schema browser prior to notifying users." Survey Data Release: Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts NJC reported good progress with large-file support for application codes to produce the UDS mosaic. Software installation and tests will continue with RSC to make sure that the latest CASU toolkit codes are in place to do the final runs on DXS and UDS for DR1. NJC has asked the UDS guys in Nottingham to review the test mosaic. NCH reported that he and SJW have got a little bogged down in the QC, having realised that the situation as regards repeated data (both within MSBs inside groups, and with entire MSB reobservations) is more complicated than previously thought. Some instances of apparent pipeline malfunction have been identified and communicated to CASU. Hopefully, the final procedural issues can be closed out next week and 05B QC finalised within a week or so. Non-survey Data Release: JB reported that three more non-survey databases have been created and the info passed to MAR so they can be released. Astrogrid deployment: MSH stated that the VO side are happy with the accomodation available for the new Astrogrid servers. Finally, NCH asked if there was any further news on security updates in the DSA publishing software, and if the WSA team needed to push ahead with integrating the WSA into the AG publishing system. MSH replied that as far as he was aware, some remaining issues needed to be sorted out in the DSA. Miscellaneous: MSH asked the team if they would like a discussion forum set up for the WSA user community. After some discussions about integrating the existing wsa-support system, the team felt that this sounded good and asked MSH to set up a prototype for review internally before going live to the outside world.