From nch@roe.ac.uk Fri Jun 16 15:28:06 2006 Date: Fri, 16 Jun 2006 12:32:01 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 16th June 2006 Minutes of WFCAM Science Archive meeting: 16th June 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: MAR, NCH, NJC, PMW, RSC, ETWS, MSH Apologies: AL, JPE, JMS, RGM, JB NB: DONM: 10am, Friday 23rd June 2006 Actions discharged this week: ----------------------------- ACTION: PMW to take around the progress charts for filling in work done for May. Discharged Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 2/06/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES: Note that this is on hold until after DR1. ACTION: JMS to contact JPE regarding progress on October UK VDFS review organisation - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted that the Vista Public Surveys Panel (PSP) meets at ROE next Tuesday/Wednesday(am) for co-ordination of the selected survey proposals. Both panel members and PIs will be present and available for informal chats during lunch etc. It is not clear yet how much participation from VDFS archive team members would be required during the meeting itself. PMW noted that a date has been fixed for the 25th July for the next UKIDSS consortium meeting at Edinburgh. The latest UKIDSS newsletter is available on the Astrogrid Wiki at http://wiki.astrogrid.org/bin/view/UKIDSS/VarDocs#News10 WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes as of this morning. Networking: ETWS noted that the latest transfers from CASU started out well at rate 13 MB/s but then dropped yiedling a final, just about acceptable, average rate of 6 MB/s. The UKLight bid is now in the system. WSA Operations: NCH gave an update on QC: this has been closed out for one shallow survey (the GCS) to enable progression to later CUs; NCH promised NJC that he would close out the DXS QC today since the eyeball deprecations for the DXS have now been received. Otherwise, all is normal on the operations front. Hardware: PMW suggested that MSH start looking into central computing services provision of mass storage, namely the SAN at Roslin. The team discussed the baseline requirements, and NCH suggested that MSH ask RGM in the first instance for contact names to get the ball rolling, noting that JB should be kept in the loop also since he has been looking into NAS solutions for local storage in the interim. ACTION: MSH to begin looking into provision of mass storage at the University's SAN at Roslin. Software: NCH and NJC noted that outstanding issues regarding the CASU source extraction software (namely enhanced sky noise estimation, and accurate parameterisation of oversampled images) need to be closed out asap in order to progress with the DXS/UDS, otherwise the fall-back solution of using S-Extractor would have to be invoked for the UDS at least. ACTION: NCH to email MJI concerning the status of enhancements to CASUExtractor RSC noted some minor issues regarding the source merging software (CU7). An obscure NFS bug sometimes results in our own wonderfully efficient software getting ahead of DB outgests from the catalogue servers; this bug needs to be fixed asap for DR1 ACTION: RSC to apply a work-around to the CU7 NFS bug between applications software on client and DB server. RSC also noted that he and JB have now enhanced CU16 to be able to access external catalogues on any named catalogue server, and as a result we will be able to provide SDSS-DR3 cross matched with UKIDSS-DR1. MAR reported: "Worked on the jpeg/fits cut-out code to make sure it works and is effecient on large files. Will probably still need to put a limit in on number of pixels extracted as some of the deep stack mosaics are around 25000X25000 pixels." NJC reported: "This week I have been mainly working on a wrapper for SExtractor. This has progressed well: Emmanuel Bertin has told me that the ISO levels are at exponential levels between the detection threshold and peak level for each galaxy, whereas CASU goes up in factors of 2 from the detection level. Mike Irwin emailed me about Hall magnitudes, with a bit more info, but he said to be specific he would have to look up the code. He also said that he doesn't use them much himself and doesn't think they are so useful these" Survey Data Release: Following last week's rant from NCH, an emergency executive decision was made by AL, SJW and NCH to go with 05A processing version 2, not 3 as originally intended, for the DR1 release. This week's rant concerns the UDS. It now transpires (now that the WG, i.e. our friends in Nottingham, have looked at the 05B data) that the standard VDFS products are unacceptable for the UDS. There are criticisms concerning sky subtraction, standard stacking SW (nearest-neighbour interpolation at resampling as opposed to any higher order technique), and of course source extraction. NCH wondered why the folks in question had not raised some of these issues sooner than four weeks before the release deadline. In any case, the UDS would prefer to use Terapix/Swarp and S-Extractor to produce the UDS data for DR1, possibly back-tracking to interleaved frames (or even, god forbid, normal frames) to revisit the sky subtraction. This is all well and good, except there is now little time to do all this work, let alone get the data into the archive. Omar & Seb are coming to Edinburgh next week on the 21/22 to discuss all of these issues and to try to do some comparisons and reach a decision on exactly what is to be released. In the meantime, NJC has been hard at work looking at squeezing S-Extractor attribute output into the archive data model, mostly successfully. We could do without this sub-optimal, eleventh-hour arsing around, though. There is now a big danger that the UDS will hold up the DR1 release... Another rant concerning the DXS has been intercepted by the DXS person finally getting the eyeball deprecations for QC to NCH, after the latter spat his dummy out at the beginning of the week over the lack of input from UKIDSS. Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts We are now between 1 and 2 weeks behind the ideal schedule... need to catchup with source merging the shallow surveys next week. Non-survey Data Release: JB and MAR have pointed out the requirement (overlooked by NCH) for source merging for seamless catalogues even when a non-survey dataset is single-passband. This will require a few software tweaks, although nothing major, and will be done after DR1. Mar reported releasing 10 new 05B non-survey DBs and informing the users. Astrogrid deployment: MSH noted that the AG servers are now switched on and ready for OS installation, but that a show-stopper has been hit in that the locally recommended linux flavour Debian does not support 64-bit architecture. (This is an issue for WSA server khafre also - it seems we will be unable to take advantage of the power of 64 bits). NCH expressed amazement that somebody in IT Support (you all know who) hadn't pointed this out sooner... resisting the temptation to introduce more expletives at this point since these minutes will go on the locally public TWiki. Anyway, MSH is bravely looking into alternatives to Debian to get around this... the team wished him the best of British luck on that score. Miscellaneous: Finally, NCH noted that another ADASS announcement has recently gone out concerning the next meeting in Tucson, Arizona, 15-18th October. NJC volunteered to attend this year; PMW noted that we can send one other person if anyone fancies a trip States-side in the Autumn (or should that be the Fall).