From nch@roe.ac.uk Sun Mar 28 12:01:47 2010 Date: Fri, 26 Mar 2010 17:40:35 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archives weekly meeting minutes, 26/03/10 Minutes of WFAU VDFS Science Archive meeting: March 26th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RGM, RSC, RPB, MAR, NJC, PMW, MSH Apologies: JPE, AL, KTN, ETWS DoNM: 10am, Friday April **9th** 2010 in the VISTA Hut Actions discharged this week: ----------------------------- ACTION: RPB to backup the cluster-deployed SSA DB. Discharged ACTION: NCH to prod UKIDSS SHs concerning DR8 eyeballing. Discharged ACTION: NCH to contact Transit survey PIs to ask about requirements for an updated DB release. Discharged - see Non-survey below ACTION: RPB & NJC (with help when required from RSC) set up cross-neighbour tables between Stripe 82 and WSA/VSA surveys as appropriate. Discharged for VHS, VIKING, VIDEO. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 12/03/10 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: RGM noted that this is PMW's last weekly meeting. NCH thanked PMW for all his hard work over the last decade. PMW noted that he'll be gathering together all the spreadsheets and task lists etc. to pass on to KTN next week. JPE has agreed to organise a VDMT, date TBC, following a request from PMW. WFCAM & VISTA updates: Nothing in particular this week. Comments and issues arising from CASU minutes: The team noted the recent communications between NJC and MJI/JRL concerning the CASU pipeline codes needed for integration at the archive end. Many thanks to CASU for the help and info; RSC noted a concern regarding the exact versioning of the codes that are being run as there was some confusion as to which was the appropriate tool the various jobs and if we have the most up-to-date for running against VIRCam data. This will be a topic of discussion at the next VDMT (please; see also NJC's parting shot below under Software). Networking: The team noted that the recent extremely poor network transfer performance CASU-WFAU was finally traced to a dodgy 10 femtobit/s fibre switch installed at CASU. The embarrassing trickle of 1MB/s was then initially restored to a satisfying flow of 7MB/s, followed closely by improvement to a proud stream of 12MB/s as originally up to the end of last year. MSH noted that his monitoring at this end shows that the figure of 12MB is an average, and that the situation is actually bursts of 26MB/s followed by equally long pauses of zero, and that he would be looking into this. In the week, JPE circulated a report on the Paranal-Garching network link; MSH noted that he had sent some comments and suggestions. NCH wondered what the situation would be in transfering Garching-CASU if the carpet bags of disks are no longer being used... WSA/VSA Operations: RPB noted that the reingestion of the nights of 09A July 12/13 following recovery of file nos. > 999 has caused an ingestion hiccup (well, more of a great big belch plus a little bit of vom). Anyway, ETWS is off on Easter hols and it seems considerable operational knowledge resides only in his head as opposed to also on the TWiki - this is unfortunate. RSC volunteered to help RPB look into the PK violations etc. Interestingly, transfer and reingest of some broken files from 08B caused no such problems, but RPB noted that regeneration of the deep stacks was causing some issues. Again, RSC volunteered to help out. v0.8 VIRCam data is now slurping up at a great rate (see previously). RSC noted that we should try to be ready for ingestion into a recreated VSA DB before the transfers finish if at all possible. Hardware and Systems: MSH reported (earlier in the week): "Menkaure is now up and running on the infiniband network. An initial test indicated around a 7x faster permformance on copying data from the cluster to the local disks. All the necessary changes have been made to the hosts file so it will default to the infiniband network when connecting to the cluster and hatshepsut. Hatsheput is pretty much ready to go too. I'm running some final tests for generating UKIDSS statistics on it, and after they complete it's free to use. It has three disk volumes of 1TB each that are labeled G, H, and I as requested. There is a share folder on the G drive that is already mounted on the curation servers." MSH also noted that work on the infiband driver/firmware updates to the MS Windows cluster nodes will continue next week, and that the new VO server "fenrir" has arrived. RPB noted that Eclipse will be in later today to install heavy-duty rackmount rails for the rather massive 96TB NAS box. PMW note that an LTO4 drive has been ordered for the tape library. Software: RSC reported: "We'll need to release NJC's improvements to the template schema design prior to recreating the VSA for v0.8 data ingest. So I've been concentrating on ensuring the latest trunk version of the software is passing all regression tests, which has required many modifications to ensure non-survey releases continue to work smoothly. I've also started to implement the switch to supplying scripts with a compulsory command-line argument for the database rather than this being an option. However, this change may not be ready in time for the next release." NJC reported: "This week I have been putting final touches to the ProgrammeBuilder driven schemas, including indices. I have also worked on CU9, sorting out the arrangement of data for this, and doing some basic tests. More tests will have to wait until we have the new CASU software. I have emailed Jim and Mike about the version 0.8 software and I have had responses from Jim, which have indicated that it is almost ready, but will be run in slightly different ways to before, so we will need a few software changes and it seems that we have been running a version of the stacker that was not the main pipeline version. It would have been nice if this had been made clear earlier in the last 5 years." Regarding schema updates for the latest VIRCam (V0.8) processed files, NJC noted that the catalogue changes are all done, but that another look through the image files is probably in order to check for new additions (in particular a new version attribute). ACTION: NJC to check the v0.8 VIRCam data for new image metadata attributes. MAR has made a small tweak to the UI crossid facility to correct a glitch. Survey Data Release: MAR noted that he'll be asking the DXS and GPS PIs to eyeball some additional data from the nights of 12/13 July that got lost first time around upstream in the VDFS (once they're ingested). Non-survey Data Release: NCH noted that following some communications with the transit campaign PIs, there were no particularly urgent requirements for a proprietary or world release of a synoptic catalogue dataset, and that given the pressure of other work such a release will be put on the back-burner until such time as there is a more specific request, or even until WFCAM observations for the campaign are completed. RSC asked about the status of the four UH non-survey datasets held back for an additional proprietary period. ACTION: RPB to append U/06B/H54, U/07A/H16, U/07B/H60, and U/07B/H60B to WFCAMOPENTIME for world use. Astrogrid deployment & Data Analysis services: MSH noted that in response to a user request, help has been given to enable bulk secure cone-search queries on UKIDSSDR7+, and that sofar things seem to be going fine (MAR expressed surprise that the VO server could cope... ). Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================