From nch@roe.ac.uk Fri Jul 18 14:12:54 2008 Date: Fri, 18 Jul 2008 14:08:19 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 18/7/08 Minutes of WFAU VDFS Science Archive meeting: 18th July 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, ETWS, PMW, RSC, MSH, NJC Apologies: JPE, JMS, AL, LGR, RGM, BCW, MAR DoNM: 10am, Friday 25th July 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH, RSC, MAR, ETWS, NJC & RPB to meet next Friday 4th July 2pm to sort out non-survey curation automation Discharged; see Non Survey below ACTION: NJC to email Seb/Omar to see how they're getting on with the DR4 UDS Discharged; a cautious response from Omar, but the indefatigable Seb is aiming for early August for the DR4 stack to be ready for ingestion. Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 4/07/08 meeting: ---------------------------------------------- ACTION: RPB to formulate backup plan arising from meeting Continues; currently wholesale new backups running; formal policy to be documented. Specific points and new actions: -------------------------------- Project management: Nothing new this week. WFCAM & VISTA updates: Nothing new this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week Networking: Nothing of note. WSA Operations: ETWS noted that the ingest of reprocessed GPS CU4 data has finished and he is checking and fixing missing files; finished the CU1 makeover and it is now up and running transfers of the May and June 08A data; also started work on the code for general and non-survey post-CU3 processing. RPB reported: "Tests have been performed to compare the speed of a database mounted on a NAS box compared with the same database on directly attached storage. A trawl on the database which completed in 13m 54s on the directly attached storage took over found and a half hours to complete on the NAS mounted database. It was concluded that shared storage on a NAS box is not a suitable solution to the storage problems. Results are summarised in http://apache.roe.ac.uk/twiki/bin/view/WFAU/NasMountedDatabaseTests BestDR5 on Thutmose was updated with the latest version of the QuasarCatalog table from SDSS. In line with RGM's concerns about becoming out of synch with the official released copy of BestDR5, the table was uploaded to the server as "QuasarCatalogBrian", since it was requested by BCW. The ongoing problem with /mnt/ahmose being impossible to mount from any machine except khufu has been fixed. The mount point was effectively deleted and recreated, and this mounted perfectly happily. Still not sure what the problem was, but at least have found a way to fix it." Finally, NCH asked about the SSA rearrangements. RPB reported that the new full SSA is backed up and now BestDR2 is restoring on hatshepsut. The old DB needs backed up from amenhotep before deletion. ACTION: RPB to finish off rearranging the SSA. Hardware As reported above, RPB noted that DB trawl tests of NAS mounted DB files perform badly as expected (e.g. 13min for a locally attached storage trawl versus over 4hr over NAS). The team noted that this takes NAS out of contention for the upgraded higher capacity DB server solution. We will continue to pursue SAN, or failing that, locally attached storage (e.g. as in hatshepsut's SAS) solutions. ETWS, RPB and MSH noted that ahmose now seems to be all better following misbehaving earlier in the week. MSH noted that web server horus needs rebooted to clear a zombified process, and NCH noted that it's time sync seems to be 1hr ahead of BST for some reason. ACTION: MSH to check out thoth and reboot scheduled Sunday evening. MSH noted that he has the go-ahead to recast a little-used VOTech development server as a test-bed for 64-bit Windows + SQL Server. Software: RSC noted: "Implemented in CU19 the on-the-fly creation of the Source table from the MergeSource table for the release database. Now investigating the efficiency of this process and ways to optimise the (re)seaming of the MergeSource table. Updated CU16 & the WSA to exclude deprecated detections in SourceXDetection neighbour tables." NJC has been shaking down SetupProgrammeRequirements and CU6 with non-survey data and updating the Twiki page with known issues/possible solutions. SetupProgrammeRequirements and CU6 mostly work, but need a little tidying up in places. Another week of tests should sort them out. Survey Data Release: Non-survey Data Release: Following a marathon non-survey support meeting last Friday afternoon, various actions have been discharged. NJC had already considered most of the issues in automating the set-up of non-survey databases; NCH has prototyped the QC procedure and handed on to RSC for implementation; MAR will update the registration website; and RSC and ETWS will work on streamlining the post-ingestion (CU3/CU4) process in the light of the helper scripts developed by NJC. The idea is to simplify the registration process (users will always get merged source tables) and automate the production of stacking, cross matching, etc. using a simplified schema prescription. The result of all of this is that production of non-survey databases will become much more automated and a routine part of operations. For internal developer use see NonSurveyAutomation TWiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/NonSurveyAutomation A couple of enquiries have been received concerning the calibration data set in the WSA. A potential release of a non-survey database of all the calibration data will have to wait while the non-survey science programmes are sorted out. Finally, NCH noted that a new flipping non-survey mode is under test at JAC and CASU and that we need to keep an eye open for the data arriving at the archive to check that it all ingests OK. Other than that, NCH emphasised that flat-file access only will be provided for such new non-survey like observing modes (i.e. there will be no new curation modes developed for such things, so no enhanced DB-driven products can be expected). Astrogrid deployment & Data Analysis services MSH noted that UKIDSS DR4 is now available through the AG DSA. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================