From nch@roe.ac.uk Fri May 9 13:47:47 2008 Date: Fri, 9 May 2008 12:26:35 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 9/5/08 Minutes of WFAU VDFS Science Archive meeting: 9th May 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, RSC, MAR, LGR, MSH Apologies: JPE, JMS, AL, NJC, BCW, RPB, RGM, ETWS DONM: 10am, Friday 16th May 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MSH to liaise with IT Support over set up of NFS file share from hatshepsut Continues; has been set up from the Windows side, but "permission denied" from the linux side when attempting to access /mnt/hatshepsut ACTION: ETWS to investigate /mnt/hatshepsut and liaise with ITSG as necessary All discharged; turns out user name mapping was not set up. NCH has sorted this out, with linux/windows maps mar<=>mar, wfu<=>scos and nch<=>nch and the share is accessible now from the linux side. ACTION: MSH to set up usernames for MAR, LGR and NCH on the new AG community Discharged; some initial teething troubles with logins have been sorted out and it all appears to work OK now. ACTION: ETWS to reparse & check-in the WSA SQL schema constraints. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to create open time/proprietary metadata DBs on public server Partly done; proprietary DB is now available from the interface; automated mirroring via cron jobs of the ingest DB metadata is not yet implemented - RSC noted: "WFCAMOPENTIME - created last Friday, just needs to be visible in interface and announced. WFCAMPROPRIETY - interface will switch over to this when new software is released allowing a daily cron job to run and schema changes begin." Actions carried forward from 02/05/08 meeting: ---------------------------------------------- ACTION: RPB to convene a meeting of interested parties to discuss locations and importance of all data help on WFAU servers to organise a consistent backup and DR strategy. Continues; RSC noted that new data products (e.g. as for DR4) must be considered, and some kind of automated copying into backup staging space might be a good idea. Specific points and new actions: -------------------------------- Project management: NCH noted that RSC will represent WFAU at the UKIRT Board meeting next Thursday. WFCAM & VISTA updates: Nothing of note this week. Comments and issues arising from CASU fortnightly minutes: With reference to last week's minutes: "The team also noted the hiccup with cfitsio, and wondered whether there may be any trouble at this end ... which version/distribution of cfitsio are we talking about?" MJI kindly replied that the bug was in all versions of cfitsio up until the latest one but only affected summit conversions, not the stuff CASU sends to WFAU nor our uncompressions. Networking: MSH noted that UKLight will have some downtime during the day, but that this won't affect ops since data transfers from CASU are all up-to-date. WSA Operations: NCH noted that the last weekendly incremental ingest DB backup appears to have worked fine, so we have a complete backup as at immediately before the big *Detection table partitioning prior to marathon recalibration. RSC recommended we grab the set of full+diffs backup tapes and keep safe in case of any hiccups - NCH has done this, and the tapes are in the cupboard in his office. RSC instigated the partitioning script yesterday; RSC reported: "For the detection table splitting, I've put GPS and CAL to the back of the list, and I think the non-surveys will be copied first. Also, I will be copying the UDS detections marked as reprocessed for the mosaic cross-talk flagging after-all, as I discovered earlier in the week that just using CurrentAstrometry.max/minRA/Dec doesn't help since I rely on the original WFCAM detections anyway to ensure the TWOMASS star is detected. It's not worth rewriting the algorithm to avoid the need for WFCAM detections, especially since we won't be doing any cross-talk flagging of mosaics for DR4, we'll just be copying the flags back over from DR3." RSC anticipates completion early next week. See "Survey release" below for the forward plan. Hardware MSH noted that he will be fettling NAS box takelot and new web server horus mounts during the day. ETWS informed us earlier in the week that ITSG have successfully installed 64bit Debian4.0 on khafre, so the software installations/tests can now take place when ETWS returns from hols. Software: RSC has been identifying various software issues as follows: * regarding photometric field corrections in CU4/exnumeric: in the code we skip calculation of the photometric distortion correction for mosaics now, but NJC highlighted an issue with deep stacks with large position offsets. Is this still a problem or do we have neat solution for it in the code? * some worries over efficiency and lookouts of cursor.fetchone/many in Python mxODBC - RSC and MAR will try to investigate with server-side SQL traces; * release database query plans sometimes don't use indexes when they should; MAR discovered that sp_updatestatistics seems to cure this problem; Survey Data Release: The team discussed the forward plan for DR4 following *Detection table partitioning. MAR noted that QC1 should be closed out early next week as SJW has promised the LAS eyeball checks by Monday; as RSC noted, the partitioning should finish early next week. Quality error-bit flagging and difference imaging for the GPS H2-K can take place before recalibration. Subsequently the mammoth runs of CU4 for recalibration can be instigated. Non-survey Data Release: MAR noted that flat-file access up to the end of Feb 08A is up-to-date for registered 08A non-surveys. Astrogrid deployment & Data Analysis services MSH noted that new AG server odin needed a bit of TLC earlier today, and this has cured some teething trouble with VOdesktop logins. All have tested their logins, and it all seems to be working fine. MSH also noted that the *Detection tables in UKIDSSDR1+ are now cone-searchable following user requests. LGR noted that his new list-driven image analysis facility needs some final testing, but is almost ready to go; MAR noted that a good chunk of storage needs to be mounted on the web server to cache all the SDSS images. MSH suggested some of the freshly cleared out space on khafre for this purpose. ACTION: MAR to get ITSG to mount a RAID filesystem on thoth to cache SDSS pixels. Miscellaneous: The East Tower posse are planning a BBQ this lunchtime to make the most of the good weather. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================