From nch@roe.ac.uk Fri Feb 3 16:05:17 2006 Date: Fri, 3 Feb 2006 14:11:21 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 3rd Feb 2006 Minutes of WFCAM Science Archive meeting: 3rd February 2006 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, ETWS, NJC, RSC, PMW, JB, RGM, JDT, MAR, AL Apologies: JPE, JMS DONM: 10am, Friday 10th February 2006, plate library Actions discharged: ------------------- ACTION: JB to ask ITSG (HME) about provision of archiving of the messages coming through wsa-support. Discharged; awaiting HME's return from leave; our understanding is that there is such an archive somewhere by default. Actions partly discharged but continuing: ----------------------------------------- ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Continues. Actions carried forward from 27/01/06 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES ACTION: ETWS to nudge U.Chicago over SDSS-DR3 again. - CONTINUES Specific points and new actions: -------------------------------- Project management: Nothing new this week, but see below for one item re: the putative UK VDFS review. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: PMW noted that there are, apparently, some VDFS UK review terms-of-reference flying around that we have not seen. ACTION: PMW to chase JMS/JPE for VDFS UK review TOR NCH and JB noted the confusion over who was doing what stacked product for the DXS. Quoting from the minutes of the WFAU meeting of 20/1/06: "DXS/UDS stacks (CU13/14) : Initial work in parallel starting Tues next week; schedule a little unclear at present (will be clarified with survey heads and NJC mid-week next week) This is progressing well, with both DXS and UDS representatives having come to WFAU earlier this week to thrash out the plan of work. The idea is to have QC1 finished for both of these at the beginning of next week to enable stacks to be produced; in the case of UDS, Swarp is being used to create the final mosaiced product, while CASU toolkit codes are being used for the DXS multiframe stacks (UDS wide dithering strategy and requirements for source extraction results in the requirement for a single mosaiced product to be used as opposed to individual detector stacks). In both cases, the CASU source extractor and subsequent software (classification and astrometry) will be used to create a catalogue for archive ingest." Hence we don't really understand the confusion; but just to make it clear: WFAU have made DXS stacks from the intermediate pipeline products already delivered, and these have undergone the standard source extraction etc. using the CASU toolkit codes and will be included in the EDR release. BUT see Survey Release below for related issues that have arisen ... Networking: Nothing new this week. WSA Operations: JB reported normal operations have been resumed (backups and database curation) after a day spent sorting out the load server ahmose. NCH notes (not in the meeting) that wsa-support seems to be up and running, with one request for help already discharged (albeit forwarded to wsa-support from being emailed directly to NCH). Hardware: NCH noted that as regards the spec of the thrid catalogue server, we should make sure we factor in enough storage space to host a complete SSA (4TB) so that this can be moved away from the WSA public server. JB asked PMW about the status of the new file store server djedefre; PMW will chase up Eclipse Computing once the dust has settled over the EDR release. Software: RSC reported: "I've spent the past week putting the finishing touches on the new Cu16 script, which should now be ready for a real run." ETWS reported: "Updated ingestion software, tested possibility of ingesting single extension fits files with the UDS mosaics, and fixed minor problems with the parser. Investigated the problems with AST and the WCS variables for different projections, which can be set by the user and which override sensible presets. Ingested the UDS stacks and the DXS mosaic files into the database. Installed the plotting software biggles and pygplot." NCH has been sanity-checking the EDR release products (finding only one major issue - see below) and also preparing for release DB creation, and has found time to start writing the WSA paper... Survey Data Release: Shallow surveys plus the DXS were ready to go on Mon 30th; some issues with UDS tangent plane astrometry need to be sorted out and have been so that everything *was* ready to go on 2 Feb. However a problem has been identified in the CASU stacking/cataloguing procedure. As we understand it (but we're not entirely sure) apparently the philosophy of using integer pixel values (BITPIX=+32) is causing some problems in coaveraged stacks when the noise value approaches the digital quantisation of 1 ADU; furthermore the sky noise estimation procedure in the catalogue software (and possibly elsewhere) has a safety-valve feature of being clipped at 2.5 units (presumably to prevent screwy shallow frames from being thresholded at stupidly low values and producing zillions of crud catalogue detections). This second bug was a major issue in the DXS J stacks resulting in them being the best part of 1mag less deep than they should have been, and we had to try to fix CASU extractor (probably in a ham-fisted way) to get around it and reingest the catalogues (the same problem also affected some of the K stacks and is also an issue for the production of intermediate stack products in the pipeline that are delivered to WFAU). The upshot of all of this is that WFAU cannot be sure that the stack and catalogue products for the DXS are trustworthy; JRL has been unavailable for comment all week and MJI is away (but is replying to email, thankfully). The problem does not affect the UDS since the stacked mosaic was produced elsewhere by coadding, and the sky noise level is well above the 1 ADU digital quantisation limit. NCH expressed the opinion that despite the problem in the DXS, we really do not have any more time to investigate further (or to redo what has already had to be redone) and will have to proceed with release DB preparation today. AL further suggested that a list of "known issues" should be included as part of the release documentation (web pages, papers etc.) NCH asked ETWS and MAR if all was ready with the enhanced web pages for the EDR; all functionality will be tested early next week when the prepared DBs have been installed on the public server prior to switching them on to the main URLs. Non-Survey Data Release: No news this week. Astrogrid deployment: Nothing new this week. Miscellaneous: Nothing else to report this week.