From nch@roe.ac.uk Sat Feb 25 00:15:03 2006 Date: Fri, 24 Feb 2006 12:30:44 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 24th February 2006 Minutes of WFCAM Science Archive meeting: 24th February 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RSC, PMW, MAR, RGM Apologies: JPE, JMS, AL, JDT, NJC, JB DONM: 10am, Friday 3rd March 2006, plate library Actions discharged: ------------------- ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Discharged; apparently the entire AstroClientRuntime work bench GUI thing is being refactored, so this is no longer appropriate at this time. ACTION: RSC to review the Software Architecture Document (SAD) in the light of implementation differences and any new OO approach under development Discharged; the SAD is aptly named - a complete overhaul is required ACTION: RSC to implement binary catalogue ingest in CU4. Discharged; JB is testing the new hyper-efficient ingest of catalogue data. ACTION: NCH to contact AA at JAC to discuss possible input from JAC to help us time releases of non-survey data. Discharged; see Non-survey data release below; NCH will forward the response from JAC to JB. ACTION: RSC write an abstract for NAM, aiming for a talk with the emphasis being to encourage people to both make use of the WSA and more importantly highlight the power and simplicity of SQL searches for data mining etc. Discharged Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 10/02/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS to go thru the CUs one by one and note any issues as regards performance/scalability. - CONTINUES ACTION: PMW to start the ball rolling on getting quotes for a new storage server with an expanded spec over khufu/djedefre. - CONTINUES ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: The team completed the usual progress update chart for January (this has been delayed until now by the EDR release fire-fight). WFCAM update: Call for 06B proposals has gone out and WFCAM is scheduled to be on telescope from Nov '06 to Mar '07 (note the continuation into Semester 07A). Comments and issues arising from CASU fortnightly minutes: No new minutes as of 24/2/06 AM. Networking: DR3 has completed transfer from Chicago; WFCAM data transfers from CASU are continuing for processed 05B data. WSA Operations: PMW noted that the IfA WFAU pages need updating to reflect WSA developments. ACTION: MAR to update the IfA WFAU web pages. NCH noted that there are a few missing JPEGs from 05B Aug/Sep, ETWS noted that these are probably hangovers from one or two ingest glitches and will be picked up when the JPEGing process (CU2) is launched again next week. Hardware: ETWS noted that the latest file store server djedefre is all up and working. NCH noted that djoser/thoth are offline today to get OS kernel upgrades; electrical testing over this weekend requires the entire archive system to be offline on Saturday (hopefully back up again on Sunday). The team discussed the requirements for holding online SDSS catalogue databases. DR1 (0.5TB) was not used significantly, so is currently offline, but DR2 (0.9TB) is being used extensively by UKIDSS EDR queries, so really needs to stay online for the forseeable future, even if it is notionally superseded by DR3 (1.4ishTB). It would seem advantageous to be able to keep DR2 et seq. online regardless of any future release copies held (e.g. DR4 1.7TB). RGM noted that this would also be a nice service to offer to UK users via AstroGrid. While the team acknowledged that WFAU was never intended to be a general data-centre like CDS, in the case of SDSS releases an exception should be made due to the close relationship with the UKIDSS headline science goals. RGM expressed the thought that it may be possible to source some external monies (i.e. outwith VDFS/WFAU budgets) to purchase a specific catalogue server for SDSS databases, to ease the load on the WFCAM public catalogue server(s). ACTION: RGM to look into the possibility of external funds for an SDSS catalogue server. Finally, RGM pointed out that as regards scalability, VDFS documentation etc. it would be good to consider looking at SQL 2005 and it's advantages over the current enterprise edition of SQL 2000. ACTION: RGM to seek out MSDNAA SQL 2005 CDs. Software: NJC had reported earlier in the week: "I have had a look at the variation in extinction across the LAS. Using the Schlegel dust maps, I find the following values for min, max, 3-sigma clipped mean, 3-sig clipped sd for the E(B-V): min 0.0029, max 0.5881, mean 0.0269, sd 0.0131. I have also calculated A/E(B-V) values for the filters: Filter A/E(B-V) Z 1.587 Y 1.211 J 0.889 H 0.578 K 0.360 This converts the above numbers: Filter Max-Min mean sd Y 0.712 0.033 0.016 J 0.523 0.024 0.012 H 0.340 0.016 0.008 K 0.212 0.010 0.005 I have confirmed the numbers for the maximum using the NASA Extragalactic Database. These numbers only apply to areas where there were at least 3 measurements of the dust extinction and no strong sources that would affect the result. I would say that the sd gives the typical variation across the LAS. Since this <0.02 it is probably good enough for our purposes. However, the mean is often larger and gives a systematic offset in photometric redshift or SED fitting codes. The other worry is that there are some places with higher values and these may cause some discrepancies. I suggest that we do add in the extinction correction. It could be calculated per detector rather than per object, since the resolution of the maps is approximately (2.37 arc min)^2 per pixel." The team discussed the above, and agreed that we should aim to implement this for the next data release, by including an Eb-v attribute in the relevant Source tables (for ease of end-user querying), i.e. LAS, DXS and UDS. Recording Eb-v (as computed by Schlegel) allows any reddening law to be applied to correct WFCAM or any other photometry being accessed in a WSA query. ETWS reported: "Finished the new parsing script to handle multiple releases. Installed complete WSA software package (including gcc4, python 2.4) on djedefre. Upgrading general CVS system files to be compatible with the new version of CVS. There may be still some hiccups with notifying emails." NCH reported continuing work on the WSA write-up. Survey Data Release: NCH reported that 05B QC1 has been started, in collaboration with SJW. Non-Survey Data Release: NCH noted that he has had an email exchange with AA at JAC concerning notification of non-survey PI programme observation completion. NCH has forwarded this to JB, suggesting that we should take the initiative and regularly contact JAC to get an update on completetion of non-survey programmes. Astrogrid deployment: Nothing new this week. Miscellaneous: Nothing else this week.