From nch@roe.ac.uk Fri Jun 3 12:05:04 2005 Date: Fri, 3 Jun 2005 11:56:54 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly project meeting, 3rd June 2005 Minutes of WFCAM Science Archive meeting: 3rd June 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, NJC, ETWS, PMW Apologies: MCH, AL, RGM, JPE, JDT, MAR, JMS DONM: 10am, Friday 10th June 2005, plate library Actions discharged: ------------------- ACTION: ETWS to send MJI the details of new FITS keywords for archive purposes Discharged; these have been kindly incorporated into the standard pipeline MEF writing, and are included in the first batch of SV data. ACTION: ALL to think about simple modifications to the top-level web site to help totally novice users get to the data they want. Discharged; the suggestions are to have a "First time users start here" page, linking from the left-hand navigation bar; and also to have active pop-ups that give one-sentence summaries when hovering over a given navigation point. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 27/05/05 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: NCH noted the upcoming VDMT telecon this afternoon at 2pm, and thanked PMW for doing the usual report writing. PMW noted that the deadline for the readvertised developer position has now closed, and that we have 49 applicants. A quick scan indicates a good bunch of potentially employable folks. The plan now is to individually scan the list (PMW, NCH, JMS, RGM) and then collate to shortlist by the end of next week so that interview invitations can be issued asap. RGM and PMW noted that the deadline for the operator post has been extended (mainly because the advert didn't appear until a week or so before the old deadline...) WFCAM update: Survey data being taken apace; see the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 03/06/05. Networking: NCH noted that Jan at JHU has gone a little quiet over the transfer of SDSS DR3 (perhaps Chicago are having problems transfering within the US). ACTION: NCH to gently nudge Jan to see what the hold-up is over DR3. CASU have made available the first three nights of UKIDSS SV data for transfer. No transfers have been done owing to more pressing matters (see below). Hardware: All archive hardware are functioning normally. Software: The major bad news this week is that ETWS and NCH have identified a big problem with the catalogue ingest code (originally written by IAB). This C++ code is extraordinarily memory-hungry (and consequently slow), to the point that it cannot reliably handle more than of order ten catalogue files at a time before crashing. This problem has not arisen until now because the amount of data to be ingested has been small up to this point. After much investigation and trouble-shooting, it now appears that there is a fundamental problem with the use of C++ new/delete and underlying unintelligent malloc procedures that are unable to reuse increasingly fragmented memory for large requested allocations during a single process. Advice is being sought from Andy Vick and John Lightfoot on solutions, and we seem to be on the right track to solving the problem. But until this is fixed to our satisfaction, catalogue ingest has stalled. This is delaying the release of the reprocessed commissioning data, but hopefully will not hit the schedule for SV releases (but at this point there are no guarantees). NJC reported: "I have been working on the photometric recalibration code for CU8. I have rewritten the non-linear correction code from Numerical Recipes C++ to the C code which the library has a copy of, so we do not have any legal issues. I have thoroughly tested it and added it into the CVS. I am currently working on the python wrapper which will iterate through the linear and non-linear parts of the code, until a good solution is found and will test the residuals. I am paying particular attention to the non-ideal nature of the data that we will gather, making sure the code will run when there are missing calibration data." Finally, NCH reported setting up some backup policies and infrastructure on the archive load server. Data Release: NCH again noted the delay in releasing the reprocessed phase-I commissioning data, due to the ingest code problem identified above. Apologies all round, but if we can't offer software developers permanent contracts, then this kind of thing will inevitably happen from time to time... SSA: Nothing new this week Astrogrid deployment: No news this week. Miscellaneous: NJC noted that he is still awaiting feedback on his global photometric calibration document.