From nch@roe.ac.uk Sat Mar 17 10:25:50 2007 Date: Fri, 16 Mar 2007 15:52:59 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 16/3/07 Minutes of WFAU VDFS Science Archive meeting: 16th March 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, MAR, JB, RGM Apologies: JMS, JDT, BCW, AL, JPE, MSH DONM: 10am, Friday 30th March 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JPE to contact Tim Naylor concerning scheduling a VDUC meeting. Discharged; no response as yet. ACTION: ETWS to finish of CU5 by driving it from the relevant merge log for difference image pairs. Discharged; testing in progress Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; BestDR2 is still being shrunk in preparation for moving to Thutmose while 2mass is a little too busy to take offline at the moment; will wait for a week or so until the clamour over DR2 dies down a bit before transfering it to Thutmose. ETWS discovered that the empty tables we have in BestDR5 are empty in the SDSS publically accessible version too. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; NCH, RSC and MAR discussed this in the week and because of some subtle complexities it will be put on hold until after DR2. RSC has made some code changes to make the curation applications robust against this very minor schema change when it does happen. Actions carried forward from 09/03/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. Continues; some contact with JAC concerning likely timescale for fix-up of the late-Oct/Nov/early-Dec data. Specific points and new actions: -------------------------------- Project management: NCH and PMW reported back from last Friday's VDMT meeting. Some small concern was expressed over progress, but PMW pointed out that effort expended on scaling operations to Vista data volumes could be reasonably put down as progress. WFCAM & VISTA updates: NCH noted that AA has sent a heads-up email over the four WFCAM campaign proposals endorsed by the UKIRT Board at their recent meeting: - transit survey (which is basically DXS-alike fields repeated ad nauseam); we've already seen this under PATT. - GRB overrides (again, already seen this and it's straightforward blank-field jitters) - narrow-band high-z galaxy survey (basically the same as U/06B/18 - S(1), nbJ and a yet-to-be-installed nbH filter), in the DXS fields. - Coma (galaxy) Cluster shallowish K band survey. May have a non-standard jitter size requirement but apart from that, don't expect there to be too much odd. JB raised the question as to what project IDs these would be coming through with; NCH noted that whatever these programmes are, we can assign the frames to an old existing programme (or conversely reassign old frames to new programmes) using the ProgrammeFrame feature of the data model, as necessary. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB reported: "Edinburgh hardware for the UKLight Edinburgh end has finally been given the go ahead to be installed with the UKLight UKERNA hardware going in on Friday. Once these are in place (should be by the end of Friday) then the link will be provisioned. Our final hop will be sorted out once Duncan can get up here to put the splitter in place with Sam tending the other end, this was intended for this week but EUCS is currently being reoganised. CU1 has been run and we have five days of 06B data here (download speed was 3.04MB/s). An IP address is being sorted out for the NAS box and I have discussed with Horst how we are to setup the UKLight connection to our machines." Note added in proof: apparently DHL have lost a critical piece of hardware that was to be installed in Leeds to enable the link. And so continues the saga... WSA Operations: JB reported: "CU7 for the GPS continues, it has now started reseaming the data. CU1 completed for five days of 06B data and we have started testing the data files and our code on it. Writing MFID's worked fine but building the ingest files (CU3) and ingesting those files has run across some problems, we have fixed them all now (thank you Eckhard and Nigel). We do however have two files that contain NaN values that I intend to send back to CASU for looking at. Tests continue and hopefully we can cleanly ingest them some time in the next couple of working days. System backups continue as normal." NCH notes (after the meeting!) that we should communicate with UKIDSS as soon as these files and JPEGs are ingested so that the end users can check over the data for any nasties also. ACTION: ETWS to email SJW as soon as some 06B data are available for perusal in the WSA. Hardware: JB noted: "A disk failed on Ahmose, we rebuilt it and then Eclipse replaced the spare - straight forward as far as we can tell. The NAS has had it's firmware updated and is now waiting on an IP address." ACTION: JB to set up the NAS as soon as ITSG have provided an IP address JB raised the subject of a new load server to reduce pressure on the venerable ahmose and also provide extra grunt well in time for Vista survey operations. The idea is that the older machine would be kept as a development test bed and fail-over for emergencies. This was unanimously supported by the team. ACTION: PMW to get a quote for a new load server (hatshepsut?!) PMW also noted that Eclipse have come back to us with info concerning cost and integration of a new LTO-3 drive into the existing Overland library unit; an order has now been placed. Software: RSC reported: "Various patches to CU7 to make it more robust and faster (trac ticket 38). Slight redesign to CU16 to make it more consistent with the new way we do things for CU19. Patched exmeta to automatically fill the versionNum attribute (trac ticket 2), and moved my wfcamsrc test scripts into the in src/testers CVS directory so that it is available to all developers (instructions are on the TWiki at SoftwareTesting). Moved the db-lock file from user's home directories to "/disk01/wsa/sys", so that it's visible on all curation servers and made it only lock out runs of the same CU ID on the same database (trac ticket 19). Finally, a whole bunch of DB and Schema API enhancements, mostly to enable automatic creation of views and stored functions for CU19 (trac ticket 25)." ETWS reported: "Worked mainly on re-factoring the difference image creation (CU5). Finished creating jpegs for the 5 days of 06B data. Fixed minor problems and improved the logging in the ingesting software." JB noted: "I've dug out and updated the CuPerformance documentation on the twiki and help find and describe bugs/new features in CU3, CU7 and CU19 - others did the actual fixing though." MAR reported: "Tidied up stored procs and functions scripts by moving survey only stuff to WSA_UKIDSS_StoredProcsAndFuncs.sql and master HTML procs into WSA_ConfigureServer.sql" MAR noted the user request for writing 64-bit INTs into FITS tables using the new standard. All code is in place, but tests show that some client side applications (e.g. GAIA) do not support the latest standard (although topcat, which we understand is heavily used by WSA users, does indeed support the format). RGM suggested contacting Mark Taylor prior to going live with the new 64-bit INT support just in case there may be other unforeseen issues. Survey Data Release: GPS is progressing slowly towards completion; sources are now merged and ingested and CU7 is now reseaming the survey. NCH reported that RGMcM in Cambridge had spotted a bug in the LAS source merging, and that the source of the problem (which affects about 5% of LAS frame sets, i.e. those where retiling between YJ and HK observations was done) has been traced to some experimental code features that were tried prior to DR1 propagating through to DR2. This has been fixed, but the question now arises as to whether we now take the opportunity to fix the LAS in the replacement release that is being prepared now including the GPS. The operations team expressed willingness to do this if time permits once GPS is done, noting that it will a few days of CPU to re-sourcemerge and cross-match the DR2 LAS. Non-survey Data Release: Nothing to report this week. Astrogrid deployment: Nothing new to report this week. Miscellaneous: JB noted: "I met with MIS (now being reorganised to be part of IS alogn with EUCS and renamed into three main groups: NSD (Network Support Division) who we are sorting UKLight out with; SSD (Systems Support Division) who deal with non-Network hardware; and ASD (Applications Support Division) who deal with the software (and MyED)) about MyED and have gotten them to send me details of how the system works and what sort of integration with MyED is possible (lots of levels). This should be relatively straight forward to sort out and will consume almost no time on our end. Actually a link on their end pointing to http://www.roe.ac.uk/ifa/wfau/ is probably sufficient (NB this page needs the WSA data release updating in the main text!)"