From nch@roe.ac.uk Sun Nov 20 18:55:15 2005 Date: Fri, 18 Nov 2005 13:59:53 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly project meeting minutes: 18 Nov 2005 Minutes of WFCAM Science Archive meeting: 18th November 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, MAR, NJC, AL, JB Apologies: MCH, RGM, JPE, JMS, JDT DONM: 10am, Friday 25th November 2005, plate library Actions discharged: ------------------- ACTION: MAR to switch on the "save to MySpace" WSA Interface button. Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH and PMW to meet on Mon 31st to assemble the Q4'05 and Q1'06 VDFS work plans Continues; a further meeting is scheduled for next week (see Project Management below). ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Continues; MAR & JDT need to talk over some final details. Actions carried forward from 4/11/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH reported back from the IR survey workshop and UKIDSS Consortium meetings held last week. The main topic of interest is the wish-list of WSA features feeding back from the user community (not in any particular order): 1/ Add in the GLIMPSE external catalogue and create cross-neighbour joins with the GPS 2/ An http post/get SQL service that accepts free-form SQL and streams results sets back to the client 3/ Quality control: flagging of residual cross-talk images, 128 pix either side of v. bright / saturated objects in single-passband catalogues 4/ Application of 2d photometric correction maps of systematic calibration errors (of order a few %) measured from all 2MASS standard residuals, averaged as a function of position on frames 5/ Link to CASU survey progress information from WSA pages 6/ World release to take place 18 months after ESO release, and not 18 months after observation dates (and similarly for non-survey data as well, presumably, but with 12 month proprietary period) 7/ New data model required for poor-seeing time programme (multi-epoch photometry in search of planetary transists) NCH suggests we look at this wish-list in the context of the Q4 plans, and revise if necessary. PMW suggested another planning meeting for next week to discuss these (and other) issues ACTION: NCH & PMW to have a planning meeting next week, Tues AM. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 18/11/05 AM. Networking: NCH noted the UKIDSS GPS would like the GLIMPSE catalogue ingested and joined in the WSA. ACTION: JB to look at the GLIMPSE info and begin SQL implementation with help from ETWS and NCH. NCH reported the continuing saga of the corrupted USNO-B. Communications with JHU indicate that their copy of our SQL database is identically corrupted. The WSA/SSA copy remains offline until further notice, but the VO team noted that it was being used for AG workbench tests so the DB will be reattached for use only by the AG PAL access point. In the meantime, JB is looking into the backups of the flat files obtained from LEDAS and created by C++ processing applications to see if we can identify the source of the whacky data. ACTION: JB to continue to investigate the USNO-B problem by restoring flat-file backups. NCH noted developments down the hill at KB where the new interface to SuperJANET equipment has been installed to support UKLight connectivity; George Howat has additionally been in touch with his Cmabridge counterpart Chris Cheney on the CASU/WFAU potential usage. PMW suggested we make some decisions as to who is going to do what at the WFAU end at next week's meeting. Finally, NCH noted the new CASU WFCAM catalogue document being made available, and ETWS will examine this for consistency with our own online documentation, and also the ingest procedures. WSA Operations: JB reported: "Amenhotep has had all it's databases restored and is now back up and running as it was (excepting USNOB and BestDR1). Half of the WSA tape collection has now been catalogued with the rest hopefully being finished some time Monday. Djoser had a disk fail for the second time in a month and so it has been replaced. Ahmose has had some problems network and SQL problems, the SQL problems may now be being solved, the network problems are potentially symptoms of a rogue system process (though not spyware as both Ahmose and Amenhotep have been checked for this). Ahmose has had two of five disks defragmented now with a view to improving performance. UKIDSSR2 has been attached on Amenhotep. Tape backups have now been made of: last week's system disks, this week's system disks, UKIDSSR1, UKIDSSR2, the WSA as of the R2 release, log files up to the R2 release and the tarballs that needed to be installed on Khufu after Debian (Sarge) had been installed." JB additionally noted that CU3 was now running again (to ingest the remaining 05A data). Hardware: ETWS reported finishing the installation of third party software on khufu. NCH asked the team's opinion on when the UPS should be installed on the khufu rack. Discussions at the last VO team meeting suggested that the end of November would be a good time. ACTION: NCH to schedule installation of the khufu rack UPS at the end of Nov. PMW noted that expansion estimates for power consumption requirements in C1 were now required by premises; NCH suggested that this be done offline at the planning meeting next week. Finally, ETWS and RSC noted that djoser would need an operating system upgrade before it is much older, as the current Debian installation is no longer supported, and emphasised the need for all development work to migrate to khufu at the earliest convenience. Software: ETWS reported making the documentation for the new release, refactoring of the cu2 wrapper, and work on the non survey parsing scripts. RSC reported: "I've made an attempt to ensure the next data release will run more smoothly than the last! Fixed issues with the Cu7 code, preventing infinite loops from occurring and patched memory leaks. Made wfcamsrc more robust against schema errors / evolution, with more graceful error checking. I've investigated speeding up mkmerge, but unfortunately its relative slowness is a product of its overall design ethos. A complete redesign making it slightly less flexible will speed it up considerably, but will take a while to develop. I'm closing the book on wfcamsrc for the time being today, and from Monday I will be concentrating on the flashy new Cu16 script with go-faster stripes." NCH has been fire-fighting a performance problem with CU16 for the last week, finally finding a work-around to enable release. The suspicion is a hardware issue with the load-server rather than any fundamnetal scaling problem in the software. Otherwise, CUs 18 and 19 performed well for the latest release (see below). Finally, NCH suggested that CU13/14 should now be a high priority for NJC. Survey Data Release: NCH noted that the SV 1+2 release finally happened on 17th Nov PM. So far, > 200 queries have been logged from outside of the ROE. There has been a wide-ranging discussion as to release plans for survey data; at this stage the main point to note is that we are committed to releasing the 05A survey data by end Jan '06, and that discussions are currently underway as to whether to bolster this release with a subset of 05B data. AL asked what the cut-off point was, as far as WFAU were concerned, on when a decision had to be made. NCH tentativel suggested the first week of December. NCH noted that an interesting issue has arisen over making the continually ingesting archive DB available for limited queries (i.e. the image/catalogue metadata browser and flat-file access to basic pipeline products) to the outside world. This has been suggested as a good interim measure to get data to users in the period between ingestion and release of full-blown DB enhanced static DB products, by both CASU and JAC. A detailed discussion on the logistics threw up no show-stoppers, and while this goes against the previous policy, it was generally agreed that this was doable and would be a good thing provided sufficient security limits and user warnings were put in place. NCH suggested that we make a final decision on this next week, as part of the review of plans in the light of this and other goal-post moves, and the user community wish-list noted above. Finally, NCH noted that SJW had suggested that the world-readable DB should contain a small taster of nice UKIDSS SV data from the current release, and that he would provide details of a suitable small data subset in due course, after which the world-readable DB would be updated. Non-Survey Data Release: NCH suggested that we should, as top priority, now try to get some non-survey data out to PIs who have been waiting (patiently in most cases) to get at their data. ACTION: NCH, MAR & ETWS to meet this afternoon to review the state of readiness for release of non-survey DB products. As a further point, the team discussed the level of support for non-survey products, after noting the MAR had received enquiries as to the possibility of WFAU creating bespoke stacks for one PI. At this stage, no support can be provided for individual users, but in the fullness of time, interface functionality would provide this facility on-the-fly; AL noted that it is in fact possible to do this kind of thing now on the AG workbench. Astrogrid deployment: NCH noted that a communique had been received from Nic Walton concerning representation on the new Astrogrid Deployment Advisory Group (AG-DAG). MAR has kindly agreed to be the WFAU representative on this august body. Miscellaneous: Finally, NCH reported that there had been so far 3 responses to the main-journal WSA paper outline posted on the UKIDSS TWiki with the only substantive point (from SJW) being that this (or some other paper) should have a user-guide as well as the more technical design details.