From nch@roe.ac.uk Fri Jan 11 16:37:07 2008 Date: Fri, 11 Jan 2008 16:27:42 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 11/1/08 Minutes of WFAU VDFS Science Archive meeting: 11th January 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, ETWS, LGR, NJC, PMW, RGM, MSH Apologies: BCW, JPE, AL, JMS DONM: 10am, Friday 25th January 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MSH to bring hatshepsut onto private network, and to set up a login for MAR MAR has the admin pwd so no worries; MAR will have a poke around to see if there are any server settings that need to be tweaked to allow client connections (as an aside, the full-blown SSA is all but finished and ready for testing through JDBC via the original interface pages; NCH needs to liaise with JNTD over backups of this rather large database). ACTION: JB to set dfg_1 shrinking in the WSA this weekend. Discharged by MAR and NCH. ACTION: NCH to buzz MSH about getting the SSS transfers instigated before Xmas. Discharged; pressure is off somewhat because the server has been temporarily relocated to allow time to slurp the legacy data off and migrate onto newer hardware (in fact, will probably do it by LTO-3 tape so as to secure a backup before the original RAID server is finally retired). Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. Actions carried forward from 14/12/07 meeting: ---------------------------------------------- ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) Continues; team agreed to postpone until after DR3 now; also awaits instance of SQL 2005 on the batch catalogue server which in turn awaits full SSA tests against user interface JDBC applications... we're getting there... Specific points and new actions: -------------------------------- Project management: NCH noted that the next VDMT is now scheduled for 21st Jan at 3pm. PMW brought along the progress charts Q4'07 for completion during the meeting, and suggested he and NCH meet offline after this meeting to complete the forward plan for Q1'08 in the light of any altered priorities following interaction with the community at the UKIDSS ESO Workshop (see below). PMW noted that the job advert for JB's replacement has been live since before Xmas, and encouraged all to point any potentially interested candidates to this. Several people (already known to us) have expressed an interest... WFCAM & VISTA updates: NCH noted that WFCAM/UKIDSS observing is being finalised for the upcoming block, but that volunteers are still needed for Feb 15-19 and 20-24 if there were any team members interested. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 3rd January 2008. Prompted by various reprocessing issues, e.g. as described in the latest CASU minutes, PMW asked what the official policy was on retransfer/ingest of small numbers of manually fixed processed frames (i.e. very small amounts of data that do not warrant wholesale reprocessed new versions for an entire night). ETWS explained that he has been working on a procedure to handle these a little more elegantly at the WFAU end given recent experience in ad hoc manual fixes required. Networking: Following JB's departure, NCH got PSB a login account on djedefre and following the latter having a poke around, the transfer speed bottleneck at WFAU is looking increasingly as if it is associated with the NFS rat's nest on the WFAU LAN; further investigations and tune-ups will be attempted on advice from PSB and help from MSH, ETWS and local IT Support. WSA Operations: NCH noted that since JB's departure, everyone is mucking in to look after routine ops. Before Xmas, preparations for GPS release went very well with source merging/seaming finishing during the ESO workshop at which point NCH & ETWS launched CU16 for crossmatch, MAR shrunk the database and then then NCH backed up the WSA on return from Munich and set the release DB creating prior to disappearing for Xmas hols. Unfortunately, an unexplained glitch in copying out the 0.5TB GPS detection table resulted in a crash on the 21st, and NCH's efforts to diagnose the problem have not found the cause. In any case, the full DR3 GPS tables are being loaded one by one manually, then CU19 will be relaunched in append mode to complete the job. On Monday this week, a disk failure on one of the NAS boxes followed by a somewhat bum steer from the maintenance contractor resulted in the loss of a logical volume from that storage unit. The standard VDFS disaster recover plan (i.e. copying from the online dataset at CASU) was immediately instigated for the 23 nights affected (mainly 05A/B) along with recreation of the lost JPEGs; hopefully everything will be back to normal before any of the hordes of world-access DR1 users notice anything is amiss (thanks to ETWS for looking after this). Some discussion ensued concerning general backup policy, and the results of this were: ACTION: MAR to look after Windows system/DB backups prior to the new operator arrving It was decided also to backup certain flat-file datasets (e.g. DB-driven image products; DB outgest files for crossmatch) on LTO-3 once a stand-alone drive is brought over from the old Alpha RAID server currently serving the SuperCOSMOS Sky Survey data. Hardware RSC noted that IT Support want to know how they should apply Linux updates to the archive servers, especially as regards making sure all relevant software, on 32 and 64 bit platforms as appropriate, works OK. ACTION: MSC in consultation with RSC to liaise with IT Support over linux upgrades. PMW noted that the new NAS box is finally on order, following interminable arguments with UoE over tendering. NCH asked as to the status of the order for the new web server; again, some tendering jiggery-pokery is complicating the issue. ACTION: PMW, MAR and MSH to liaise over progressing the new web server procurement. Software: MAR reported: "Provided verions of the GPS mosaic for publicity purposes (AAS and Imiloa Centre's planetarium). Assisted Mark Taylor (Astrogrid) in looking at HTM indexing (2MASS and SDSS) and speeding up small cone searches for several thousand coordinates with the aim of incorporating it into the DSA." ETWS has been looking after operations (particularly restoring the data files lost earlier this week); NJC has been working on some non-survey mosaic stacks for a user; and RSC has been working on merging the various SVN branches left over from last year. NJC has been working hard on an updated generalised synoptic data model, and sent around some info for everyones perusal. ACTION: ALL to look at the synoptic data model from NJC and provide comments. NJC raised a few questions concerning VISTA tiles that need a bit of thought at the archive end: ACTION: NJC, NCH, MAR, ETWS and RSC to meet on Wed 23rd Jan at 2pm in the Vista Hut to brainstorm database aspects of Vista tiling. The team then went through the unofficial wish-list that came out of the UKIDSS ESO Workshop at Garching before Xmas (combination of stuff from discussion with individual users, and from the workshop discussion at the end of the meeting, not in any particular order, and not including pipeline things, and not meant to be complete): List-driven photometry in (SDSS) optical images at IR source positions: The team felt this would be a good thing to implement, starting by scoping the problem and perhaps implementing a VO-style solution using the Astrogrid infrastructure - ACTION: LGR to coordinate a kick-off meeting between himself, RGM, NCH and Kona to discuss the way forward on list-driven IR/optical pixel analysis Automated feedback of user-determined junk flagging e.g. perhaps by defining a new objects (userErrBits and/or new view definitions?) for specified selections; the team felt that examining any community pages (e.g. the Astrogrid UKIDSS Wiki zone) may inform this, but that a more formal web forum may be needed to gather the necessary input ACTION: NCH to suggest to AL/SJW that formal feedback on user priorities (perhaps via a web forum) might be helpful in planning WSA enhancements like user-defined junk flagging. Colour image production: SJW and AA in particular are keen on this; NCH noted that, for PR purposes for both WFCAM and Vista, this may be worth investing some time into; SJW is offering to interact on the details of any implementation. ACTION: MAR to take forward the possibility of archive and/or on-the-fly colour image production. Documentation enhancements: apart from the inevitable (very) odd whinge concerning complexity in the online documentation, the users seem happy and able to get the information they need, but a couple of requests: ACTION: MAR to include a link from WSA publications to UKIDSS science publications list maintained by SJW ACTION: MAR to incorporate basic info on using flat files Schlegel extinction map derived attribute in the GPS: the team feel that this is a complete can-of-worms and likely to confuse most users, so: ACTION: NCH to pass the buck to AL/SJW/PWL concerning an extinction attribute in the GPS: "we'll do what you tell us, but if users screw up, on your heads be it" Additional cross-matches with SDSS: for LAS, a crossmatch with the unexpurgated PhotoObjAll; for GCS, crossmatch with PhotoObj which covers Praesepe and ComaBer. (no action, because already done by RSC and NCH; crossmatch tables will appear in the full DR3 release). Finally, RSC noted that IT Support would like to retire the old CVS server, and kindly volunteered to coordinate export of the remaining projects (SSA, VDFS docs etc) into SVN. ACTION: RSC to coordinate migration of the residual CVS-tracked projects into SVN. Survey Data Release: MAR noted making the necessary updates to webpages and interface to allow world access to DR1 to coincide with an AAS announcement & press release (Wednesday 9th). Non-survey Data Release: MAR and ETWS generated and released two non-survey databases wanted for a PhD student. ETWS has fixed a small snafu concerning default values for quality bits in non-surveys; RSC asked about the approach to keeping non-survey schema scripts in line with changes to the template. ACTION: ETWS to examine CU21 to see if it can be tweaked to propagate template schema changes into pre-existing non-survey static SQL scripts Astrogrid deployment: MSH noted that the world release of UKIDSS DR1 has been VO-enabled by setting up a new DSA (with new features) rather than switching off the secured access to the existing old one. Miscellaneous: A free prize draw for a bottle of scotch that mysteriously appeared just before Xmas was held at the end of the meeting. RSC won.