From nch@roe.ac.uk Fri Jul 15 12:50:02 2005 Date: Fri, 15 Jul 2005 11:57:04 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes: 15th July 2005 Minutes of WFCAM Science Archive meeting: 15th July 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, PMW, MAR, ETWS, NJC Apologies: MCH, JPE, RGM, JDT, AL, JMS DONM: 10am, Friday 22nd July 2005, plate library Actions discharged: ------------------- ACTION: NCH, PMW, RGM & MAR to have a shortlisting meeting for the archive operator post sometime during 11-13 July. Discharged; 6 candidates selected for interview on 4th Aug ACTION: NCH, ETWS & PMW to discuss offline the requirements for expansion of RAID storage for the archive servers. Discharged at this meeting - see Hardware below. Actions partly discharged but continuing: ----------------------------------------- ACTION: PMW to initiate removal of old hardware from C1 by communicating with all concerned parties (JNTD, HMG, RL, ...) Meeting with HMG took place and agreement reached on relocation of cabinets in SuperCOSMOS computer room. PMW hopes to instigate storage cab removal next week; Eclipse will then be called in to relocate the older RAID server & arrays form C1 back into SuperCOS. - continues. Actions carried forward from 7/07/05 meeting: ---------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: NCH noted the VDMT meeting of last Friday. Substantive issues raised were: The UK VDFS review: the idea now is to revisit the top-level functionality breakdown of VDFS versions 3 to 5 (as defined in the VEGA e-Science2 bid) and then present the revised plan to VDUC to obtain a steer as to the scope of the review; also the VDMT's opinion on supplying tape copies of bulk data to users was not thought to be appropriate (see last week's minutes under Data Release). PMW noted the recruitment update: Operator interviews of 6 candidates are to be held on the 4th August. Also noted that the 2005 Q3 plan needs to be assembled in the light of the previous requirements from VDMT. ACTION: PMW & NCH to meet sometime next week to assemble the Q3 plan. VDUC is tentatively scheduled for early September, but watch this space... WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 12/7/05. PMW noted the VST Surveys Meeting feedback from MJI, and that we have a VDMT action to consider the resource requirements for servicing any science archiving requirements for those public surveys that may request such work. NCH noted the Cambridge University network monitoring service's "apoplexy" at the data transfer, and that so far, we have had no comments from the Edinburgh University end (maybe they haven't noticed, or don't care...) Networking: Note added after meeting: Portsmouth Uni are transfering a copy of the SDSS DR1 catalogue database from WFAU to test out their fledgling SDSS Catalogue Access Service mirror. NCH is staging backup files for them to do this. Hardware: Hardware situation is back to normal: environmental control seems to be stable again, and we have an action plan to relieve the power consumption/dissipation congestion in C1 (see continuing actions above). NCH noted that in addition to the cosaxp6 RAID server and disks, we should make sure that cosaxp3 is shifted out. JDT also offered to move the grendel rack out as well, contingent on agreement from IT Support. NCH asked MAR to oversee this. ACTION: MAR to liaise with MCH/JDT/ITsupport concerning shifting of the grendels out of C1. NCH noted the need for expansion of the pixel store for WFCAM data. Meeting agreed to order one new storage block (24-bay, 400 GB SATA file server with 2 12-port 3Ware RAID controllers yielding ~8TB of storage) plus a new rackmount cabinet to ensure sufficient lebensraum in C1 when space is vacated by older kit. ACTION: PMW to order new file node "khufu": as per sneferu, but 24 bay rather than 16. NCH noted a discussion with JNTD concerning the future direction of mass storage hardware in C1 in the medium term (i.e. before any possible farm-out to central UoE providers, i.e. SAN). Networked Attached Storage may be a lower maintenaince solution, and should be investigated. He also suggested asking around for details of other people's experience in this area. ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. Software: NCH forgot to heap praise and thanks on ETWS during the meeting for spending considerable out-of-hours time last weekend and during evenings this week to catch up on ingesting image and catalogue data following the previous week's hardware problems... the beers are on me. CU4 (catalogue ingest) has required more work this week, with some small problems delaying the last of the SV1 ingest. ETWS continues to investigate. CU8 (global photometric recalibration) development continues; NCH suggested that NJC look into the data model (schemas) for recalibration data handling with a view to examining the functionality required for recalibration applications. ACTION: NJC to examine photometric recalibration data schema and SQL required to service CU8 applications. NCH noted that in the course of revising curation procedures for SV1, he has almost completed CU18 (archive table indexing) automation for routine archive operations. MAR has been working on the user interface in readiness for SV1 release. Data Release: SV1 release is still pending, following the recent hardware problems (which are now all fixed). SV1 ingest is nearly complete, but there is one final software problem, and also some ingest burps/regurgitation caused by half a dozen or so wierd catalogue FITS files ACTION: ETWS to communicate details of the handful of FITS catalogues that have caused archive indegestion. NCH cautiously predicted SV1 release sometime next week. SSA: Nothing new this week Astrogrid deployment: NCH noted AL's message concerning WSA publication through the Astrogrid portal. With reference to AL's options, viz: "(i) The WSA can be published to the AstroGrid Registry using DSA pretty much now, and formally visible as a resource. Of course only the parts that have become public will be useable through the AstroGrid portal, and this will be negligible for some time. But we should still do this." Fine by us; we will wait for MCH/JDT to ask for the relevant schema metadata (which we are set up to automatically generate for the WSA using ETWS's wonderful schema parsers). "(ii) We can set up a workbench mini-app that basically provides a link to the WSA web page. This can also provide text that explains that users can browse the system, but can only access the data if they have registered with WSA. This might sounds like a cheap trick, but it genuinely gives access to WSA through AstroGrid." OK by us - up to AG how to implement this. "(iii) The next step that makes it REALLY useful is the ability to save results from WSA queries to MySpace - image files, VOTables, etc. Then you can look at results with TopCat etc. Ideally what we want is a button on the WSA page that says "save results to MySpace". We can implement this without having to solve the authorisation problem. So at this stage, from the AstroGrid portal you (a) fire up workbench (b) select "connect to WSA" which opens WSA page in your web browser (c) login with your WSA username as necessary (d) run job using WSA query builder (e) save result to MySpace (f) click back to workbench, run Aladin or TopCat, play with results. Sounds pretty good to me. " JDT and MAR have been discussing and working on this: JDT wrote "Executive Summary: We can do it. Reckon it will take about a week. Tech details: The proposed"Save To MySpace" Button would be linked to the URL for a small java application delivered by webstart. This application would be passed the URL of the saved results from the query. It would then ensure that the AstroGrid Client Runtime was up and running on the client's machine. If not, it would start it, downloading the software if necessary. It will then use the ACR to transfer the data, by passing it the URL. User experience: From the user's point of view, they will click on the button and if they have done this before, the upload should just happen. If they have never done it before, there will be a very short delay while the application downloads (it should be very very small). If the user is already logged into the ACR, then the next thing that will happen will be that a VOSpace explorer dialog will pop up inviting them to choose where to put the data. If the user has not already logged into the ACR, then they will be first prompted to log in. Again, if they have never used the ACR (or the ACR has released a new version), then there will be a short delay while the code downloads." ACTION: JDT and MAR to continue sorting out AG access option functionality for saving WSA results to "MySpace". "(iv) By about October we should have preliminary agreement on IVOA authorisation. AstroGrid can start to implement this, but somehow we need to match on to the existing WSA registration scheme. Shouldn't be HARD, but isn't TRIVIAL, so maybe we might get this working by about spring 2006. When this works, we add two things to the stuff above : (a) you can submit a query using the the AstroGrid query builder as well as the WSA query builder, and (b) you can include such a job within a workflow. The second thing is the really important bit of course." OK (but we won't hold our breath...!). Miscellaneous: NCH noted the flurry of emails from UKIDSS PI/CSS concerning calibration issues, and the replies from STH. NCH asked NJC to keep a watching brief on this and attend any relevant meetings as WSA photometric recalibration representative. ACTION: NJC to liaise with STH on any WFCAM calibration meeting. PMW noted proposed accomodation for new software developer Ross Collins: the plan is to put him in P9 (JDT's office) once JDT has relocated. In the meantime, PMW asked MAR if Ross can share with him for a week or so.