Minutes of WFAU VDFS Science Archive meeting: 3rd August 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, MAR, NJC Apologies: JDT, BCW, JPE, JB, RGM, AL, MSH, JMS, LGR DONM: 10am, Friday 10th August 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. Expurgated. ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. Expurgated. ACTION: RSC, NJC & ETWS to meet on Monday 30th July at 11:30am for a SW architecture meeting. Discharged; meeting was most constructive. ACTION: NJC to contact the Nottingham UDS folks to find out their intentions and timescale for DR3. Discharged; UDS team schedule gels quite nicely with ours; Seb Oliver is visiting Edinburgh soon to discuss catalogue extraction details. ACTION: NJC to contact GPS survey head concerning apermag 2/6 attributes in the merged source schema. Discharged; GPS Survey Head has asked for aperMag1 in place of 6 in the DR3 et seq. schema. NJC has implemented the changed and checked into SVN. ACTION: NJC to liaise with his other half over possible dates for a celebratory evening meal. Discharged. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details (included in last week's minutes) Actions carried forward from 27/07/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: MAR to switch off proprietary status of UKIDSS EDR in the WSA interface on August 10th. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH and PMW reported back from last Friday's VDMT. No substantive issues; STH advised us that processing of early 06B was steaming ahead but there may be one or two data/processing issues (possibly sky-sub?) that need to be checked before the data are released as ready to copy. The team filled in progress charts for July. NCH noted that some development staff effort was now moving over to GAIA (where urgent progress is now required); the displaced effort will be covered by operational effort (which was to ramp up now due to the arrival of VISTA data - clearly not required at present). WFAU response to the JAC MoU awaits input from JPE (and possibly AL). WFCAM & VISTA updates: UKIDSS observations starting again for 07B; no VISTA news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 03/08/07 am. Networking: NCH and PMW noted a communication from PSB concerning network investigations. ACTION: PMW to speak to JB concerning CASU-WFAU network investigation. WSA Operations: MAR reported starting QC for semesters 06B and 07A in preparation for DR3. ETWS reported ingesting the last days of 07A which means that all available data of 06B/07A is now ingested. JB reported tidying directories and planning on moving disk01 over the next week (once it is confirmed that the disk03 move went well). Hardware: NCH raised the subject of available mass file storage, and noted that the priority must be to reserve sufficient storage for early 06B, and migrating old WFCAM (and what's more, the old SuperCOS) data must not be allowed to interfere with DR3 preparations. Software: NJC reported: "This week I have tested the mosaicing and extraction software on SWARP mosaics, to make sure that updayes to 3rd party software and updates for other curation use cases has not affected this code. I have also made some small test mosaics for quick checks." MAR reported: "A user was issuing very inefficient queries to the WSA. They would have been better off using the crossID access but they need to join in other tables as they need the epoch of observation (note this will be OK in DR3 as the epoch will be in the source tables). So provided a test version of crossID that allows for the joining in of other tables." RSC reported: "Held the software meeting on Monday, which was very helpful to allow us to prepare a new release branch for operations, release_2. We also took the opportunity to make the most of SVN's ability to painlessly move files/directories to place the helpers scripts directory together with the other Python codes within the curation directory. Also created a new development branch, dev_1, for working on DbSession refactoring to aid the parallel CUs. Updated the SoftwareIntroduction article on the Twiki to reflect change from CVS to SVN, and to note set-up instructions for password-less login between curation servers." ETWS reported: "Switched the webpage creation scripts from CVS to SVN where possible. Enhanced the ingester to use the given process order also as ingest order. The ingester is running the constraints checks on the table immediately after this table has been ingested. I'll do some tests on my test database to see if there is a best ingest order." NCH suggested that if the CU4 ingester is not in auto-commit mode, a possible speed up may be to load, check and then commit on a table by table basis, rather than a single commit transaction at the end... NCH noted a detailed performace profile study of seaming for the GPS, which has led to some speed-ups, but the process is still rather slow: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SpeedProfiles#Notes_on_overall_performance JB reported by email: "I went to a meeting by IS on Identity Management on Wednesday, it was only of tangential interest to us in the end but it may be of use in the future since they may be able to provide us the web page login functionality (usernames, passwords and institutions) in a devolved way that would allow us to control who has access but not to have to maintain any information about each user unless we wanted to (they were even promising to be able to provide known good email addresses of users). This is all in the (nearish) future for them at the moment though. I can provide a report at some point and provide the slides used more quickly." Survey Data Release: No update possible on T0 (we note that 16 nights of early 06B data have now been processed but not checked and released): T0+ What: 3.0 CU1 (transfer - assuming sustained 3MB/s) 1.5 CU2 (jpegs) 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 2.0 Final CUs. Watch this space for further updates... Non-survey Data Release: Nothing to report this week. Astrogrid deployment: ETWS & MAR reported creation of the 6df metadoc for AG publication of the same DB. Miscellaneous: Nothing else this week.