EPICS V4 Working Group face-to-face Meeting, Day 1, 17-Oct-2012

Agenda, Day 1.

Wed AM - Review and Demo of key EPICS V4 functionality achieved

* Minutes

* Agenda Review.

* Status review:

Go through the line item charter exit demos per list [1] *Greg*
I predict this will take the whole morning. We really aim to demo every piece of 
infrastructure sw we've created, so it should take a long time! 

Wed PM - Services
1:30pm Relational Database service - demo. Dovetails into Db meeing. *Greg*
2:00pm Archive Service - *David*
2:30pm Model Service - *Guobao*
3:00 break
3:30pm Directory service demo, and it's applications for management and in CSS *Ralph*, *Kunal*
4:00pm  Masar service and python applications *Guobao*
4:30pm  Unit conversion service *Guobao*
5:00pm PXDB - beamline database user management demo Rick Buono


Wed AM - Review and Demo of key EPICS V4 functionality achieved

* Agenda Review.

* Status review:
Go through the line item charter exit demos per list [1] *Greg*
I predict this will take the whole morning. We really aim to demo every piece of 
infrastructure sw we've created, so it should take a long time! 

Wed PM - Services
* Relational Database service - demo. Dovetails into Db meeing. *Greg*
* Archive Service - *David*
* Model Service - *Guobao*. 
* Directory service demo, and it's applications for management and in CSS *Ralph*
* Elog Service and it's applications for management and viewing *WHO*
* Masar service and python applications *Guobao*
* Unit conversion service *Guobao*

Thurs AM - Requirements and Planning for beam line controls 

9:00am  Application of V4 architecture in High Level Applications Bob Dalesio
9:30am interoperaboility V3 V4     Marty Kraimer
10:00 break
10:30 am  Beam line requirements for scanning, data acquisition, and analysis, Daron Chabot.
11:00 am  synApps, record types and data. Tim Mooney.
11:30 am Use of areaDetector and synApps for biological data analysis. John Skinner.
12:00 noon Requirements for experiments at SNS. Kay Kaswemir
12:30-1:30 lunch
1:30-3:00 pm Version 4 IOC planning to support beamline controls.  Discussion. Bob Dalesio.
3:30 – 4;00 break
4:00 – 4:30pm Normative Types review. *Greg*
                         * EPICS URI. Review of use of the EPICS URL. *Greg*
4:30pm – 5:00 Proposal for "editing down" and implementation of N-types *Michael*

Friday AM - Next 12 months.

* Python physics applications that the BNL physics group has developed,
 What they like and what they want.  *Lingjun*

* Next charter deliverables and roadmap for the next 12 months. 
 + Review requirements generated since Wednesday.
 + Charter items proposal. *Greg*
 + Charter discussion.

======NEW TOPIC=================
= Status Review
Scribe: GS

MD: is there way to call ca://
BD: what is the min requirement?
GW: pvname is moved into part of URI
GW: V3 is easy. ca:///channelname
  V4 is complicated...
BD: should be status report. Details should be later
BD: for "organization structure", is it for people, or software?
GW: Software
BD: Does everyone agree with recommandation?
RL: Hardly yes
MS: Yes
DH: Yes

BD: Hold items to Friday

GW: review of the list of nominal tests for V4. 
relation of pvget/put and eget. 
MS: eget is oriented towards piping to different applications etc.
pvget is a caget-equivalent for pvAccess

GW: is there a eput
MS: no.
TK: pvget takes a long time of exit, reason?
MS: I will take a look at that
Defaults for processing with puts
MK: default for puts is now to process. It can be specified if processing should not take place
Documentation for request strings and their semantics is in pvAccess documentation. Should be in pvData.
Discussion about error messages in database initialization. Related to non-matching record types
AI: MK to revise the example database in the V3 channel example
MK: use of -r request string in eget automatically implies using structure mode in the reply
(issue: client then needs to parse the return string to get the value of the specified field)

Getting the values of all V3 standard fields
GW: even if I specify a different field, why does eget return the val field?
MK: possibly a bug. Will take a look at it
AI: MK to look at the behaviour of what the eget returns
MS: the bug of returning all fields even if one was specified has been fixed
GS: is there a utility similar to cainfo available
MS: not yet, but it is planned to have one

MS: pvput to arrays has the same syntax as caput
GS: in caput with arrays you have to explicitly to state that you are doing an array put. How does pvput know that this should
be an array put without the argument
DH: is the different sytax needed. create put knows about the channel type and should be able to figure out that the arguments
are array values
GS: how does pvput know that the first value stating the number of values is that, and not the first value
GW: keep the syntax for pvput. For eput, just use the values (without specifying the number of elements. Determine programmatically
the number of elements) eput can be used in scripts etc. to pipe 
RL: how would eput know where to separate the values for different channels (when putting to two channels)
GW: in that case use pvput (syntax will be kept)
pvput can do puts to different fields but not to two pv's
DH: will this work with string arrays
GW: yes, it should work
DH: when do we need quotes
GW: when using eput on the command line (as opposed to piping files)
GW: eget with arrays adds a blank to the end. This will break pipes
GS: how about getting a defined number of elements with eget
GW: it is not possible yet. Is this useful?
GS: yes.
RL: propose to use the syntax in 3.15. (first element, last element, step) Will send a proposal to Matej
MK: this implies changing ChannelArray
GW: is it useful to do data selection on the server side
(many): yes it is
is the dynamic array sizing supported in V4
MK: yes it is, however not with V3 pv's

eget doubleArray01 doubleArray02
  1  6
  3  7
  7  8
  eget -t doubleArray01 doubleArray02
1 3 7
6 7 8

KK: can I ask a service what arguments does it support
GW: it has to be supported by the service writer, but yes it is possible (details still subject to discussion)

eget -a columns=3 -a rows=5 -s testNTMatrix
    0.434822  0.0547287   0.824523
    1.75967     1.8181    1.73792
    2.17249    2.00028    2.71234
    3.34916    3.27615    3.28958
    4.01409    4.73464    4.08091

eget -a columns=3 -a rows=5 -s testNTMatrix -t
  0.435558   1.87279   2.59468   3.21029   4.12483
  0.419281   1.89822   2.81356   3.32535   4.98442
  0.851423   1.32008   2.56621   3.11472   4.10631

eget -a columns=3 -s testNTTable
   column0   column1   column2
  0.435738    1.1459   2.22137
   0.44465    1.0613   2.60045
  0.230346   1.23703   2.74881
  0.421738   1.77958   2.17418
  0.151863   1.33592   2.42832
  0.356142   1.88846   2.81326
  0.676708   1.36269   2.54078
  0.427452   1.77496   2.95499
  0.190023   1.67901   2.56949
  0.708225   1.13174   2.44391

eget -a columns=3 -s testNTTable -t
column0     0.435746  0.576188  0.985951  0.880388   0.68463  0.575101  0.723752  0.106317  0.869319  0.642918
column1       1.5294   1.58071     1.068   1.81415   1.38939   1.41816   1.03384   1.78265   1.06243   1.19859
column2      2.63886   2.28722   2.33778   2.10462   2.27449   2.34019   2.63275   2.60891   2.98549    2.1451

AI: on GW: define examples of cases where one needs terse output, or/and transposed output

Wed PM - Services
Archive Service - *David*

DH: Problems with getting the specific client to work with the new server model (no pvIOC), so the example can only use eget and misses the extensive formatting options of the specific client.

The specific client allows more versatile timestamp specification (on input) and e.g. more types of output formatting (e.g. time stamps as time-string, or as double for plotting).

BD: What functioniality do the physics apps expect?
Needed: Weekly reports on things. ~dozens of fixed query plots generated.

GW: SLAC uses Matlab tools for data mining, finding devices or physics entities that are drifting etc.

Diamond is looking forward to the NT Image Server - that should be made a deliverable.
-> Thursday - beamline session

KK: Non-streaming service might run into trouble, as archive queries might call for large amounts of data, which is a resource issue when the service has to create the complete answer structure befoe sending it.

GW: Don't waste time working on the solution for 1% of teh queries.
MD, RL: There have been recent issues with channelarchiver clients not being able to retrieve sets of waveform data.

GW: shows LCLS web pages that do reports (last hr - last yr) on important machine parameters ("how did we do last week?").

RL: Directory service demo
V4 service is a (separate) layer on top of the Glassfish implementation. Takes the request structure, converts it to the API of the directory service. Results are again converted to pvData structures and put on the wire.

KK: Could the directory service go directly to the database?
RL: it *could* but the implementation has a lot of businesslogic implemented in the database, it isolates the implementation from the database system below and provides a more robust system.

AI: on RL to implement sorting the results by multiple items, and show columns in the specified order

Discussion about handling the results in CSS. With the (current) V4 results CSS cannot know that the
returned data is in fact channel names. The problem would be solved by a NT type that specifies that
a string array contains channel names.
GW: it is planned (NTMultiChannelArray) but not yet implemented nor fully specified

NEW TOPIC: Channel Viewer

Kunal showed a CSS widget library for creating "channel finder driven" CSS displays.

And example bump display was shown.

NEW TOPIC: MASAR Service and Experiance at NSLS II
Guobao Shen

MASAR is a save and restore service.

Masar is now a V4 service. It doesn't presently use NTURI but it is using NTNameValue pair for arguments, and NTTable for returned data.

Masar uses SQLlite database for configs.

pvclient -> MASAR -> V3 iocs

MASAR can take a snapshot of any kind of V3 data, including waveform

The MASAR service only save and serves "snaphots", it doesn't itself restore snapshots. That is, it 
doesn't write to setpoints of the machine. Physicists use the sanpshots and write their own
software to implement those snapshots.

Lessons Leant
Design your configurations carefully. 
Tell physicists to be careful not to compare the values in two snapshots without
also being conscious of the times at which the channels were processed (which is also recorded in the event snapshot).

BD: Is there a way to search all the snaphsots for a given PV?
GS: No, short of greping the SQLLite database file

RL: Saving readbacks with the setpoints adds valuable information (e.g. setpoints from a ps that was switched off might be dangerous to restore).
Some restore operations need pre and post operations, e.g. to switch a device to "manual" before restoring setpoints, and back to "automatic" after.