============= Meeting 23-Apr-2012 =============

Agenda: http://sourceforge.net/mailarchive/message.php?msg_id=29152072

------------------------ Minutes ----------------------------

Present: BD, DH, GW, MK, GJ,  TK, AM, AJ,DC,RL,MD
Scribe: DH

NEW TOPIC: Status of unsigned and Field move, and V3/V4 interoperability changes *Marty*

 pvData: The remaining tasks are:
Convert tMust check carefully.
I have not asked Matej for the status but he is moving the caV3 client support from pvIOC to pvAccess. I will guess that he wants to do more testing.
The main task is to redo pvCopy, which is the code that allows a client to access an arbitrary subset 

 of the fields in a PVRecord.

Matej has to implement new code for c++ for regular version 3 channel access calls on client side in version 4.

GW: Can version 4 client use v3 IOC using channel access with no changes to Version 3?

MK: Yes.
AJ: Additional functionality MK wrote 2008 is going to be merged into 3.15 which allows "put process get" operations in V3.  The V4 server and the V4 client over CA layers could be modified to take advantage of this (but won't support it without changes)

GW: What's status of unsigned and field move?
MK: Other than checking convert and redoing pvCopy, pvAccess ready for Java.
Matej to move CAV3 move from pvIOC to pvIOC in progress. Not till done that 
will  we be able for version4 client to talk to version 3 IOC without change to v3.

If client and server both JAVA now ready need swap new API to new one.
Although need more testing.
GW: Concern that new user will download latest and examples won't work.
Can we do it now in meeting?
MK: Not done documentation.

MK; Don't want to everything.
DH: Happy to do hello world.
GW: MK, GW, DH sit round and make changes to hello world and maybe rdb example.

This is the current focus of my attention. The effort to move fieldName from Field to Structure is about the same as for Java. 
For implemention unsigned the effort is similar to that for Java. C++ templates ARE a big help. But Convert was not implemented via templates. It is being reimplemented via templates. I think this is nearly done but completion delayed because of std::vector discused below. When std::vector is done Convert should only take a day or so.
What is taking some additional effort is:
This is for array: size, capacity, and indexes. 
There are lots of changes but easy changes

std::vector. This is a BIG change.     It is the current focus of my effort.

Waiting for pvData
Waiting for pvData

MK: size_t change will allow C++ to handle really big arrays, like image data.
GW; How will a Java client talk to 64 bit C++ server? (Interop issues).
MK: Throw exception.
GW: What's exception? Interop issues cross language?
MK: Going to run into these issues cross languages.s to
GW: Need a "gotchas" document. Have existing doc called troubleshooting.

AI: On ?? Add interoperability to architectures document, particualrly unsigned and arrays size on 64 bit architectures on Java and C++.

MK: std::vector change is big change. Jossutis - if you put objects in vector must use smart pointers. MK tried to put raw pointers get's into trouble. Everything has to be vectors and smartpointers.

MD: Is there way of hiding the exact type of iterator  from client, so doesn't know its vector::iterator.

GW: What's situation with VxWorks
MK: If have tr1 then should be ok.
AJ: Will need version 6.
GW: To be clear v4 IOC will not be targetted at vXworks v5 then?
MK: Not till support tr1::smart_pointer.
MD: On behalf of Gubbao python picky about tr1 impl.
MK: Using boost::python.
AJ: Not impossible on older versions of GCC but may have to do some work [to make V4 work on vxWorks v5].

MK: MK reckons about a week (not this week) on pvDataCPP then Matej can pvAccessCPP- MK can work on pvIOC++ in parrallel.

GW: Head is incomplete and will be for several weeks.

AI: On GW to put note on web page and read me to say head incomplete.

NEW TOPIC: Normative Types
Normative Types are Applicaiton Level data types which will have a binding to an API. The Normative Types specification document allows these types to be interoperable between pvAccess endpoints. Instances of the data types that will be transported to PVAccess agents over pvAccess. They are implemeted in PVData. 
 Time Domain Arrays 

RL: What are NTs [in context of Time Domain data]?
GW: Application level structure definitions. In most cases types of fields are fixed, in some cases the value can be of one of the basic types.
E.g.: Samples from a digitizer do not have a timestamp for each of the samples, but a fix delta time between samples. Missing/broken samples could be marked by NaN values.

MD: Must include engineering units to make sense for general purpose clients.

NTFrequencyDomainArray needs two value arrays (I/Q, value/angle).

BD: Asks MD/DC to make a proposal on how to represent both use cases in one NT.
Will NTMatrix or a similar NT be able to serve this case?

RESOLUTION: NTTimeDomainArray and NTFrequencyDomainArray are dropped in favor of NTContinuum

MK: MASAR needs a different normative type to handle waveform data.  Not NTMultichannelArray because different channels have different types and the value array can only hold one particular data type.

NEW TOPIC:  NT Self-identification

GW: NT Self-identification.  Greg's proposal is:


1. Very quickly establish WHETHER the structure purports to carry an Ntype

2. Very quickly thereafter establish WHICH Ntype the structure purports to carry

3. User / institutional reserved bits. Say an institute wants to say "the structure carries EPICS V4 Normative type "MultichannelArray" but clients should expect it to additionally carry PSI defined fields

4. Ntype specialisation. Field value should indicate generally that the structure is a specialisation of a given nytpe. This should be interpreted as MUST carry all the required fields, may carry the defined optional fields.

5. Extensible, so new Ntypes may be added in future

6. Identifies which version of the Normative Types specification the carried normative type must conform to.
Firstly: Given an arbitrary pvData Structure, the joint condition of presence of a Field named "_NTYPE" and
whose type is uint (unsigned 32 bit integer) is necessary and sufficient to identify that the given
structure is intended to conform to one of the Normative Types. 

Secondly, value of the uint, that is named "_NTYPE", identifies the normative type, by the method:

xF F F F F F F F 
         | | | |-- Type id, int
         | | |----    "
         | |------ Specialization 
         | -------    "
 | | | |---------- Reserved for future expansion of basic types. 
 | | |------------ Reserved for future use.
 | |-------------- Institutional Nibble. Each institution can decide values and interp.
 |---------------- Draft number compliance and expansion. 

          N  Type Name                     x0000 FFFF mask of _NTYPE to identify type.
          -  --------------------          --------------------------------------------
        • 1  NTScalar                                         0001 0x1  2^(1-1) = 1  
        • 2  NTEnum                                           0010 0x2  2^(2-1) = 2 
        • 3  NTMatrix                                         0100 0x4  2^(3-1) = 4
        • 4  NTVariantArray                                   1000 0x8  2^(4-1) = 8
        • 5  NTNameValue                                 0001 0000 0x10 ...
        • 6  NTTable                                     0010 0000
        • 7  NTMultichannelArray                         0100 0000 
        • 8  NTTimeDomainArray                           1000 0000 
        • 9  NTFrequencyDomainArray                 0001 0000 0000 
        • 10 NTHistogram                            0010 0000 0000 
        • 11 NTAggregate                            0100 0000 0000 0x400 2^(11-1)= 1024
        • 12 NTImage                                1000 0000 0000 0x800 2^(12-1)= 2048

The Draft number compliance and expansion nibble
0x7000 0000 mask used to identify Normative Types specification draft (that is, so long
as 0x8000 0000 is down - see below). That is, 3 bits are taken to identify the
Normative Types draft, giving 8 drafts in all (including use of 000). Do we only need 2 bits, giving 4 drafts?

0x8000 0000 reserved for future use by EPICS V4 WG, 
and such used may change interpretation of 0x7000 0000 mask. That is, 3 bits are taken to identify the
Normative Types draft, giving 8 drafts in all (including use of 000). Do we only need 2 bits, giving 4 drafts?

The Normative Types specification draft identification number, 0-7 (base 10) will be included
in each draft of the Normative Types document. This number identifies the version of the specification to 
which the types defined in that document comprise.

Clients SHOULD be written to handle Normative Types of the draft for which they were written, and all previous drafts.

Servers MUST include a draft number in the 0x7000 0000 mask. 
Clients SHOULD check for draft number at least once, for each server agent to which the connect. If it is greater than the draft number with which they were written to be compliant, they SHOULD issue a warning at least. 

In some class, probably in pvService or pvData, there is a public class NormativeTypes.java

                        PVUInt normativetypeField = pvResult.getUIntField("_NTTPE");

                        if ( normativetypeField == null ) 
                                System.err.println("Unable to get data:"+
                                                   " unexpected data structure returned from "+ SERVICE_NAME +"."+ 
                                                   "Expected field member named '_NTYPE' but no such Field is present.");                                System.err.println(pvResult);
                        int type = normativetypeField.get();
                        if (type & 0x0000FFFF != NTTABLE_TYPE_NAME )        
                                System.err.println("Unable to get data: "+
                                                   "unexpected data structure value returned from "+SERVICE_NAME + ".");
                                System.err.println("Expected normativetype member value NTTABLE_TYPE_NAME ("+
                                                   NTTABLE_TYPE_NAME+"), but found type = " + type);
                        /* Ok, you got an NTTable. */

GW: Wants it to be able to say "this is a NT, but also express that this represents a subtype of it (i.e. plus these extra fields), which may be site-specific"

Withdraw the idea of using highest set bit to indicate the NT, replace it with 8 bits for an NT ID number plus 8 bits for specialization.

MK: Don't need to include institutional-specific info in the _NTYPE field.

MD: Why not make it a string?

RL: Does a client/server have to support multiple NT spec versions at the same time?

LD: Include a major and minor version number, incompatibility requires a new Major number.

DH: Why can't we identify the normative type using the introspection interface. E.g. field name starting with '_' is for NT Type ID purposes.
GW: Because you'd have to look for fields named _NTScalar, _NTEnum, etc...

Major Question: Should we use introspection data to get the NT type, or use value data.

RL: We need to know the type before we get the data.
AJ: It is possible to just ask for the _NTYPE field without the rest of the item, although that is another round-trip.

MK: First thing connect to the channel.  Then request the introspection data for it (separate round-trip, after connection).
LD: Optmise for the most common case, which is connect to service and get data.  Archiver with huge number of channels.

MD: Compares CA with PVA for round-trips on connection and finding the underlying NType.

AI: on Marty by 25-Apr. Ask Matej whether an NType string/ID can be returned  with the channel connection response. Also ask functionally, what is the most efficient way to establish whether a structure is an ntype.

Spec Versions: Still have to have any client version be able to talk to any server version, every server somehow has to tell the client the oldest and newest spec versions that this type is compliant with.

RL: If an NType is not backward-compatible with an older version of the spec, it MUST be a new type.

AJ: So we could use a uint32 for the NType ID, because a new type implies a new ID number.  The spec should list all type IDs.

1. Need input from MS on how to efficiently establish ntype of a PV
2. type conformance will be by specification document, not by individual type.

-- Coffee Break --

NEW TOPIC: Directory service
RL:   Considerable amount of time was spent changing the way this service is being packaged. There are bash scripts running the service and the client in the reference design, which are setting up the classpath in a non-portable way. The pvIOCJava loads XML files from withing the source directory structure to configure. The project was changed using a maven plug-in to create distribution tars.This is at the point of being able to get the distribution, but it is not connecting on the pvAccess layer. This does not connect error is  not a sufficient piece of information.

AI Matej: Exceptions on the server side should log the exception on the server. Additionally,  we need work to send back a reasonable error to the client. Connection error is given when a connection was made and any error occurred on the server. (In this case the server did not find the class that was needed to run the service - was the root cause of the original problem.) Different systematic errors are throwing the same exception: version incompatibilty, the name in XML does not match the name, and more causes - the client just shows "cannot connect".

ISSUE: - V4 development team: We need to have an error handling framework/policy for the server and client.

After getting the mechanism in place, completion of the Directory Service should be straight forward. 

Ralph will create a top level project for creating a distribution tar of the JAVA stuff (most of the needed works was already done as part of the directory service project).

When David, Marty and Greg finish the new release of the reference services, PV service dependencies now in existence should clean up.

NEW TOPIC: Support for save sets of scalars or vector of different types

AI: on Greg / Marty 25-Apr-2012 - make the new NT type - MultiChannelVariantArray which is a small variation of multichannel array that supports a list of PVs that can be either scalar or vector.

NEW TOPIC: Archive service demonstration
The data being returned is serveral arrays: timestamp, alarm serverity, alarm status, and values. This could be a normative type - with just a little more metadata.

The server connects to the archiver 

AI - David: Ready to publish now. 

AI: on  DH by : Publish this as the first C++ service available.