UNDER CONSTRUCTION

basic-arch

If you want to use the Smarties in a wall display application, it might be not necessary to read what follows. Indeed, it is simpler to use a Smarties library with the default client software. What follows is useful for developing new clients, developing libraries or better understand what is behind the Smarties system.

The protocol use OSC and follow a "server - clients" logic. Typically the server application is the application that control the "slaves" that are in charge of the rendering. The clients should send messages to server (only) and the server send messages to the clients. All messages "address argument must start with the following prefix stying: "/Smarties/". We omit this prefix in what follows, thus when we talk about a message foo, the address argument should be "/Smarties/foo". You should never forget that the clients are totally agnostic to the semantic of the wall display application.

The default port for communication from the clients to the server is 57110, the default port for communication from the server to the clients is 57120. Other ports can be use of course, in particular if the server want to manage different type of Smarties client.

Hello Protocol

When a client starts it should send a NewConnection message to the server. Then, if the server runs, the server must respond with a HelloMessage, and, then the protocol must continue as explained in the next paragraph. If there is no server running a client must continue to send messages (of any type) to the servers in response to user interaction (e.g., touch messages).

Indeed, whenever the server receives a message from a client for the first time (the server never receive a msg with the ip address of that client), the server must send a HelloMessage to this client. Then, the client should reset itself and reply by GetHelloMessage, and then the server should send a series of messages to provide information to the client.

The above logic allows for "any time connection". Indeed, a client can be asked to re-connect to a (new) server at any time by receiving a HelloMessage message. Thus, it is (a priori) never necessary to restart a client. In the other hand, the NewConnection message allows for immediate connection if a server run.

  • NewConnection [Client to Server; 1 argument]
  • Type What
    String IP address of the client
  • HelloMessage [Server to Client; 5 arguments]
  • Type What
    String Version of the protocol
    int32 width of the wall display
    int32 height of the wall display
    int32 number of screens in the x-axe
    int32 number of screens in the y-axe

    Notes:
    - For now there is only one version of the protocol, version "1.0", we will take care of this later.
    - For now we support only simple wall configuration (uniform grids). In the future we might support more complex configuration.
    - Above information are just hints (e.g., to draw a grid on the touchpad area of a client, or to compute an appropriate transfere function for pointing). All "x,y" information send by the client to the server are relative.

  • GetHelloMessage [Client to Server; 3 arguments]
  • Type What
    String IP address of the client
    int32 width of the touchpad area
    int32 height of the touchpad area

    Notes:
    - Above information are just hints. All "x,y" information send by the server to the clients are relative.

    Initialisation Messages

    After the server and a client connect with the above protocol, the server should send the widgets, the smarties and some defaults to the client. This is done with several messages.

  • Defaults [Server to Client; 3 arguments]
  • Type What
    string Application name
    int32 multi-touch protocol, 0: Smarties, 1: raw
    int32 suggest how the client should perform smartie selection TBD

    See below for the description of the multi-touch protocol.

  • WidgetsList [Server to Client; variable number of arguments]
  • Type What
    int32 grid width
    int32 grid height
    int32 number of widgets
    Widget[ ] an array of (number of widgets) widgets

    Here the definition of a Widget:

    Type What
    int32 unique id of the widget
    int32 widget type (Button, Slider, ...) see below
    int32 visibility state (see below)
    float x coordinate of the widget position in the grid
    float y coordinate of the widget position in the grid
    float width of the widget in the grid
    float height of the widget in the grid
    string the main label of the widget
    string the on label (for switch button)
    bool off (false) or on (true), for switch widget
    int32 a value for valuator widget (e.g. sliders)
    int32 size of the items list for widget that use a list (could be zero)
    string[ ] the array of items label for widget that use a list
    (size defined above)
    int32 the select item for widget that use a list and support only one selected item
    -1: no item selected; -2: see next line
    bool[ ] if above is -2 the array specifying which items are selected in the list
    (size defined above)

    Wiget type:

    #define SMARTIES_WIDGET_TYPE_TEXTVIEW      0
    #define SMARTIES_WIDGET_TYPE_BUTTON        1
    #define SMARTIES_WIDGET_TYPE_TOGGLE_BUTTON 2
    #define SMARTIES_WIDGET_TYPE_TEXT_BUTTON   3
    #define SMARTIES_WIDGET_TYPE_CHECKBOX      4
    #define SMARTIES_WIDGET_TYPE_SLIDER        5
    #define SMARTIES_WIDGET_TYPE_SPINNER       6
    // NOT YET IMPLEMENTED ....
    #define SMARTIES_WIDGET_TYPE_RADIOGROUP     7
    #define SMARTIES_WIDGET_TYPE_POPUPMENU      8
    #define SMARTIES_WIDGET_TYPE_POPUPMENUCB    9
    #define SMARTIES_WIDGET_TYPE_POPUPMENURG    10
    

    Visibility state value:

    #define SMARTIES_WIDGET_STATE_ENABLED   0
    #define SMARTIES_WIDGET_STATE_DISABLED  1
    #define SMARTIES_WIDGET_STATE_HIDDEN    2
    

  • NewSmartie [Server to Client; 6 arguments]
  • Type What
    int32 unique id >= 0, id -1 is used for "no smartie"
    float x position of the smartie in relative coordinate [0;1]
    float y position of the smartie in relative coordinate [0;1]
    int32 "cursor" type to display inside the smartie (see below for definitions)
    int32 lock status 0 (for free) and 1 (for locked); or 2 in the case of an answer to a device
    that send a AskNewSmartie msg (and in this case the server could typically send
    a lock status to the other devices)
    int32 colors in #RRGGBB web format, -1 tells to use the default color
    (depending on the id)
    int32 flags (undefined for now)

    Cursor type:

    #define SMARTIE_TYPE_NONE      -1 
    #define SMARTIE_TYPE_CROSS      0
    #define SMARTIE_TYPE_ARROW      1
    #define SMARTIE_TYPE_POINT      2
    #define SMARTIE_TYPE_SELECT     3
    #define SMARTIE_TYPE_TEXT       4
    #define SMARTIE_TYPE_CIRCLE     5
    #define SMARTIE_TYPE_MOVE       6
    // etc ...
    

    Creation and management of a Smartie

    Only the server can create a smartie, the clients can only ask the server to create a new smartie. Formally, a client asks the server for a new smartie with the AskNewSmartie message (see below), and then the server respond (or not) by sending NewSmartie messages (see above) to the clients (with, for instance, a lock status of 0 for the asker and of 1 for the other clients).

  • AskNewSmartie [Client to Server; 3 arguments]
  • Type What
    String IP address of the client
    float x position of the smartie in relative coordinate [0;1]
    float y position of the smartie in relative coordinate [0;1]
  • SelectSmartie [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie

    Notes:
    - A client should use id = -1 to simply unselect.
    - A client has only the right to select a free smartie (see LockState and NewSmartie)
    - The server should take care of updating the locking status of the smarties for each devices (depending on the sharing policy).

  • ShareSmartie [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie

    Note: under certain sharing policy could be equivalent to SelectSmartie with id = -1

  • StartMoveSmartie [Client to Server; 5 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie
    int32 mode, 0 for hover, 1 for drag
  • MoveSmartie [Client to Server; 4 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie
    float x position of the smartie in relative coordinate [0;1]
    float y position of the smartie in relative coordinate [0;1]
    int32 mode, 0 for hover, 1 for drag (same than for StartMoveSmartie)
  • EndMoveSmartie [Client to Server; 4 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie
    int32 mode, 0 for hover, 1 for drag (same than for StartMoveSmartie)
  • DeleteSmartie [Client to Server; 3 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie
  • StoreSmartie [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie
  • UnstoreSmartie [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the smartie

    After a client ask for an action the server should update the state of the smartie on the other clients. This is done with the following messages.

  • MoveSmartie [Server to Client; 3 argument]
  • Type What
    int32 id of the smartie (might be -1)
    float x position of the smartie in relative coordinate [0;1]
    float y position of the smartie in relative coordinate [0;1]
  • DeleteSmartie [Server to Client; 1 argument]
  • Type What
    int32 id of the smartie
  • StoreSmartie [Server to Client; 1 argument]
  • Type What
    int32 id of the smartie
  • UnstoreSmartie [Server to Client; 1 argument]
  • Type What
    int32 id of the smartie
  • LockStatus [Server to Client; 2 arguments]
  • Type What
    int32 id of the smartie
    int32 state, free = 0, locked = 1

    Touch messages

    One finger touch (gesture, not tape beside smartie selection) is reserved to manipulate the smarties (that are a priori the application cursors) with the [Start,End]MoveSmartie client to server messages. The protocol provide two protocols for multi-finger touch. The "raw" protocol that roughly forward low level multi-touch events and the Smarties touch protocol. The protocol should be choosen by the server at connection time with the Defaults message (see above).

    The raw multi-touch protocol [Not yet implemented]

  • RawMTDown [Client to Server; 6 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 id of the "pointer"
    int32 x position of the pointer
    int32 y position of the pointer
    int32 time of the event for the client
  • RawMTUP [Client to Server; 6 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 id of the "pointer"
    int32 x position of the pointer
    int32 y position of the pointer
    int32 time of the event for the client
  • RawMTMotion [Client to Server; variable number of arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of pointers
    MotionEvent[ ] array of motion event for each pointer
    int32 time of the event for the client

    A motion event:

    Type What
    int32 id of the "pointer"
    int32 x position of the pointer
    int32 y position of the pointer

    The Smarties multi-touch protocol

    Smarties proposes simple (but powerfull) hight level multi-touch events.

  • MultiFingerMultiTape [Client to Server; 4 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of tape
    int32 number of finger of the last tape
  • StartMFMove [Client to Server; 9 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of fingers (>= 2)
    float x coordinate of the barycentre of the fingers contact points
    float y coordinate of the barycentre of the fingers contact points
    float sum of the distances from the the fingers contact points to the barycentre
    (mainly useful for StartMFPitch)
    float angle in degree formed by the oriented line passing through the 2 more ditsant
    fingers contact points and an oriented vertical line (left to right orientation)
    int32 number of tape before the move gesture start
    (could be used as a global modifier)
    int32 number of additional down fingers added during the gesture
    (must be 0 here)
  • MFMove [Client to Server; 9 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of fingers (>= 2)
    float x coordinate of the barycentre of the fingers contact points
    float y coordinate of the barycentre of the fingers contact points
    float sum of the distances from the the fingers contact points to the barycentre
    (mainly useful for StartMFPitch)
    float evolution of the angle definided in StartMFMove
    int32 number of tape before the move gesture start
    (must not change = to the corresponding StartMFMove value)
    int32 number of additional down fingers added during the gesture
    (could be used as a local modifier, e.g. to slowdown/speedup the control-action gain)
  • EndMFMove [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
  • StartMFPitch [Client to Server; 9 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of fingers (>= 2)
    float x coordinate of the barycentre of the fingers contact points
    (mainly useful for StartMFMove)
    float y coordinate of the barycentre of the fingers contact points
    (mainly useful for StartMFMove)
    float sum of the distances from the the fingers contact points to the barycentre
    (the pitch)
    float angle in degree formed by the oriented line passing through the 2 more ditsant
    fingers contact points and an oriented vertical line (left to right orientation)
    int32 number of tape before the move gesture start
    (could be used as a global modifier, e.g. zoom vs resize)
    int32 number of additional down fingers added during the gesture
    (must be 0 here)
  • MFPitch [Client to Server; 9 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie
    int32 number of fingers (>= 2)
    float x coordinate of the barycentre of the fingers contact points
    (mainly useful for StartMFMove)
    float y coordinate of the barycentre of the fingers contact points
    (mainly useful for StartMFMove)
    float sum of the distances from the the fingers contact points to the barycentre
    (mainly useful for StartMFPitch)
    float evolution of the angle definided in StartMFPitch
    int32 number of tape before the move gesture start
    (must not change = to the corresponding StartMFPitch value)
    int32 number of additional down fingers added during the gesture
    (could be used as a local modifier)
  • EndMFPitch [Client to Server; 2 arguments]
  • Type What
    String IP address of the client
    int32 id of the selected smartie

    Widgets Messages

    Some messages are sent from the clients to the server when the user use the widgets. The server can also send properties of the widgets. The idea is that a widget visibility/state/value/action might depend on the selected smartie.

    Client to server widget messages

    Server to client widget messages

    Soft keyboard

    The server can ask a client ....