UNDER CONSTRUCTION
If you want to use the Smarties in a wall display application, it might be not necessary to read what follows. Indeed, it is simpler to use a Smarties library with the default client software. What follows is useful for developing new clients, developing libraries or better understand what is behind the Smarties system.
The protocol use OSC and follow a "server - clients" logic. Typically the server application is the application that control the "slaves" that are in charge of the rendering. The clients should send messages to server (only) and the server send messages to the clients. All messages "address argument must start with the following prefix stying: "/Smarties/". We omit this prefix in what follows, thus when we talk about a message foo, the address argument should be "/Smarties/foo". You should never forget that the clients are totally agnostic to the semantic of the wall display application.
The default port for communication from the clients to the server is 57110, the default port for communication from the server to the clients is 57120. Other ports can be use of course, in particular if the server want to manage different type of Smarties client.
Hello Protocol
When a client starts it should send a NewConnection message to the server. Then, if the server runs, the server must respond with a HelloMessage, and, then the protocol must continue as explained in the next paragraph. If there is no server running a client must continue to send messages (of any type) to the servers in response to user interaction (e.g., touch messages).
Indeed, whenever the server receives a message from a client for the first time (the server never receive a msg with the ip address of that client), the server must send a HelloMessage to this client. Then, the client should reset itself and reply by GetHelloMessage, and then the server should send a series of messages to provide information to the client.
The above logic allows for "any time connection". Indeed, a client can be asked to re-connect to a (new) server at any time by receiving a HelloMessage message. Thus, it is (a priori) never necessary to restart a client. In the other hand, the NewConnection message allows for immediate connection if a server run.
Type | What |
---|---|
String | IP address of the client |
Type | What |
---|---|
String | Version of the protocol |
int32 | width of the wall display |
int32 | height of the wall display |
int32 | number of screens in the x-axe |
int32 | number of screens in the y-axe |
Notes:
- For now there is only one version of the protocol, version "1.0", we will take care of this later.
- For now we support only simple wall configuration (uniform grids). In the future we might support more complex configuration.
- Above information are just hints (e.g., to draw a grid on the touchpad area of a client, or to compute an appropriate transfere function for pointing). All "x,y" information send by the client to the server are relative.
Type | What |
---|---|
String | IP address of the client |
int32 | width of the touchpad area |
int32 | height of the touchpad area |
Notes:
- Above information are just hints. All "x,y" information send by the server to the clients are relative.
Initialisation Messages
After the server and a client connect with the above protocol, the server should send the widgets, the smarties and some defaults to the client. This is done with several messages.
Type | What |
---|---|
string | Application name |
int32 | multi-touch protocol, 0: Smarties, 1: raw |
int32 | suggest how the client should perform smartie selection TBD |
See below for the description of the multi-touch protocol.
Type | What |
---|---|
int32 | grid width |
int32 | grid height |
int32 | number of widgets |
Widget[ ] | an array of (number of widgets) widgets |
Here the definition of a Widget:
Type | What |
---|---|
int32 | unique id of the widget |
int32 | widget type (Button, Slider, ...) see below |
int32 | visibility state (see below) |
float | x coordinate of the widget position in the grid |
float | y coordinate of the widget position in the grid |
float | width of the widget in the grid |
float | height of the widget in the grid |
string | the main label of the widget |
string | the on label (for switch button) |
bool | off (false) or on (true), for switch widget |
int32 | a value for valuator widget (e.g. sliders) |
int32 | |
string[ ] | the array of items label for widget that use a list (size defined above) |
int32 | the select item for widget that use a list and support only one selected item -1: no item selected; -2: see next line |
bool[ ] | if above is -2 the array specifying which items are selected in the list (size defined above) |
Wiget type:
#define SMARTIES_WIDGET_TYPE_TEXTVIEW 0 #define SMARTIES_WIDGET_TYPE_BUTTON 1 #define SMARTIES_WIDGET_TYPE_TOGGLE_BUTTON 2 #define SMARTIES_WIDGET_TYPE_TEXT_BUTTON 3 #define SMARTIES_WIDGET_TYPE_CHECKBOX 4 #define SMARTIES_WIDGET_TYPE_SLIDER 5 #define SMARTIES_WIDGET_TYPE_SPINNER 6 // NOT YET IMPLEMENTED .... #define SMARTIES_WIDGET_TYPE_RADIOGROUP 7 #define SMARTIES_WIDGET_TYPE_POPUPMENU 8 #define SMARTIES_WIDGET_TYPE_POPUPMENUCB 9 #define SMARTIES_WIDGET_TYPE_POPUPMENURG 10
Visibility state value:
#define SMARTIES_WIDGET_STATE_ENABLED 0 #define SMARTIES_WIDGET_STATE_DISABLED 1 #define SMARTIES_WIDGET_STATE_HIDDEN 2
Type | What |
---|---|
int32 | unique id >= 0, id -1 is used for "no smartie" |
float | x position of the smartie in relative coordinate [0;1] |
float | y position of the smartie in relative coordinate [0;1] |
int32 | "cursor" type to display inside the smartie (see below for definitions) |
int32 | lock status 0 (for free) and 1 (for locked); or 2 in the case of an answer to a device that send a AskNewSmartie msg (and in this case the server could typically send a lock status to the other devices) |
int32 | colors in #RRGGBB web format, -1 tells to use the default color (depending on the id) |
int32 | flags (undefined for now) |
Cursor type:
#define SMARTIE_TYPE_NONE -1 #define SMARTIE_TYPE_CROSS 0 #define SMARTIE_TYPE_ARROW 1 #define SMARTIE_TYPE_POINT 2 #define SMARTIE_TYPE_SELECT 3 #define SMARTIE_TYPE_TEXT 4 #define SMARTIE_TYPE_CIRCLE 5 #define SMARTIE_TYPE_MOVE 6 // etc ...
Creation and management of a Smartie
Only the server can create a smartie, the clients can only ask the server to create a new smartie. Formally, a client asks the server for a new smartie with the AskNewSmartie message (see below), and then the server respond (or not) by sending NewSmartie messages (see above) to the clients (with, for instance, a lock status of 0 for the asker and of 1 for the other clients).
Type | What |
---|---|
String | IP address of the client |
float | x position of the smartie in relative coordinate [0;1] |
float | y position of the smartie in relative coordinate [0;1] |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
Notes:
- A client should use id = -1 to simply unselect.
- A client has only the right to select a free smartie (see LockState and NewSmartie)
- The server should take care of updating the locking status of the smarties for each devices (depending on the sharing policy).
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
Note: under certain sharing policy could be equivalent to SelectSmartie with id = -1
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
int32 | mode, 0 for hover, 1 for drag |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
float | x position of the smartie in relative coordinate [0;1] |
float | y position of the smartie in relative coordinate [0;1] |
int32 | mode, 0 for hover, 1 for drag (same than for StartMoveSmartie) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
int32 | mode, 0 for hover, 1 for drag (same than for StartMoveSmartie) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the smartie |
After a client ask for an action the server should update the state of the smartie on the other clients. This is done with the following messages.
Type | What |
---|---|
int32 | id of the smartie (might be -1) |
float | x position of the smartie in relative coordinate [0;1] |
float | y position of the smartie in relative coordinate [0;1] |
Type | What |
---|---|
int32 | id of the smartie |
Type | What |
---|---|
int32 | id of the smartie |
Type | What |
---|---|
int32 | id of the smartie |
Type | What |
---|---|
int32 | id of the smartie |
int32 | state, free = 0, locked = 1 |
Touch messages
One finger touch (gesture, not tape beside smartie selection) is reserved to manipulate the smarties (that are a priori the application cursors) with the [Start,End]MoveSmartie client to server messages. The protocol provide two protocols for multi-finger touch. The "raw" protocol that roughly forward low level multi-touch events and the Smarties touch protocol. The protocol should be choosen by the server at connection time with the Defaults message (see above).
The raw multi-touch protocol [Not yet implemented]
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | id of the "pointer" |
int32 | x position of the pointer |
int32 | y position of the pointer |
int32 | time of the event for the client |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | id of the "pointer" |
int32 | x position of the pointer |
int32 | y position of the pointer |
int32 | time of the event for the client |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of pointers |
MotionEvent[ ] | array of motion event for each pointer |
int32 | time of the event for the client |
A motion event:
Type | What |
---|---|
int32 | id of the "pointer" |
int32 | x position of the pointer |
int32 | y position of the pointer |
The Smarties multi-touch protocol
Smarties proposes simple (but powerfull) hight level multi-touch events.
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of tape |
int32 | number of finger of the last tape |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of fingers (>= 2) |
float | x coordinate of the barycentre of the fingers contact points |
float | y coordinate of the barycentre of the fingers contact points |
float | sum of the distances from the the fingers contact points to the barycentre (mainly useful for StartMFPitch) |
float | angle in degree formed by the oriented line passing through the 2 more ditsant fingers contact points and an oriented vertical line (left to right orientation) |
int32 | number of tape before the move gesture start (could be used as a global modifier) |
int32 | number of additional down fingers added during the gesture (must be 0 here) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of fingers (>= 2) |
float | x coordinate of the barycentre of the fingers contact points |
float | y coordinate of the barycentre of the fingers contact points |
float | sum of the distances from the the fingers contact points to the barycentre (mainly useful for StartMFPitch) |
float | evolution of the angle definided in StartMFMove |
int32 | number of tape before the move gesture start (must not change = to the corresponding StartMFMove value) |
int32 | number of additional down fingers added during the gesture (could be used as a local modifier, e.g. to slowdown/speedup the control-action gain) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of fingers (>= 2) |
float | x coordinate of the barycentre of the fingers contact points (mainly useful for StartMFMove) |
float | y coordinate of the barycentre of the fingers contact points (mainly useful for StartMFMove) |
float | sum of the distances from the the fingers contact points to the barycentre (the pitch) |
float | angle in degree formed by the oriented line passing through the 2 more ditsant fingers contact points and an oriented vertical line (left to right orientation) |
int32 | number of tape before the move gesture start (could be used as a global modifier, e.g. zoom vs resize) |
int32 | number of additional down fingers added during the gesture (must be 0 here) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
int32 | number of fingers (>= 2) |
float | x coordinate of the barycentre of the fingers contact points (mainly useful for StartMFMove) |
float | y coordinate of the barycentre of the fingers contact points (mainly useful for StartMFMove) |
float | sum of the distances from the the fingers contact points to the barycentre (mainly useful for StartMFPitch) |
float | evolution of the angle definided in StartMFPitch |
int32 | number of tape before the move gesture start (must not change = to the corresponding StartMFPitch value) |
int32 | number of additional down fingers added during the gesture (could be used as a local modifier) |
Type | What |
---|---|
String | IP address of the client |
int32 | id of the selected smartie |
Widgets Messages
Some messages are sent from the clients to the server when the user use the widgets. The server can also send properties of the widgets. The idea is that a widget visibility/state/value/action might depend on the selected smartie.
Client to server widget messages
Server to client widget messages
Soft keyboard
The server can ask a client ....