Introduction ============ Inter Process Communication(IPC) Message Router IPC Router provides a connectionless message routing service between multiple processes in a MSM setup. The communicating processes can run either in the same processor or in a different processor within the MSM setup. The IPC Router has been designed to 1) Route messages of any types 2) Support a broader network of processors The IPC Router follows the same protocol as the existing RPC Router in the kernel while communicating with its peer IPC Routers. Hardware description ==================== The IPC Router doesn't implement any specific hardware driver. The IPC Router uses the existing hardware drivers to transport messages across to the peer router. IPC Router contains a XPRT interface layer to handle the different types of transports/links. This interface layer abstracts the underlying transport complexities from the router and provides a packet/message interface with the transports. Software description ==================== The IPC Router is designed to support a client-server model. The clients and servers communicate with one another by exchanging data units known as messages. A message is a byte string from 1 to 64K bytes long. The IPC Router provides a connectionless message routing service between the clients and servers i.e. any client/server can communicate with any other client/server in the network of processors. Network Topology Overview: -------------------------- The system is organized as a network of nodes. Each processor in the network is the most fundamental element called node. The complete network is hierarchically structured i.e. the network is divided into tiers and each tier is fully-meshed. The following figure shows an example network topology. ---N1--- ---N4--- | | | | | | | | N2----N3-------N5-----N6 | | | | ---N7---- | | | | N8------N9 Each node in the complete network is identified using a unique node id (Nx in the example network). In the example network, nodes N1, N2 & N3 are fully-meshed to form a tier 1 network. Similarly nodes N4, N5 & N6 form another tier 1 network and nodes N7, N8 & N9 form third tier 1 network. These three tier 1 networks are fully-meshed to form a tier 2 network. Each transport/link in the network is identified using a unique name/id called XPRT id. This XPRT id is used by the nodes to identify the link to be used while sending message to a specific destination. In addition, each transport/link in the network is assigned a link id. This link id is used to identify the tier to which the link belongs to. This link marking is used to avoid the routing loops while forwarding the broadcast messages. The incoming messages are only forwarded onto an egress link which has a link id different from that of an ingress link. IPC Router Addressing Overview: ------------------------------- Each client/server in the network is identified using a unique combination. Node_id identifies the processor on which a client/server is running. Port_id is a unique id within a node. This Port_id is assigned by the IPC Router in that node when a client/server comes up. The Node_id & Port_id are 32 bits each. Port_id 0xFFFFFFFE is reserved for Router & 0xFFFFFFFF is reserved for broadcast messages. Each server in the network can also be addressed using a service name. The service name is of the form . When a server comes up, it binds itself with a service name. This name information along with the is broadcast onto the entire network. Control Path: ------------- IPC Router uses the control messages to communicate and propagate the system wide events to the other routers in the system. Some of the events include: 1) Node Status 2) Server Status 3) Client Status 4) Flow Control Request/Confirmation Message Header: --------------- IPC Router prepends a header to every message that it communicates with other IPC Router. The receiving IPC Routers use the header to identify the source and destination of the message, size of the message, type of the message and handle any flow control requests. The IPC Router header format is as follows: 0 31 ------------------------------------------------- | Version | ------------------------------------------------- | Message Type | ------------------------------------------------- | Source Node ID | ------------------------------------------------- | Source Port ID | ------------------------------------------------- | Confirm RX | ------------------------------------------------- | Payload Length | ------------------------------------------------- | Destination Node ID | ------------------------------------------------- | Destination Port ID | ------------------------------------------------- Message Header v2(Optimized): ----------------------------- The following optimization has been done to the IPC Router header to make it concise, align with the system requirement and enable future expansion: 0 8 16 24 31 ----------------------------------------------------------------- | Version | Message Type | Control Flag | ----------------------------------------------------------------- | Payload Length | ----------------------------------------------------------------- | Source Node ID | Source Port ID | ----------------------------------------------------------------- | Destination Node ID | Destination Port ID | ----------------------------------------------------------------- Control Flag: 0 14 15 ------------------------------------------------------------------ | Reserved | Opt. Hdr. | Confirm RX | ------------------------------------------------------------------ IPC Router identifies and processes the header depending on the version field. The Confirm RX field in message header v1 becomes part of the control flag. All the other fields are reduced in size to align with the system requirement. Optional Header: An optional header bit field is introduced in the control flag to handle any unforeseen future requirement that this header cannot handle. When that bit is set, an optional header follows the current header. The optional header format is as follows: 0 8 16 31 ----------------------------------------------------------------- | Length(words) | Type | Control Flag | ----------------------------------------------------------------- | | | Optional Header Contents | | | ----------------------------------------------------------------- Design ====== The IPC Router is organized into 2 layers: 1) Router Core layer 2) Router - XPRT Interface layer This organization allows the router to abstract the XPRT's complexities from that of the core router functionalities. The Router Core layer performs the following core functions: 1) Message Routing 2) Distributed name service 3) Flow control between ports The Router core layer contains the following important data structures to perform the core functions in their respective order: 1) Routing Table 2) Table of Active Servers 3) Table of Active Remote ports All these data structures get updated based on the events passed through the control path. The Router - XPRT Interface layer hides the underlying transport complexities and provides and abstracted packet interface to the Router Core layer. The Router - XPRT Interface layer registers itself with the Router Core layer upon complete initialization of the XPRT. The Router - XPRT Interface layer upon registration exports the following functionalities to the Router Core: 1) Read from the XPRT 2) # of bytes of data available to read 3) Write to the XPRT 4) Space available to write to the XPRT 5) Close the XPRT The user behavioral model of the IPC Router should be 1) Create a port 2) If server, register a name to the port 3) If remote port not known, lookup through the name 4) Send messages over the port to the remote port 5) Receive messages along with the source info from the port 6) Repeat steps 3, 4 & 5 as required 7) If server, unregister the name from the port 8) Close the port Power Management ================ IPC Message Router uses wakelocks to ensure that the system does not go into suspend mode while there are pending messages to be handled. Once all the messages are handled, IPC Message Router releases the wakelocks to allow the system to go into suspend mode and comply with the system power management requirement. SMP/multi-core ============== The IPC Router uses mutexes & spinlocks to protect the shared data structures to be SMP/multi-core safe. Security ======== None Performance =========== None Interface ========= Kernel-space APIs: ------------------ /* * msm_ipc_router_create_port - Create a IPC Router port * * @msm_ipc_port_notify: notification function which will notify events * like READ_DATA_AVAIL, WRITE_DONE etc. * @priv: caller private context pointer, passed to the notify callback. * * @return: a valid port pointer on success, NULL on failure * */ struct msm_ipc_port * msm_ipc_router_create_port( void (*msm_ipc_port_notify)(unsigned event, void *data, void *addr, void *priv), void *priv) /* * msm_ipc_router_close_port - Close down the port * * @port: Port to be closed * * @return: 0 on success, -ve value on failure * */ int msm_ipc_router_close_port(struct msm_ipc_port *port) /* * msm_ipc_router_send_to - Send data to a remote port * * @from_port: Source port of the message * @data: Data to be sent * @to_addr: Destination port name or address * * @return: number of bytes sent on success, -ve value on failure * */ int msm_ipc_router_send_to(struct msm_ipc_port *from_port, struct sk_buff_head *data, struct msm_ipc_addr *to_addr) /* * msm_ipc_router_recv_from - Receive data over a port * * @port: Port from which the data has to be read * @data: Pointer to the data * @src_addr: If valid, filled with the source address of the data * @timeout: Time to wait for the data, if already not present * * @return: number of bytes received on success, -ve value on failure * */ int msm_ipc_router_recv_from(struct msm_ipc_port *port, struct sk_buff_head **data, struct msm_ipc_addr *src_addr, unsigned long timeout) /* * msm_ipc_router_register_server - Bind a local port with a service * name * * @server_port: Port to be bound with a service name * @name: Name to bind with the port * * @return: 0 on success, -ve value on failure * */ int msm_ipc_router_register_server(struct msm_ipc_port *server_port, struct msm_ipc_addr *name) /* * msm_ipc_router_unregister_server - Unbind the local port from its * service name * * @server_port: Port to be unbound from its service name * * @return: 0 on success, -ve value on failure * */ int msm_ipc_router_unregister_server(struct msm_ipc_port *server_port) /* * msm_ipc_router_lookup_server - Lookup port address for the port name * * @name: Name to be looked up for * * @return: Port address corresponding to the service name on success, * NULL on failure * */ struct msm_ipc_addr * msm_ipc_router_lookup_server( struct msm_ipc_addr *name) User-space APIs: ---------------- User-space applications/utilities can use the socket APIs to interface with the IPC Router. IPC Router, in order to support the socket APIs, will register a new socket Address/Protocol Family with the kernel Socket layer. The identity of the new Address/Protocol Family will be defined using the macro AF_MSM_IPC/PF_MSM_IPC (hardcoded to 38) in include/linux/socket.h file. Since IPC Router supports only message oriented transfer, only SOCK_DGRAM type of sockets will be supported by the IPC Router. Driver parameters ================= debug_mask - This module parameter is used to enable/disable Router log messages in the kernel logs. This parameter can take any value from 0 to 255. Dependencies ============ Drivers in this project: ----------------------- The following drivers are present in this project, listed in the bottom - up order of the stack. 1a) Router - SMD XPRT Interface driver. This driver is used to interface the Router with the SMD transport. 1b) Router - HSIC XPRT Interface driver. This driver is used to interface the Router with the HSIC_IPC Bridge transport for off-chip communication. 2) Core Router driver. This driver performs the core functionalities 3) Socket - Router Interface driver. This driver enables the socket interface to be used with the IPC Router. In the write/send direction, these drivers interact by invoking the exported APIs from the underlying drivers. In the read/receive directions, these drivers interact by passing messages/events. Drivers Needed: --------------- 1) SMD 2) Kernel Socket Layer 3) Platform Driver 4) HSIC IPC Bridge Driver To do ===== Improvements: ------------- The IPC Router is designed to route any messages, as long as the system follows the network architecture and addressing schemes. But the implementation in progress will route only QMI messages. With few additional enhancements, it can route existing RPC messages too.