The hardware board used in this article is the Forlinx embedded OKMX8MP-C development board, the system version is Linux5.4.70+Qt5.15.0, and it mainly introduces the MJPG code stream transmission based on HTTP web server and UDP host computer.

As a continuous transmission video stream, MJPG format is widely used in the field of remote monitoring, and there are two most common third-party applications for this kind of remote monitoring: browser HTTP webpage and UDP host computer.

Both have their own advantages and contrast, among them:

• UDP host computer: high transmission efficiency, easy to write on the host computer.
• HTTP web page method: the client does not need to install a host computer, only a browser application is needed; the client access server supports cross-platform support, whether it is a computer, tablet, mobile phone, or Linux system, Windows system and Android system, as long as All browser applications can be accessed, but the UDP host computer is limited by the target platform and is not easy to transplant.

Both applications have advantages and disadvantages, and both must be mastered by embedded developers.

1. HTTP web server

The HTTP web server obtains the code of the MJPG stream. First, OKMX8MP-C establishes a TCP server on the development board:

int TCP_Server_Found(socklen_t* socket_found , char* ip , int port)
{  struct sockaddr_in servaddr;
    socklen_t addrsize = sizeof(struct sockaddr);
    bzero(&servaddr , sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = inet_addr(ip);
    servaddr.sin_port = htons(port);
    int ret;
    IF( (*socket_found = socket(AF_INET , SOCK_STREAM , 0)) == -1)
        {
            printf("Create socket error: %s (errno :%d)\n",strerror(errno),errno);
            return -1;
        }
    int on = 1;
    if(setsockopt(*socket_found , SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) < 0)
    {
        printf("setsockopt error\n");
    }
    ret = bind(*socket_found , (struct sockaddr *)&servaddr , addrsize);
    if(ret == -1)
    {
            printf("Tcp bind faiLED!\n");
            return -1;
    }
    if(listen(*socket_found , 5) == -1)
    {
            printf("Listen failed!\n");
            return -1;
    }
    return 0;
}

The setsockopt() function is optional and is generally only used to avoid the establishment error of the socket() function.

After the TCP server is established, the returned socklen_t type actual parameter needs to be used in the following HTTP web server.

The TCP operation to which the HTTP web server belongs requires a separate polling thread to allow the client to perform the accept() handshake operation. The listen() before accept() only needs to be executed once. The accept() handshake operation and recv() ) The receive operation requires the creation of an infinite loop thread:

pthread_create(&tid_tcp_web_recv , NULL , Thread_TCP_Web_Recv , NULL);
void * Thread_TCP_Web_Recv(void *arg)
{
。。。
while(1)
{
            fd_socket_conn = accept(socket_web_server , (struct sockaddr *)&sockaddr_in_conn , &addrsize);
           printf("fd_socket_conn = accept()\n");
    。。。
    recv(fd_socket_conn , recvbuf , 1000 , 0);
}
。。。
}

MJPG frames can be obtained using Grab operations. The obtained MJPG frames need to be read in the TCP thread and written in the Grab operation thread. Such resources accessed by multiple threads need to be locked to prevent read-write conflicts, that is, the resources are written by Grab operations. When entering, it needs to be locked, and other threads are not allowed to access. When the operation is completed, it needs to be unlocked to allow other threads to access:

    pthread_mutex_lock(&pmt);
    pic_tmpbuffer = pic.tmpbuffer;
    pic.tmpbytesused = buff.bytesused;
    pic_tmpbytesused = pic.tmpbytesused;
    pthread_cond_broadcast(&pct);
    pthread_mutex_unlock(&pmt);

The thread mutex needs to be initialized before use:

pthread_mutex_t pmt;
pthread_cond_t pct;
int main(int argc, char* argv[])
{
...
TCP_Server_Found(&socket_web_server , (char*)argv[2] , PORT_TCP);
pthread_mutex_init(&pmt , NULL);
    pthread_create(&tid_tcp_web_recv , NULL , Thread_TCP_Web_Recv , NULL);
    pthread_create(&tid_tcp_web_send , NULL , Thread_TCP_Web_Send , NULL);
...
    while(1)
    {
        V4l2_Grab_Mjpeg(false , MJPEG_FILE_NAME);
...
    }
...
}

Then there are the details of sending. Before sending image files, you need to send HTTP standard headers, which is equivalent to paving the way for sending images or other types of streaming data:

#define STD_HEADER "Connection: close\r\n" \
    "Server: MJPG-Streamer/0.2\r\n" \
    "Cache-Control: no-store, no-cache, must-revalidate, pre-check=0, post-check=0, max-age=0\r\n" \
    "Pragma: no-cache\r\n" \
    "Expires: Mon, 3 Jan 2000 12:34:56 GMT\r\n"
#define BOUNDARY "boundarydonotcross"
    printf("preparing header\n");
    sprintf(buffer, "HTTP/1.0 200 OK\r\n" \
            "Access-Control-Allow-Origin: *\r\n" \
            STD_HEADER \
            "Content-Type: multipart/x-mixed-replace;boundary=" BOUNDARY "\r\n" \
            "\r\n" \
            "--" BOUNDARY "\r\n");
    if(write(fd, buffer, strlen(buffer)) < 0)
    {
        free(frame);
        return;
    }

After sending the HTTP standard header, you need to send the Content-Type. The Content-Type here is image/jpeg. Similarly, the image supported in the HTTP standard protocol is far more than jpeg. After the content header is sent, it is the body and boundary. At the end, the complete HTTP header of the frame is sent to the specified TCP GET address, and the picture just sent will be displayed in the browser:

       sprintf(buffer, "Content-Type: image/jpeg\r\n" \
                "Content-Length: %d\r\n" \
                "X-Timestamp: %d.%06d\r\n" \
                "\r\n", frame_size, (int)timestamp.tv_sec, (int)timestamp.tv_usec);
        printf("sending intemdiate header\n");
        if(write(fd, buffer, strlen(buffer)) < 0)
            break;
        printf("sending frame\n");
        if(write(fd, frame, frame_size) < 0)
            break;
        printf("sending boundary\n");
        sprintf(buffer, "\r\n--" BOUNDARY "\r\n");
        if(write(fd, buffer, strlen(buffer)) < 0)
            break;

In addition, it should be noted that the TCP server thread sends the MJPEG stream in an infinite loop. Therefore, after the TCP client sends the GET command, it will receive the image cache sent by the TCP server in a loop, and the TCP client will be busy. In the waiting state, no GET or POST commands can be sent to the outside world. From the perspective of the client user, the effect is that the web page has been waiting.

2. UDP host computer

UDP sending operation also needs to establish a UDP Socket first:

int UDP_Send_Found(socklen_t* socket_found , struct sockaddr_in *addr , char* ip , int port)
{
    *socket_found = socket(AF_INET, SOCK_DGRAM, 0);
    if(*socket_found == (~0))
    {
        printf("Create udp send socket failed!\n");
        return -1;
    }
    addr->sin_family = AF_INET;
    addr->sin_addr.s_addr = inet_addr(ip);
    addr->sin_port = htons(port);
    memset(addr->sin_zero, 0, 8);
    return 0;
}

UDP file sending is much simpler than HTTP sending. It only needs to slice the file, each slice is a fixed-length UDP frame length, and can be sent frame by frame:

while(fend > 0)
{
memset(picture.data , 0 , sizeof(picture.data));
fread(picture.data , UDP_FRAME_LEN , 1, fp);
if(fend >= UDP_FRAME_LEN)
{
picture.length = UDP_FRAME_LEN;
picture.fin = 0;
}
else
{
picture.length = fend;
picture.fin = 1;
}
//printf("sendbytes = %d \n",sendbytes);
sendbytes = sendto(socket_send, (char *)&picture, sizeof(struct Package), 0, (struct sockaddr*)&addr,addr_len);
if(sendbytes == -1)
{
printf("Send Picture Failed!d\n");
return -1;
}
else
{
fend -= UDP_FRAME_LEN;
}
}