The webcam in question was a microsoft HD-3000 which in theory is capable of 720p at 30fps. As with most webcams (which don't encode H264) it outputs YUYV (YUY2) and more importantly MJPEG. On linux you can easily determine what your webcam capabilities are by launching v4l2-ctl, eg:
This post provides a guide to what is possible with a gstreamer-imx plugins and the examples provided should not be treated as production ready. I'm assuming the examples will work on other usb webcams and or potentially other v4l2 devices. However the examples may require alterations depending on your webcam and it's capabilities, therefore background reading on gstreamer is recommended.
Testing was conducted on debian jessie and the target device was a AR6MXQ board provided by BCM Advanced Research. The examples should run on most imx6 devices as there is no device specific dependencies.
While testing with the HD-3000, the webcam occasionally stop sending a stream if I configured the wrong resolution. The workaround was to unplug the device or reset the usb port (replace '1-1.3' with the correct usb host and port id for your webcam) eg :
echo 1-1.3 > /sys/bus/usb/drivers/usb/unbind
echo 1-1.3 > /sys/bus/usb/drivers/usb/bind
Output to screen
Now back to the webcam, first step was getting output displayed to the screen. This requires reading the v4l2 source and converting the input so that is compatible with imxeglvivsink. Initial testing revealed decoding MPEG from the webcam performed significantly better than decoding YUY2. Another issue encountered (thanks to Carlos) was imxvpudec outputting Y42B something imeglvivsink currently can't cope with hence the inclusion of imxipuvideotransform element. So here is the final pipeline (webcam input is MPEG 720p at 30fps) :
gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! im
xvpudec ! imxipuvideotransform ! imxeglvivsink sync=false
CPU usage for the above was around 10%, while for YUY2 (720p at 10fps) it rises to 25% with this pipeline:
gst-launch-1.0 v4l2src ! 'video/x-raw,width=1280,height=720,framerate=10/1' ! imxipuvideotransform ! imxeglvivsink sync=false
Simple time lapsed video recording
Now lets implement a simple timelapsed video recorder that outputs to file and screen. I limited to the input MPEG stream to 10fps, reduced the bitrate and encoded as h264 to reduce the output file size. Additional CPU load occurs due to the inclusion of the clockoverlay element, without the element it is difficult to know when the recording was taken. Without this clockoverlay element CPU load is < 10%.
gst-launch-1.0 v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' ! imx
vpudec ! imxipuvideotransform ! clockoverlay time-format="%Y/%m/%d %H:%M:%S" ! tee name=splitter ! queue ! imxvpuenc_h264 bitrate=1024 ! filesink location=test .mp4 splitter. ! queue ! imxeglvivsink sync=false
The above pipeline generates approximately 400Mb per hour so probably not practical for production use.
Streaming to VLC
To enable streaming to VLC we need to create a 'sdp' file, this instruction VLC to act as a rtp server, below are the contents of the file:
m=video 5000 RTP/AVP 96
c=IN IP4 127.0.0.1
Save the contents to file eg 'imx6.sdp' and launch VLC, because there is no h/w acceleration for VLC on the imx6, VLC was running on a PC :
vlc --no-audio imx6.sdp
On the imx6, we are a rtp client submitting h264 payloads to the VLC server (note the inclusion of the host/ip address of the PC) :
gst-launch-1.0 -v v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !
imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 bitrate=1024 ! rtph264pay ! udpsink host=<host/ip of PC> port=5000
Simple RTSP server
And lastly, lets try running an rtp server. Fortunately there is an additional gstreamer plugin (gst-rtsp-server) with rtp support that includes an example test server. The downside is that it needs to be built from sources. You will need to checkout the 1.1.90 tag and build (similar to gstreamer-imx). Once built we can launch the example test server and pass it a pipeline similar to that used when outputting to screen (you need to export the library path so that libgstrtspserver-1.0.so is found) eg:
./test-launch '(v4l2src ! 'image/jpeg,width=1280,height=720,framerate=10/1' !imxvpudec ! imxipuvideotransform ! imxvpuenc_h264 ! rtph264pay name=pay0 pt=96 )'
The test server listens on port 8554 therefore the rtsp URL is
rtsp://<host/ip of imx6>:8554/test
You can test by launching VLC and opening a 'Network Stream' to the URL. While streaming the CPU load on the imx6 hovered around 60%. Given this is example code it should be possible to optimise the pipeline/code to bring down this figure.