Errors when using package

Error: Cannot find the camera on the network

•  user@prX: roslaunch wge100_camera.launch
   [FATAL] 1239932051.129499000: No cameras found

   user@prX: roslaunch wge100_camera.launch
   MAC Address: 00:24:CD:00:00:67
   Product #6805018 : Unit #0003
   PCB rev B : HDL rev 103 : FW rev 108
   [FATAL] 1239932051.129499000: Couldn't find camera with S/N 10

Solution:

• Does the camera have power?
  • Check that the camera board LEDs are on.
• Does the camera have a working ethernet connection?
  • If one of the LEDs on the camera board is yellow, the connection is probably bad. If the link light on the router is off, the connection is probably bad.
• Is the Linux anti-spoof filter swallowing response packets from the camera?
  • The filter for your ethernet interface should be set to 0:

     user@prX: cat /proc/sys/net/ipv4/conf/eth?/rp_filter
     1
     user@prX: sudo bash
     root@prX: echo 0 > /proc/sys/net/ipv4/conf/eth?/rp_filter

Reset Cameras

You can try to reset all wge100 cameras on a PR2. The reset command sends a special command to the firmware that can allow the cameras to recover without power cycling the cameras.

• rosrun dynamic_reconfigure reconfigure_gui
• Select "pr2_camera_synchronizer"
• Check the "Reset" button at the bottom
• Wait until the reset button becomes unchecked.

After running this command, the cameras should recover. You can verify that all cameras are present by running pr2-systemcheck.

Error: Trigger mode set error

•  user@prX: roslaunch wge100_camera.launch
   Warning, was unable to create ARP entry (are you root?): Operation not permitted
   [ WARN] 1239932049.130394000: Unable to create ARP entry (are you root?), continuing anyway
   [ INFO] 1239932049.130577000: Configured camera #0, S/N #10, IP address 10.0.3.68
   [FATAL] 1239932051.129499000: Trigger mode set error

Solution:

• Is the camera ARP entry missing?
  • Run the node as root:

     user@prX: sudo bash
     root@prX: roslaunch wge100_camera.launch

    This will create the ARP entry during camera configuration, as you can verify:

     user@prX: arp
     Address                  HWtype  HWaddress           Flags Mask            Iface
     10.0.3.68                ether   00:24:CD:00:00:67   CM                    eth0

    After this running the WGE100 node as normal user should work fine, until you reboot the computer or attempt to use a different camera.
• Is the IP address assigned to the camera visible?
  • This is likely to be a problem if you are running the node on a robot for the first time. Check ifconfig:

     user@prX: ifconfig
     eth0      Link encap:Ethernet  HWaddr 00:01:80:76:67:e3  
               inet addr:10.68.0.2  Bcast:10.68.0.255  Mask:255.255.255.0

    The camera IP address 10.0.3.68 is not visible to this interface. Find an unused visible IP address with ping (let's say 10.68.0.68) and change the ip_address parameter in the launch file:

     <param name="ip_address" type="str" value="10.68.0.68"/>

    Don't forget to run the WGE100 node as root after this.

Error: Discover error / ioctl failed

•  user@prX: roslaunch wge100_camera.launch
   wgIpGetLocalAddr ioctl failed: No such device
   wgEthGetLocalMac ioctl failed: No such device
   Unable to create socket
   [FATAL] 1239943371.795521000: Discover error

Solution:

• Is the node broadcasting to the correct ethernet interface?
  • In wge100_camera.launch, set parameter "if_name" to "eth0" or "eth1" as needed.

Error: I'm getting images from the wrong camera

•  user@prX: roslaunch wge100_camera.launch
   MAC Address: 00:24:CD:00:00:67
   Product #6805018 : Unit #0003
   PCB rev B : HDL rev 103 : FW rev 108
   MAC Address: 00:24:CD:00:00:6E
   Product #6805018 : Unit #0010
   PCB rev B : HDL rev 103 : FW rev 108
   [ INFO] 1239943961.243337000: Configured camera #0, S/N #3, IP address 10.0.3.68

Solution:

• Is the node discovering multiple cameras?
  • In wge100_camera.launch, set parameter "serial_number" to that of the camera you want. If you don't know, check the list of discovered cameras printed by the node (Unit #). In this case we probably want:

     <param name="serial_number" type="int" value="10"/>

• Is the camera in the list of discovered cameras?
  • If not, see Error: Cannot find the camera on the network.

Error: Many packets are dropped

•  user@prX: roslaunch wge100_camera.launch
   [ INFO] 1239993196.341647000: Configured camera #1, S/N #10, IP address 10.0.3.68
  [ WARN] 1239993197.820380000: Short frame (video lines were missing)
  [ WARN] 1239993197.842489000: Short frame (video lines were missing)
  [ WARN] 1239993197.932087000: Short frame (video lines were missing)
  [ WARN] 1239993197.985493000: Short frame (video lines were missing)
  [ WARN] 1239993197.985972000: Short frame (video lines were missing)
  [ WARN] 1239993197.986327000: Frame was missing EOF, no frame information available
  [ WARN] 1239993198.009111000: Short frame (video lines were missing)

Solution:

• Is the Linux UDP buffer size large enough?
  • Set the UDP buffer size to 8MB:

     user@prX: sudo sysctl -w net.core.rmem_max=8388608

Dropped Frames and "rmem_max"

One common reason for dropped frames on Ethernet connected sensors is a buffer overrun in the kernel. The default kernel buffer size is rmem_default. Programs can increase the buffer size up to rmem_max using setsockopt, but they cannot change rmem_max. Only root can change rmem_max. When working with Ethernet cameras it is usually necessary to increase rmem_max so that the camera driver can allocate a large enough buffer. A poorly written driver might not know that it needs to increase its buffer, in which case rmem_default can also be adjusted (this will cause all applications to use large buffers though, so beware).

Temporary Changes

You can view rmem_max using sysctl

sysctl net.core.rmem_max

or using the /proc filesystem

cat /proc/sys/net/core/rmem_max

The same two ways are available to set rmem_max

sudo sysctl net.core.rmem_max=20000000

or using the /proc filesystem

echo 10000000 | sudo tee /proc/sys/net/core/rmem_max

Permanent Changes

The changes made using sysctl or writing to /proc are lost when the system is rebooted. To make the changes permanent, add a line to /etc/sysctl.conf

net.core.rmem_max=20000000

Programatic Changes

To set rmem_default for a socket, do

size_t bufsize = 20000000; 
if(setsockopt(s, SOL_SOCKET,SO_RCVBUF, &bufsize, sizeof(bufsize)) == -1) {
   ### Handle Error Here ###
}

and read it using

socklen_t bufsizesize = sizeof(bufsize);
if( getsockopt(s, SOL_SOCKET,SO_RCVBUF, &bufsize, &bufsizesize) == 0) {
    ### Use value here ###
}

Note that the read-back value will be double the written value on linux systems. Linux allocates more than is requested because user programs are not expected to anticipate packet storage overhead in the kernel.

You will need to include <sys/socket.h> (and possibly <sys/types.h>, see the setsockopt man page).

Error: No frames received for more than one second.

Solution:

• Is the trigger controller running?

Error: Trigger matcher dropping frames.

Solution:

• Is the driver running on a different computer from the controller? Is there a clock skew between the two computers?

Error: Failed to set trigger mode/IP.

Solution:

• Does the ARP table have a stale entry?
• Is the IP you specified or the default IP correct for the subnet the camera is on?

Warn: Camera Intrinsics have bad Checksum

Solution:

• Has the camera been calibrated?

  • Use the camera_calibration package to calibrate the camera.

Warn: Camera Intrinsics have bad Checksum

Solution:

• Use the camera_calibration package to calibrate the camera.

Error: Image resolution from intrinsics file does not match current video mode

Solution:

• Use the camera_calibration package to calibrate the camera for the current video mode.

• Revert to the previous video mode.

Camera Calibration is Wrong

Solution:

• Have you changed the Region of Interest settings since calibrating the camera?

  • You can use the camera_calibration package to calibrate the camera for the current video mode.

  • You can try reverting to the former Region of Interest settings.

Things that can go wrong and that need to be handled

• 100 Mbps cable to the head.
• Projector board not powered.
• Cameras not named right.
• Wrong firmware version.