SerialPort visible in device manager but SerialPort.Open() reports that it cannot be found

I have a .NET application that listens on a COM port. The other side of the connection is a UDOO board running lubuntu.

After the udoo board is rebooted, the COM port temporarily goes missing and then re-appears on the device manager. However, when I call the "Open" function of the SerialPort object, I get an exception which says "The port COM4 does not exist". The full exception stack trace is as follows.

 at System.IO.Ports.InternalResources.WinIOError(Int32 errorCode, String str)
   at System.IO.Ports.SerialStream..ctor(String portName, Int32 baudRate, Parity parity, Int32 dataBits, StopBits stopBits, Int32 readTimeout, Int32 writeTimeout, Handshake handshake, Boolean dtrEnable, Boolean rtsEnable, Boolean discardNull, Byte parityReplace)
   at System.IO.Ports.SerialPort.Open()

Even when this exception is thrown, device manager shows the very same COM port as visible as in the following screenshot.

And yes I did refresh the device list. Unplugging and plugging the cable back to the computer solves this problem but, it isn't elegant.

Are there any thoughts on how I could work around this problem ?