Issue between FTP Transmissions | Thales IoT Developer Community
March 21, 2016 - 9:39pm, 6415 views
I am having issues between FTP transmissions, mixture of fput, put transmissions.
Before the fault I notice I get a URC ‘Data Connection already open’ message.
It appears as though even a AT^SISC= 1 ‘Internet Connection Close command’ is issued between the FTP transmissions, and appears to accept the command, and therefore I would have expected the connection to close and allow me to establish a new connection immediatedly.
What can cause the connection not be closed after a at^sisc command?
If I inserted a big delay between the FTP transmissions it will complete ok – presumably after some other timeout delay.
Any thoughts on curing this issue would be appreciated.
I would have included more code, but the Spam filter is stopping me .
^SIS: 1,0,2100,"S:125 Data connection already open; Transfer starting."
^SISS detected.
SISW: 1,1 detected, ready to accept data
ftpStrLength4 = 174
FTP: Write String Request: AT^SISW=1,174
^SISW: 1,174,0
Uncaught exception: java.lang.NullPointerException: 0
at aL.a(), bci=30
at au.run(), bci=260
Senario2:
URC’s as above until 'DataConnection Open'.
^SISS detected.
URC - ATEvent
^SIS: 1,0,2100,"S:125 Data connection already open; Transfer starting."
^SISS detected.
Service State, SISI:AT^SISI=1
^SISI: 1,4,0,0,0,0 … indicating the FTP connection is up and running.
Repeating the above.
Bobg
Hello Bob,
As I can see you are using AT commands from the Java MIDlet. Have you also tested this scenario without Java?
There are in fact two connections in ftp: one for controlling the connection with ftp protocol and one for data transfer. So the information would suggest that the data connection is not closed yet. How big is that delay that is needed? Have you tried to check the service state after closing the connection? Looks like the connection would be kept longer for some reason.
As I understand you need to make a few ftp connections to the same server, one after another. Have you tried to configure more profiles with SISS and not to use the same for subsequent connections? Maybe that would help.
I can see one bad thing in your log: java.lang.NullPointerException - this suggests some problems in the application itself - not necessarily it is connected with the above problem but NullPointerException should not happen in the production version.
Regards,
Bartłomiej
Hi
Thank you for your reply which is much appreciated.
The testing I have carried out is not solely Java. The Java application is polling an external Micro-Controller - though for testing a HyperTerminal is substituted for the Micro-Controller with script files being used to provide the data to ASC0 on the EHS6T in response to the poll command.
Thus there is a delay between each FTP transmission - my response to issuing the script file.
The first file is just over 1Kbyte, and the second only 60 byte. So I would be surprised if the data connection was still open when trying to transmit so few bytes - this may give issues for later FTP transmissions which are larger files - up to 112Kbytes - which means affecting the overall FTP transmission time
I had not appreciated there are two connection. Though I already check the service state before I close the connection - I wait until the service returns the ^SISW URC indicating that is ready to accept new data - if I have interpreted this correctly - is this sufficient? - so not relying on just at^SISC. This would have been shown in my code if your web site filter hadn't stopped me adding it.
The nullPointerException only occurred once - though still one time too many !!, the 'Data Connection already Open' being the main problem.
If opening a new profile won't give an issue - ie potentially having multiple data connections open on different profiles, I'll try that, though my preference would be to just add the data connection check - unless it results in unacceptable delays between FTP transmissions. With the file sizes being transmitted varying considerably from 60 bytes to 112Kbytes, I'll need to check the overall affect on performance since this can be critical on certain applications with other factors limiting the order in which I do things.
Hello,
Thank you for the clarification. So it is the same as the commands would be issued on external interface while the custom MIDlet is running. Is the MIDlet doing any other network connections with pure Java? Please test the service state after closing the ftp connection (For FTP state 5 - closing - can mean that the command connection is already released, 4 - up - both connections are active). It would be important to know how long it takes to close the connection. At the moment I can't tell you why the data channel would last longer after issuing the close command if it really is the case. I'm not the ftp expert that much to claim if it is the server or module issue at the moment. But I'd rather not suspect that it could depend on the amount of data transferred.
We have experienced massive spam attacks some time ago and had to implement the stronger anti-spam protection and filters. So thank you for the information. I'll report it and hope that it will be improved soon.
Regards,
Bartłomiej
Hi
With the current testing HyperTerminal is being used purely to transmit an ASCII log in response to a poll command from the EHS6T - but essentially just doing the same as what would happen if the Micro-Controller is added.
Previous tests issuing AT script commands from HyperTerminal had not shown any issues - though I did not deliberately minimise any delays between FTP transmissions.
During testing today, on the 1st test, the 1st and 2nd transmissions transferred ok, and I added a 30sec delay before responding to the 2nd poll request from my MIDlet. I also added a 30sec delay before responding to the 3rd poll request, but this time it failed with the Data Connection open.
I reset the EHS6T, before running the MIDlet again- and this time it generated the Data Connection already open when I tried to do the 1st FTP transmission.
No other MIDlets are running - still trying to get one working consistently before I get more ambitious!!
Since I can get the issue even after resetting the EHS6T suggests to me that there is something on the Server side rather than the module and thus in this case would not be helped by testing the service state after closing the ftp connection.
However depending on how accurately it reports the state of the connection - it may be something to add at the beginning and end of the ftp transmission. I am not aware of other techniques to force closure of the connection - which is going to cause issues if I have to wait 5+ mintes after each ftp transmission.
Just checked the delay between and FTP transmission and the time for data to actually be seen on the FTP server using Filezilla. Noticed with the 1st ftp transmission it was of the order of 12 minutes (FFS file transfer - 1100 bytes), whereas the 2nd FTP transmission it transferred within about 1 min (at^SISW transfer, 60 bytes).
The subsequent ftp transmissions included a 47Kbyte file (FFS transfer) - and it appeared to transfer very quickly - there within 30secs.
However even though the file was seen on the server, the next FTP transmission (FFS) failed with the Data connection open issue.
Still to check for repeatability, but it appears a previous ftp transmission is only detected on the ftp server when the next ftp transmission occurs, and the 2nd one when the 3rd ftp transmission occurs. This is occuring when you have a FFS, at^SISW, at^SISW transmission. The responses indicate that the required number of bytes transmitted have been received - only with the at^sisw do you need to specify the number of bytes.
Bob
Hello Bob,
Based on your test results I think that it would be good to check with another server.
If the file transmission was finished on the module side and it is not visible on the server for a long time for another connection it might also be a server issue and some synchronisation inside the ftp provider's environment.
Regards,
Bartłomiej
Hi
I already use at^sisi to detect the session is 'Down' before I close the ftp session. I have noticed on occassions that although a transmission might have successfully occured, ^SISS is still Up / Listening and not in a 'Down' state - had to implement a timeout loop if this occurs.
Using HyperTerminal to check when 'Down' is detected and when the file detected on the FTP Server indicates that there is no noticeable delay between data being transmitted and <srvState> = 6 (Down), ^SISI: 1,6,0,n,0,0, where n is the number of bytes transmitted. There is the usual indeterminate delay 0-10 mins before the file appears on the FTP Server, so no correlation between 'Down' and when the data is detected on the FTP Sever, via Filezilla. But as indicated above, I can get a file transfer to the FTP Server without 'Down' being detected.
Athough it can appear that files are not 'visible' on the FTP Server until the next FTP transmission, this is not always the case, since the last transmission, a FFS transfer, will be seen on the FTP Server, eventually.
I tried another FTP account, but no difference - which may just indicate that this is a standard 'feature' of a 1and1 FTP.
One other strange thing I noticed was that although a 6kbyte FFS file transfer was ok via AT commands via HyperTerminal I received a ^SIS:1,0,100, "Stream Closed" error message when I tried to do a 27K FFS file transfer. There does not appear to be a <urcInfoId> 100 in the EHS6 AT Command Set.
Doing the equivalent file transfer within Netbeans appears to transfer ok (picks up the same file in FFS). In HyperTerminal the <txCount> is only showing 26229 bytes, whereas in Netbeans <txCount> is shown the correct 27398 bytes. For an unknown reason the Hyperterminal version is 1169 bytes short, even though picking up the same file in FFS.
Bob
Hello,
So this big delay seems to be the server issue. You would need to try with the completely different server to check the difference.
I also can't see <urcInfoId> 100 in the document. Does this 27K file always fail? The last thing seems strange - it shouldn't matter which terminal you are using while sending files from FFS. Is this always reproducible?
For me it looks more like the network quality problem if a little bigger (but still small) file transfer fails.
I've tried with my EHS6 module and my ftp server and I also was able to get similar "Stream Closed" message but only once. I was testing with 1.6 MB file without a problem many ***** then. Please check the network strength and if the module is in 2G or 3G.
Regards,
Bartłomiej
Hi
So far the issue is repeatable when initiating the 27K file transfer via HyperTerminal.
Sending the file transfer via a Java Midlet is considerable more reliable, and sends the full file size.
1and1 FTP Host blame the delay on the ISP, though I do not see why since the FTP commands and responses do not have these delays, only if the data was sent on yet another channel?. However large delays can also occur on file sizes < 100 bytes.
The signal strength is checked before tranmisssion when using the Java Midlet. Monitoring indicates that 3G is available most of the time, though I disable transmission if greater than -105dBm and noise level -4.0db -though it assumes the signal level remains strong during the transmission. During testing the levels are generally as high as -85dBM / -2.5db - and any issues don't appear to be related to the signal strength.
I've still to try on another FTP Host to see if this gives any further clues.
I am now starting to get occassionally getting the following responses from the FTP Server:
^sis:2,0,2100,"FTP open(FTPSite.co.uk:21)"
^sis: 2,0,2100,"220 Microsoft FTP service"
^sis: 2,0,2100, "FTP Login OK"
^sis: 2,0,2100, "ftp filename.txt"
^sis: 2,0,100, "isRunning FALSE"
^sis: 2,0,2100, "226 ABOR command successful."
^sis: 2,0,2100, "221 Goodbye."
Occassionally :
^sis: 2,0,2100, "S:425 Data Channel timed out due to not meeting the minimum bandwidth requirement."
A search for 425 indicates this may be due to an earlier connecton that has not timed out yet - though I deliberately waited > 240sec between FTP transmissions.
Subsequent FTP transmissions are ok, indicating a transient condition. Unclear why the transfer was aborted when previously it worked ok. Occurred when 3G GSM signal was -77dBm, N/R: -2.5db
Bob
Hello,
At the moment it's hard to think of anything that could connect the ftp failure while sending file from the module's file system and the terminal connected to the module unless it is the same module, same SIM card, antenna and everything. So maybe it would be worth trying with some other terminal.
If the transfer is finished on the module's side and connection closed (and maybe even module shut down) and the file is not visible on server for some time then the server is to blame for the delay.
I think that it would be a good idea to test with some other network provider. Even if the wireless network quality is not the case there may still be some provider related issue, for example the network may be overloaded. Or still some short term interference may occur.
Regards,
Bartłomiej
Hi
streamClosed - although I can get this message on the 27K file and apparent file size differences when transferred via MIDlet or transferred via AT commands - it appears the file is transferred correctly in both cases - and the file data contents is correct. Using FileZilla - the file size on the FTP Server side is different to the filesize shown when copied across to the local PC. Also I noticed the file size shown on the FTP Server can vary with different FTP Servers. For whatever reason the streamClosed is only given when using AT commands when using 1and 1 FTP Server - but not when running an equivalent MIDlet with embedded AT commands - hower if exaVault FTP server is used the streamClosed is given with both AT commands and MIDlet with embedded AT commands.
Transfer Delays - Using a different FTP Server, there was the same random delays between the final URC FTP response and when the file appeared on the FTP Server - on occassions it seemed worse.
If the files are not immediately on the FTP Server, I can stop the MIDlet and after a random delay the files will appear on the FTP Server.
1and1 claim when they tested my FTP Server on their system they did not experience the same issue. Using exaVault FTP server, with the same modem / SIM / Antenna gave similar random delays.
Network Provider - I'll check once I can get hold of another SIM provider.
isRunning / ABORT - this appears to be occuring on a 6K FTP file transfer - which is executed after the 27K transfer. although the 27K file is transferred ok and on the FTP Server. Tried adding a 5mins delay between the two FTP transmissions, but doing the same thing. This issue is occuring on two different EHS6T modems, one at Revision 02.00, and the other on 03.001
Doing the 6K FTP transfer before the 27K transfer will still result in the same fault.
Bob
Pages