Last few weeks, I was juggling with huge amount of spatial data (temporal extent: 20 years (weekly data), spatial extent: whole world) from different sources (i.e., ESA, DLR, VITO, USGS, USDA etc.) which are in different formats (i.e., *.img, *.tif, *.txt, *.ascii, .GRID etc.) and of different projections. It was a complete mess. I am supposed to process all these data by bringing all in one common format/common projection system and calculate the long-term average and provide the averaged data in a spread sheet format according to some administrative unit. ops! that was a nightmare.
What i have tried to come to a solution:
1. First, I have tried both commercial and open-source software (i.e., ArcGIS, Qgis). trust me, that was ‘THE’ most dumbest idea i have had for such processing.
2. Secondly, I tried some Java library (i.e., GeoTools). that wasn’t bad but still I had to face the problem of converting from different format to GeoTIFF and of-course projection and transformation problem; that’s usual (as some provider always have user-defined projection system and woohooo!!! that’s some crap you have to spend some more time then you might expect).
3. Thirdly, I tried Python with GDAL library. that was better but not the best. because there were still some problem such as shortage of built-in functions (i.e., dealing with file system and network security, complex file naming conventions etc. )
4. Fourthly, i tried to develop some Procedure using oracle spatial inside the oracle database/Geomatica. that’s effective, convenient but not the fastest because it has its own format of processing (i.e., *.pix) and that’s time-consuming.
5. Finally, i have found THE best solution which is simplest (70 line of code), fastest (processing required only 15 min), convenient (you can save the *.sh file and run it any time for any kind of mentioned data), time-saving (helps to stop re-doing the task over and over again) and effective (my boss got exactly what he wanted, before the dead-line…). the “Shell Scripting” is the solution. write just 40 line of basic code and that will do the whole (described above) task for you.
Shell Scripting for Geospatial data Processing/analysis:
Let me give you some example what i have used to solve the above problem. Do you want to convert all the different format/ projection/ transformation/ conversion into a common format/ projection/ transformation/ conversion or clip the raster based on *.shp file or some defined extent? use the GDAL utility inside the shell script (gdalwarp and gdal_translate) and these procedures is really awesome because these do not avoid the null values/No Data values in raster analysis/computation and so. Then, if you need to retrieve some averaged pixel value according to some boundary the solution might be using the ‘gdalinfo’ utility and that will create an *.xml/*.info file with the necessary statistics and after one can simply use the ‘grap’ or ‘xpath’ command to extract the necessary statistics from the *.xml/*.info file and print the output according to the boundary. if you need to do some mathematical operation go for ‘gdal_calc.py’. there are many other interesting actions which will make your problem easier and let you go bed early.
Then again, if you are dealing with temporal data then you might want to name the files with time stamps. If that’s the case, you can easily do that with some shell command like ‘rename’ or ‘sed’. Besides, ‘awk’ command can be very useful to find a matching name in a spread-sheet and extract the data from the spread-sheet and use that extracted value to rename the file/folder name. I also have found that the ‘date’ command can be very useful to convert date from different format.
All the tasks discussed above can be formatted separately and then if we are satisfied with the certain solution we can merge them together in a single *.sh script. next time you just put the retrieved data (from different source) in a $source folder and you will get the expected data in a $destination folder. So, your machine can think like you and work faster than you if you train and teach it.
Did you like this post? Read more and subscribe to our monthly newsletter!
#
Next article
GPS Terrorism. Image Courtesy: Fox News
Global Positioning System has slowly evolved to the point where it is the most widely used positioning system in the world. Aircrafts, Ships, Missiles, Cars and UAVs rely on the positioning provided by this system to make their next move. In a world where GPS is vital and in fact the most revolutionary positioning method since the days of “celestial object” based navigation, is the system fool proof? Can it sustain against attempts to sabotage its service and mislead people and machines?
Unfortunately the answer is a BIG NO! GPS Jamming is a real threat and its really very easy to deny GPS positioning. But it is pale in comparison with to the effect and destruction that GPS Spoofing can cause. GPS Spoofing took centre stage when Dr. Humphreys, Professor at University of Texas and his team built a GPS spoofer to misguide an UAV. The experiment made the Department of Homeland Security, USA to invite Dr. Humphreys and his team to showcase their latest findings on GPS spoofing and the real gravity of the threat.
Dr. Humphreys has done it again; his team of GPS experts from the University of Texas took control of the navigation system ob board an $80 million super yacht in the Mediterranean just using a laptop, a small antenna and an electronic GPS “spoofer” built for $3,000. Clearly proving that the GPS flaw could let terrorists hack ships and planes.
Dr. Humphreys gave a wonderful talk on TED regarding GPS spoofing and its threat. Definitely worth a watch
The worst thing is that GPS Spoofing can go undetected. Your navigation system will not find anything amiss until the fateful moment. Consider the case of an UAV that is scheduled to launch a missile; GPS spoofing can not only alter the direction of the UAV but potentially cause destruction by rigging the navigation of the missile as well. The military has it easier as they can use the encrypted signals which make it harder to be spoofed.
It gets even more frightening to note that there is no real defense for civilian aircrafts if such an attack on the navigation system is to take place as encrypted signals are not something they have access to. The GPS signals used by the civilian aircrafts are like a WiFi network without any kind of password or encryption which essentially allows anyone to latch on and communicate/control your PC or Laptop that’s connected to the WiFi. The way forward to prevent such spoofing attempts is still very much a research topic but on the face of it, encrypted signals will solve the issue. But the real problem is, it’s almost impossible to provide the encryption to everyone as it defeats the purpose of encrypting the signal. As we know, the L2 signals sent by the GPS satellites are encrypted and only the military can utilize them but I have my doubts regarding the efficiency of positioning when only L2 signals are used.
The answer to the question – gps spoofing and jamming: How grave? Really not good!
Did you like this post? Read more and subscribe to our monthly newsletter!
