Geopdf files in OruxMaps

[Off-track]

OK, I have started a new thread because keithy and wildwanderer are contributing so much useful information on this topic and I am sure interest will continue way beyond the original topic of the other thread at http://bushwalk.com/forum/viewtopic.php?f=21&t=25950.

To summarise:

You need the latest oruxmaps version 7.017 for geopdf to work.

NSW etopo maps can be downloaded from http://maps.six.nsw.gov.au/etopo.html but currently give blinking problems for some users (including me). They look identical in style and resolution to the downloaded wms rasters from six.

QLD geopdf maps can be downloaded from http://qtopo.dnrm.qld.gov.au/mobile/ (then follow: Click here to ... > Download a standard map sheet > 1:25,000 > Next > Search By (choose either Name or Number) > Next > enter your map Name or Number > Download Map). They work without blinking for me. They are higher in resolution than the downloaded wms rasters from qtopo.

An advantage over wms is that when geopdf files work well they fill the whole screen at any zoom level.

A disadvantage for geopdf maps that are standard mapsheets is that they include margin information of variable width, and they are georeferenced across the whole image including the map border. Therefore, when you come to the edge of the map you will see margin info rather than the adjacent map unless you manually select the adjacent map in OM. There may be workarounds or alternatives such as cropping and saving with geopdf references, or downloading the map areas without sheet margins - but I have not worked them out.

keithy has explained some file conversion options as workarounds for problem files, and I believe that Orux is continuing to tweak geopdf capability.

I am not sure how total size will compare with wms downloads at the same resolution, but this may not be a problem if you have a suitably large SD card.

I can see that there might be a lot of geopdf sheets with not very useful names, so avid users might find it useful to have a separate subdirectory for them in OM (under mapfiles), and to rename for easy recognition (like TyalgumQLD_9541-31.pdf instead of QTopo_9541-31.pdf).

Perhaps other forum users will post download links or usage tips.

[michael_p]

You might want to have a read through this thread about NSW Etopo maps: viewtopic.php?f=36&t=25020 . The post by mholling has info on how to trim geopdf images using gdalwarp.

Cheers,
Michael.

[wildwanderer]

In regard to the GEO PDF flashing/blinking issue with NSW topos. I suspect this may be device related perhaps to do with available memory or processor speed.

I have no flashing/blinking issues using a late model android phone.

I did try the Pottesville topo that was causing you issues off-track and it loaded fine for me and also transitioned to the neighbouring BURRINGBAR topo with no issues.

[Off-track]

Thanks for looking wildwanderer. I don't think it is memory as my phone has 2.37 GB free user ROM and more than 20 GB free on a class 10 SD card formatted as emulated internal memory. It may be processing power as my (cheap) ZTE A462 has a mere 1.1GHz quad core Qualcomm Snapdragon CPU with 1GB RAM + 8GB ROM.

I notice that the QLD maps (which display without problems for me) are pure raster and were produced in ArcMap, whereas the NSW maps (which give blink problems for me) have some vector layers and were produced in ArcCatalog. The NSW maps also confuse Nitro PDF viewer, which I used to see these things.

I think the transition problem with standard mapsheets will be most obvious going from West to East, so that you run into the wide margin of the mapsheet to the right of the map area. Certainly I run into the margins and have to manually select the adjoining mapsheet to see map there, even with the QLD georeferenced pdf maps that don't blink off for me.

Thanks michael_p. The post you mentioned also says that the margins around mapsheets will be a problem. I guess that is unless OM is smart enough to use neatline data embedded in the georeferenced pdf files - which it does not seem to do for me at least?? Did you get the extraction of the map area to a georeferenced pdf working with gdalwarp? If so, would you be willing to post the method?

The post you mentioned says there may be a way, but it leads eventually to a dead link in nabble. I can see that there might be a method, but I was scared off by talk of experimental pythonscripts etc. So if someone has it working and is willing to share the method (stepwise for dummies like me) that would be great.

[michael_p]

Did you get the extraction of the map area to a georeferenced pdf working with gdalwarp? If so, would you be willing to post the method?

No I haven't yet. Been occupied by other projects and this thing called life keeps getting in the way.

Cheers,
Michael.

[Off-track]

I know the feeling michael_p.

From what I can see, some geospatial pdfs have a neatline entry (e.g. Qtopo Lamington) whereas other do not (e.g. Qtopo Tyalgum) and others have a neatline spec that is either inaccurate or in a different projection to the associated map (e.g. http://lists.osgeo.org/pipermail/gdal-dev/2013-January/035269.html, https://2016.foss4g-na.org/sites/default/files/slides/using-gdal-to-translate-us-topo-geopdf.pdf), https://gis.stackexchange.com/questions/63105/gdalwarp-not-clipping-neatline-properly).

In that case, it seems to me that: (i) if OM switches between such maps without being confused by the map margins it uses some very neat programming and (ii) it is a pain in the **** for mortals like me to arrange cropping of such sheets to the map area in either gdal (command prompt) or qgis (GUI).

Also, wildwanderer may be right about memory and processing power as the NSW geospatial pfdfs hang my Win7 netbook when I try to open all the vector layers in qgis. The QLD (raster) geospatial pfdfs open just fine (albeit slowly). So complex vector geospatial pdfs under Android may only be for those with big processors (and/or lots of ROM).

For now I am content with the wms maps for NSW. While I would love to have a seamless mosaic of QLD geospatial pdfs (for their higher resolution and full-screen viewing) it is too hard to implement. But someone else may have a solution.

[Off-track]

I take it all back; QLD (QTopo raster) Tyalgum pdf does have a neatline. For some reason it is not revealed by qgis map properties, but is revealed by gdalinfo. And at high zoom, NSW (e-Topo vector) pdfs are higher resolution than the NSW wms maps downloaded at zoom 16. Of course, you could download NSW wms at higher zoom (up to 21). Qtopo currently offers wms to a maximum of zoom 15, wheres the QTopo geo-referenced pdfs are higher resolution (they look to be at least equivalent to zoom 16). Each zoom step has about x4 resolution (and file size). You will have to decide whether the performance hit (memory use for either raster or vector, and loading time especially for vector) justifies the resolution gain.

Also, in support of the experience reported by wildwanderer and keithy, Orux advises that in OM "Although you see the borders, the app uses the neatline as the boundary of the map, so it must load another new map by passing those limits. But I have find a lot of maps with a neatline outside the map limits."

QLD (QTopo raster) pdfs have the neatline right on the map boundary, whereas NSW (e-Topo vector) pdfs have the neatline out further, including the grid coordinates in the map margin. In either case, there is little to be gained by cropping the mapsheets. Geo-referenced pds will transition in OM as the neatline crosses the screen centre point. There is no way in OM to set up a true mosaic at the intersection of pdf mapsheets. Only one pdf map can be displayed at any one time.

In contrast, a wms topo map is a true mosaic. Although the NSW wms map (but not the QLD wms map) displays gridlines, you can not go to the margin to see grid coordinates. At some zoom levels (probably less than you use for hiking, but maybe in the range you use for driving) the wms map display will be smaller than the phone screen. If this is important you might want to experiment with wms tile size in OM from v 7.0.18.

Given plenty of sd card capacity, you can load both pdf and wms (and OpenAndro) offline map types, and choose what to view in OM (using Switch map or Switch map here) at any particular time and place. There is a time penalty when loading OM (and generating maps database) if it needs to check lots of offline pdf maps.

Geo-referenced pdf display is much more capable in OM v 7.0.18 beta3 (and no doubt in subsequent stable releases). For example, NSW (e-Topo vector) pdfs now display without blinking on my phone (ZTE A462), although loading is much slower than the zoom 16 wms map. Currently a geometry error affects display and transition of QLD (QTopo) pdfs, but hopefully that will soon be fixed. For more information, see the OM forum http://www.oruxmaps.com/foro/viewtopic.php?f=8&t=4504

[Off-track]

In case of maps (like QTopo wms) that lack a UTM grid, you can load one as a kml overlay in OM. Probably the easiest way is to obtain the UTM grid for your zone of interest as a shapefile from http://earth-info.nga.mil/GandG/coordsys/grids/utm_1km_polyline_dloads.html, then convert a selected region within it to KML using GDAL. For example, starting with the shp file for the 56SJ zone: ogr2ogr -f kml -clipsrc 152.9 -28.5 153.4 -28.0 UTM.kml UTM_1km_56S_J.shp.

Load the resulting KML into your OM overlay folder, and you can display it using Map {icon} > Overlay options > Load KML Overlay > select the overlay of interest (blue square with tick) > OK.

You should specify an appropriate region (using option -clipsrc and the coordinate system of the source data, EPSG:6326 = WGS84 {USA} projected in LL degrees in this case) or you will generate a very large kml file for an entire UTM zone, which will hang most programs including OM. I tried a region of 5 x 5 degrees which gave a kml file of 695 KB and was workable though a bit slow to load in OM. It aligns perfectly with the UTM grid built into SIX (NSW) wms. You can get appropriate coordinates for your area of interest using GE.

Those so inclined can change properties of displayed lines by editing the kml file. This is easiest to accomplish in GE, at the cost of some increase in file size (right-click the layer in the Places sidebar for > Properties and > Save Place As). It can also be done in a text editor. The most obvious effects would be to change line colour, thickness or transparency; e.g. from from red (ff0000ff) to green (7f00ff00) in <LineStyle>.

You could also set up such a grid for a smaller area by (i) specifying a smaller region in the command line above, (ii) using a program like QGIS, or (iii) copying from the worldwide UTM grid (Gridlines_utm.kml) from http://www.nearby.org.uk shown in appropriately zoomed GE, and saved to My places, then saving the displayed 10 km and 1 km gridline information (and optionally the labels) as kml or kmz files.

[Rodo]

Can anyone help with how to load a NSW Top pdf offline map into Oruxmaps? I have done this a few years ago but now cannot seems to get it working.

What I have done so far:
1. Go to: http://maps.six.nsw.gov.au/etopo.html
2. Search for required map and then download it. e.g., http://maps.six.nsw.gov.au/etopo/geopdf ... ACKING.pdf
3. After file has been downloaded move it into the mapfiles folder under Oruxmaps.
4. Then when I start Oruxmaps the following messages are displayed - 'There were problems reading some maps!', 'Please 'Reset map list' in the map selector view to see a report.'
5. I then go to 'Switch map' under the wiggly map selector at the top right of the screen, then select Offline.
6. Then I press the left most button on the right of the top of the screen which looks like two arrows going around in a circle.
7. Then a box pops up saying - 'Generating maps database!'.
8. Then another box pops up saying - 'Problems with some maps: /storage/emulated/0/oruxmaps/mapfiles/9129-4N PORT HACKING.pdf->Can not find GEO info!'.

Anyone know what I can do to get it working?

Thank you.

[michael_p]

Tried the Port Hacking etopo you linked to on my phone and tablet with Oruxmaps and I am getting the same error. There may be a problem with the map. Don't have time to look at it tonight, hopefully I can do some investigating tomorrow. Will let you know if I find a solution.

Michael.

[keithj]

Can anyone help with how to load a NSW Top pdf offline map into Oruxmaps? I have done this a few years ago but now cannot seems to get it working.

What I have done so far:
1. Go to: http://maps.six.nsw.gov.au/etopo.html
2. Search for required map and then download it. e.g., http://maps.six.nsw.gov.au/etopo/geopdf ... ACKING.pdf

I believe the downloadable Topo PDFs are NOT georeferenced.

Anyone know what I can do to get it working?

Have a read of https://maps.six.nsw.gov.au/js/channels ... _Final.pdf and email SS-SDS@finance.nsw.gov.au with your request. It will take a day for them to process it & send you a download link.
If that email doesn't work try SS-SDS@customerservice.nsw.gov.au

[wildwanderer]

I've been using the sixmaps downloadable topos for a long time with oruxmaps without issue. I have 30+ on my phone. As far as I can tell they are geo referenced or altleast the grid references and orientation are accurate when I load it into oruxmaps

Maybe there is something wrong with that specific version of the port hacking topo?

I have the 2017 version and it works fine in oruxmaps

[Rodo]

Thanks for your help guys. I also tried the Otford map and a random one (Koorawatha) with no success. I will email them and see if they can help.

[wildwanderer]

I just downloaded a new map from sixmaps. :Tabourie 1-25000. The same behaviour as Rodo, when importing into oruxmaps.. getting an error message saying there is no geo info and the map can't be displayed in the app.

It seems spatial services has released a new map series and failed to include the georefrencing or at least the same type of georeferencing.. which had been included in all previous series..

This is what the new 2020 series looks like, I noted that it says 'auto generated'

Can anyone confirm if this issue is limited to oruxmaps or the problem is with all apps trying to use the new 2020 series sixmaps etopos?

EDIT. I just loaded it into avenza. Same problem as oruxmaps.. while the map loads it has no georeferencing.

[michael_p]

I ran gdalinfo against the Port Hacking etopo pdf that I downloaded last night and got this:

Code: Select all

gdalinfo "9129-4N PORT HACKING.pdf"
Driver: PDF/Geospatial PDF
Files: 9129-4N PORT HACKING.pdf
Size is 6767, 3484
Coordinate System is `'
Metadata:
  CREATION_DATE=D:20201029114028+10
  CREATOR=Esri ArcMap 10.6.1.9270
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 3484.0)
Upper Right ( 6767.0,    0.0)
Lower Right ( 6767.0, 3484.0)
Center      ( 3383.5, 1742.0)
Band 1 Block=6767x1 Type=Byte, ColorInterp=Red
Band 2 Block=6767x1 Type=Byte, ColorInterp=Green
Band 3 Block=6767x1 Type=Byte, ColorInterp=Blue

Running gdalinfo against an older etopo pdf (Jamison) returned the following:

Code: Select all

gdalinfo "8930-2N JAMISON.pdf"
Driver: PDF/Geospatial PDF
Files: 8930-2N JAMISON.pdf
Size is 6767, 3484
Coordinate System is:
PROJCS["GDA_1994_MGA_Zone_56",
    GEOGCS["GCS_GDA_1994",
        DATUM["Geocentric_Datum_of_Australia_1994",
            SPHEROID["GRS_1980",6378137.0,298.257222101]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",10000000.0],
    PARAMETER["Central_Meridian",153.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]
GeoTransform =
  243936.0774349454, 4.232208796725235, 0.1079758749974037
  6262520.787086879, 0.1076744821412777, -4.232683836413364
Metadata:
  CREATION_DATE=D:20171020132244+10
  CREATOR=Esri ArcMap 10.3.0.4322
  NEATLINE=POLYGON ((244087.729819606 6248020.5189889,244088.12044062 6263041.61680287,269957.185806656 6263040.64787558,269957.807964339 6248020.70807781,244087.729819606 6248020.5189889))
Corner Coordinates:
Upper Left  (  243936.077, 6262520.787) (150d14' 8.81"E, 33d44'47.31"S)
Lower Left  (  244312.265, 6247774.117) (150d14' 8.03"E, 33d52'45.89"S)
Upper Right (  272575.434, 6263249.420) (150d32'41.52"E, 33d44'47.20"S)
Lower Right (  272951.622, 6248502.750) (150d32'42.46"E, 33d52'45.86"S)
Center      (  258443.850, 6255511.768) (150d23'25.19"E, 33d48'46.92"S)
Band 1 Block=6767x1 Type=Byte, ColorInterp=Red
Band 2 Block=6767x1 Type=Byte, ColorInterp=Green
Band 3 Block=6767x1 Type=Byte, ColorInterp=Blue


In the Port Hacking pdf the Coordinate System line has a tick instead of a colon.

Code: Select all

Coordinate System is `'

Makes me wonder if the creation script has a typo and the "Coordinate System" metadata is being skipped.

[michael_p]

Also, I was able to use gdal_translate to extract the geotiff file from the pdf. The geotiff seems to work fine it appears to just be a problem with the pdf wrapper.

[Rodo]

I emailed Spatial Services and received a prompt reply. They provided me with a link to download the Port Hacking map. The map now works in Oruxmaps.

They mentioned that the GeoPDF Topographic maps on their website are not georeferenced. They are working on replacing these maps with georeferenced products on their map store in the future.

If anyone needs any georeferenced maps then they will need to email Spatial Services (SS-SDS@finance.nsw.gov.au) and ask them for the required maps which currently are the 2017 versions.

[wildwanderer]

Thanks Rodo. I have now been advised by NSW Spatial Services that this issue has been resolved. Reposting below from the sticky on the NSW forum.

After correspondence with NSW Spatial Services today, they have promptly acted to rectify this issue. The https://six.nsw.gov.au/etopo catalogue has now changed back to the 2017 series which features georeferencing.
Spatial Services have advised that the 2017 maps are now online at https://six.nsw.gov.au/etopo. The 2017 version should work as intended with common phone nav apps such as Oruxmaps and Avenza.

If you have a 2020 series map you may wish to delete it and download the 2017 series in time for your next walk. This series is identified by 2017 on the cover.
Thank you to NSW Spatial Services for their very quick response and action.

[michael_p]

Thanks Wildwanderer for your efforts. Hopefully NSW Spatial Services wont make this mistake again.