Sunday, 18 March 2018

Otago Central Railway [61B]: Finishing Maps - Clyde 1

So while I am working on the Addington mosaic, that is a pretty slow process with a lot of downtime waiting for Gimp to resize images, the best part is with another computer doing only that and nothing else, I can carry on with maps on my main computer. And in actuality that is a dedicated VM which is where I do most of my mapping on different computers, because I can use multiple versions of the software, which I need to at the moment because of a serious bug in Qgis 3.0 affecting aerial photos. I need to use 2.18 to do most editing but I need 2.14 to do filtered layers editing because 2.18 deprecates that capability. 

So back into Central Otago and starting with Clyde and here is the 1976 aerial photography with everything else over the top of it.

Now this is a multi generational map that covers 1962, 1976, 1981 and later up to the present day. To make such maps work we have to be able to filter what is being displayed so that only the features like tracks, buildings, stations and so on for that timeframe are displayed. To achieve that I have extra fields for dates added to a lot of layers and run the same filter query on multiple layers at once and at the same time select the background aerial photography layers to be displayed and that is how we get to multi generational maps.

So Clyde is the westernmost area that is set up as multi generational, now I have to tidy all that up and push out the generations of maps for Clyde and then work my way eastward doing the same all the way along and just make sure all the stuff on the Google Plus collections is all nicely up to date and that we have maps up there for the entire line.

That will go on at the same time as Addington mosaics simply because it is an efficient use of time with the delays for making the mosaics while the computer does the hard work on those huge Gimp projects to make the aerial images for that part of Christchurch. I have almost completed Addington West now, I just have to mask off the area that I want from the Retrolens aerials and then save back the geojpegs for Qgis, probably finish that tonight. Then Addington North and Addington East to start on.

Saturday, 17 March 2018

Main North Line [11B]: Addington 2

The Addington mosaic has been a lot of fun. Working with so many large layers in one image is taxing on computer resources, and one relatively fast PC (its CPU is a Pentium G with 2 cores but no hyperthreading) with nothing else open and 16 GB of RAM is still quite slow working with what is now a 10 GB Gimp project and which keeps growing as the historic aerial layers have to be scaled up in size. Since I ended up with two large layers (scaled from around 9000 to around 15000 width) on the west side of Addington with a small slice in the middle where another layer has to be pasted in to fill the gap, I resorted as in the past to cropping the required piece out of the gap layer, otherwise it takes all day to get it scaled to the right size.

The idea of running Gimp side by side on two computers in order to get the smallest number of steps to maximise quality has been useful but the scanned images in this case are so sharp (the original scale is 1:3400) that it's been unnecessary to continue with this. At the same time it is hard to get much memory available on MainPC for this kind of thing because browsers and other stuff I do a lot of on it just seem to gobble up all the available memory. Because of this I put Qgis 2.18 onto mediapc as well and have been using that because the media player stuff doesn't use much RAM. This shows the great benefits of an OS like Linux which is designed to work with resource limited hardware, being resource efficient on good hardware that you might be using do do resource intensive tasks. 

As it turned out my strategy outlined above did not work and I have changed to using two big layers next to each other with a small slice at one end instead of in the middle, this is because the usual skewings and distortion are better seen at one end rather than in the middle of a critical area of the railway depot. However this means a lot of yesterday's work has had to be redone. This sort of outlines why only a few areas are going to get this aerial photography coverage - it takes a huge amount of time to create these mosaics especially for big areas like a major railway yard or depot. The small stations on the Otago Central line which usually only involve one or two base images and maybe the same number of aerials are child's play by comparison. In this case having a slice on the end is much simpler because it can simply be cropped off when the mosaic georeferenced copies of the original base images are saved, so the full size image can be used instead of cropping a piece of it. We will have to see if Gimp copes with all the historic images once they are being displayed and it may well prove necessary to split the project into two or three pieces.

Thursday, 15 March 2018

Main North Line [11A]: Addington, etc

Having completed sorting through aerial photos of the MNL and now having a complete set, the next task is to start doing stuff with the photos. The first thing that will happen is I will do a set of historical maps for Addington, as that is where the MNL starts. When I first started looking at Christchurch yards about three years or so ago I had Canterbury Maps coverage and I used to overlay it in Google Earth and then import the lines and shapes into Qgis. Even though those old layers still exist, I am going to redo it all from scratch and create mosaics for all the stations on top of current Linz coverage, because I can use the mosaic aerials when I publish the maps.

So for part of today I will have two computers both running Gimp to create the mosaics of Addington, which is both a MSL and a MNL station, being a junction. It is a very interesting location that like many other areas of Christchurch has seen huge change over the past 40 years. Doing two Gimps side by side is basically having one of them resizing and rotating as quickly and roughly as possible, and then the second doing the absolute minimum number of these operations to get a result that is as sharp and clear as possible, because every such operation reduces the final quality. 

I haven't forgotten the Otago Central Line but have just needed a break from it over the past few weeks but I will be pushing on with it today as well. I have to reorganise the aerial photography resources for it as well, and that will take some time to complete. But most of it is just editing the project files to find the new folder paths where the aerial photo collections have been moved onto a different server, and shouldn't actually take too long. 

The optimisation / VRT preparation script is going to be started fairly soon, maybe at the weekend, because I need to get on to the task of both making my life easier with fewer layers in VRTs, and reducing the number of images I need to keep copies of on the computer, because they are using up so much disk space. The Volume 10 resources are using 200 GB, Volume 11 we are only just starting on is 70 GB, and Volume 12 is 90 GB. However, an earlier series, Volume 7, is well optimised and only uses 3 GB. Volume 10 and 11 are expected to use a lot of space because the MNL and MSL both cover a lot of territory in Canterbury and pass through major urban areas. But I can't sustain the amount of space that some of these volumes are taking up because it fills up the disk and makes the backups too big as well. So this rationalisation improvement really has to happen, but it means more delays that will take me off mapmaking for a while.

Friday, 9 March 2018

Main North Line [10]: Aerial Photography

Most of the mapwork over the past few days has been with MNL aerial photography which has to be downloaded and then sorted through to pick up just the photos I want. This is a slow process because of the download limit of 3.5 GB per download from the Linz website and the number of tiles in each download being much greater than what is needed for the rail corridor. So there has been a lot of work to get full aerial coverage for this line.

The other thing that has to be done at the moment is to amalgamate aerial photos and remove the ones that are not actually used because it takes up a lot of disk space and I have had to reorganise the storage between between different computers. I am looking at writing a script to process the Qgis layer definition files that I am putting aerial photos into. I should learn how to script on Linux with Python but to speed things up, in this instance I will use Windows Powershell to process the QLR files and copy the photos listed in them to a specific folder path that the script will create. What happens after this is to use a Qgis library function to process the aerial images into something called a VRT (Virtual Raster Transform) which is supposed to make it easier to handle all the many aerial photos I am working with.

The aerial photo downloads are now more or less finished and just need a bit of extra little bits added on here and there to make them complete. I need to get on also and finish the Otago Central maps because I haven't done any work on them for the last week or two. I think I just did some mosaics of Auripo and haven't actually drawn any of the maps; nor have I worked on any of the other maps. However reorganising the resources on the computers has taken a bit of my time up lately.

I have been getting Retrolens coverage for the major deviations on the MNL as well as the current stuff and have got enough to map all of these accurately. Apart from Nonoti mentioned last time there were major ones done at Stewarts Gully, Ashley, Balcairn and Phoebe, and all of these will be addressed over further posts in the MNL series in coming weeks.

Tuesday, 6 March 2018

Main North Line [9]: Nonoti

Nonoti is a weirdly named little station in North Canterbury between Tormore and Mina. The exact distance is not recorded in the NZ Railway and Tramway Atlas, but it is between 116 and 117 km. The station closed in 1958. Around Nonoti a deviation was put in during the 1950s, apparently commissioned around 1961 (Don Spicer). 

1955 Aerial photo showing the earthworks underway.

1965: the deviation is in use and, of note, the original overhead bridge at Nonoti was replaced with another one on a slightly easier alignment and because the rail route at this area also changed. Nonoti station still had some track in place despite having shut some seven years earlier.

Complete map view.

Although drivers (e.g. Don Spicer) say the new route is much easier to work, eliminating gradients and sharp curves, the new route still requires speed reduction to 65 km/h for the curves. It is interesting that more effort was not made to ease these.

Monday, 5 March 2018

Main North Line [8]: Full aerial coverage

During the migration to Qgis 3 I have started migrating the CWNM project to this edition and it needs a bit of work to fix up various things. Alongside this I have been doing a lot to get a full set of aerial photos for every line, although particularly the Main North Line at present. 

Alongside this is a reorganisation of resources on the different computers. The ongoing issue with aerial photos has been the large amount of disk space they consume. This creates a lot of problems on mainpc as it keeps filling up the 2 TB array.

I have therefore taken a logical step to move all the aerial photos used for maps off mainpc onto serverpc. This is going to take a while to accomplish although because many of the aerial layers are not actually loaded most of the time it will not break many projects when they get moved across.

Because Qgis 3 implements a new canvas caching function that consumes a lot more resources than previous versions I am moving towards all aerial photos being loaded "on demand" in effect by implementing layer definition files saved in each source folder so that only the aerial photos currently in use actually need to be loaded. At the same time if I forget to turn on serverpc when the project is opened, reloading a qlr file will be quite easy to do.

The core custom aerial maps for each project which are quite a small number of files will probably be kept on mainpc but I will have a look into this in more detail as I progress the migration. Hopefully it will be a long time before I have to clean up any resources on serverpc.

I have changed my mind  about rushing to migrate to Qgis 3 as it still contains significant bugs. One in particular will have a major impact on aerial photography use as it gobbles up all the resources really quickly for some sort of render caching and then freezes. So back into 2.18 again. I have decided to make the most of using serverpc to run map drawing VMs and therefore am working on an optimal keyboard layout at my workstation that lets me put serverpc's keyboard directly in front of me while keeping mainpc's keyboard in the same place, therefore allowing me to use both computers simultaneously for mapwork without switching the main keyboard between computers all the time. 

Otago Central Railway [61A]: Finishing Maps

There has been quite a bit of progress with maps since the last blog post. I am just not blogging as often lately, but I will start to pick that up again. 

I have mapped the entire TGR line with all the detail available including full and half km posts and other details I can get off Youtube videos which people have done with cameras mounted to the front of locomotives. These maps will soon be published to the Google Plus collections.

The next step already mentioned is to systematically work down from Part 1 to Part 3 from Cromwell back to Middlemarch to finish all the maps throughout and update all the Google Plus collections in turn. There is quite a lot of mucking around here with particularly Alexandra and Clyde generational maps to be finished. There is also the map of Auripo in Part 3 which I think is the only station not finished therein, and various bits in Part 2. The Part 1 maps need the most work.

This has all had to be fitted around migrating to Qgis 3 which was released two weeks ago. I have spent a lot of time setting up a third computer to hold all the VMs I use to free up disk space on mainpc, and have been putting together several VMs that will be on mainpc because they will be for older versions that are archived and that need to be included in the backups, such as 2.14 which has some extra functionality not available in later editions. 3.0 is natively installed on MainPC and development builds are all on the serverpc collection of VMs.