Luxology Forum > Geo Kit (Updated on 2012-1203)

xtof

+8 signal

Belgium
304 posts

My first python script: getElevation & backdrop, matured into a kit. In glorious camel case :)

Important: you need to have a full python 2(.6.6) install along with the PIL module. Look here to get a build for PIL. Or it won't work. Mac users, try your luck with this Mac build.

I strongly suggest reading through the posts in this thread for support and more info to get you going.

Update 2012-1202: Added Takumi's routine that takes care of siteDir and PIL troubles. If you have Python installed in a different directory than the one listed in the siteDir variable in the geoLauncher script, Modo will ask you to locate the PIL.pth file. You should browse to the location and select that file. Smooth sailing afterwards, hopefully. a BIG thank you to Takumi san for helping out.

Also added a checking routine that will warn you if the mesh layer is empty and it'll exit gracefully.

geoKit 2012-1203

Update 2012-1130: It's a kit! And rewrote it completely too. By default it'll submit multiple locations per query which will allow a user to sample more elevation data on a daily basis. Added a 2 sec delay between each query by default so a user doesn't run into trouble right away. Support for vert per vert queries is still there, but the button (shift click) doesn't trigger that function yet. Same with polyline submission (control click.) Don't think there is much use for polyline anyway and it is even less accurate.

And for the backdrop part, you'll now have an option for map type too. Default is the sexy satellite style.

Still some work to do, but it works fine in most cases. If you look at the projection module I wrote, you can see it'll allow you to use a different map source for the backdrop. The functions will allow you to switch to different "coordinate" spaces if you will. Bing is certainly an option. Feel free to experiment! :)

Acces the kit by clicking on the little Luxorange globe button on the toolbar.

geoKit 2012-1130

Check the path in the geoLauncher.py file and change it if needs be. Right now, it is looking at:

siteDir = "C:\\Python26\\Lib\\site-packages"

Update v1.61: Added proper error code status parsing--thanks Colas!!! :)--and figured out how long to wait if Google is bitching about too many queries, too fast. If you find yourself waiting for half a minute and you don't see a progress bar (or it doesn't move,) you're probably out of luck for the remainder of the day. Try again later or the next day. You'll also get an error window confirming the error. Other critical errors are being dealt with and if needs be, inform the user. Both progress bars now work too!

And this wraps it up. Feel free to play around and add functions. Support for multiple locations per query and polyline submission would be nice to have :)

getElevation v1.61

Update v1.55: Added a controller for the map's zoom level and a function to estimate the resolution you'll be trying to create. That way you can see if you're overdoing it or not :)
Also added some things behind the scenes, to handle the exceptions.

getElevation v1.55

Update v1.52: If you decide on having an image in the backdrop, click on the get backdrop button. If you want elevation, click on that one. Two different operations. The image will be added to Modo and set accordingly. Sorry for screwing up the version numbers earlier today.

getElevation v1.54

Update v1.52: You can now specify a filename/type and location for the backdrop image. Also cleaned up the script and made some functions. This is getting somewhere now... Myeah, cheated on the update date :) Be sure to replace all the config files. Maybe you need to trash your Modo config too. Or else delete the getElevation entries if you know what your're doing.

getElevation v1.52

Update v1.44: Added a texture map retrieval routine. It will save a PNG to "C:\Temp\" so make sure you have that folder. Not sure about Mac users, but I'll add a filerequester later on so you can specify path and filename. The filename is always something like "-47.223,3.4555.png" for now.

Be very careful not to overdo it. Currently I stick to creating sectors that are 1km² with a total of 1089 verts, and process those as I go along. The texture will be ~2500x if you stick to those dimensions.

getElevation v1.44

Update v1.36: Added scene scale and an OSM map export toggle (it is off by default.)

getElevation v1.36

Update v1.34: Sensor part in the URL construction didn't work (anymore?) switched back to using a "&" instead of "%26"--Google would throw an error and no elevation was being set. Fixed.

getElevation v1.34

Update v1.33: you can now enter coordinates. Go to the Mesh item properties and you'll see a "Get Elevation" entry at the bottom. Keep in mind that you need to use a "." and not a "," for the decimal coordinates. Click on the button to launch the query.

What this script does is query Google maps to get elevation values returned for each selected vert in modo. Or all, if you haven't selected any.

This is the first public release and the decimal coordinates for Modo's origin are hardcoded in the script. If you want to sample another location, you should change the values in the script itself (search for "sceneLat =" and "sceneLong =".) You can use Google maps for that by RMB'ing on a spot on the map and selecting "what's here?"

So the idea is that you establish the coordinates for the origin (exactly the same as the physical sun feature,) and then you can draw planes all over the place with as many subdivisions as you need. Select verts to translate and run the script. It is now located somewhere in the Grand Canyon.

I'd suggest to use a plane which is 3 km on X and Z with 21 subdivs on each axis max, nicely centered around the origin.

Keep in mind that it can take a while (minutes) if you sample many points, so start of with something like 200 or so. The script has a error checking loop inside to keep trying until it gets a value returned. No cancel button or progress indicator yet.

From time to time you can also get a spike in your mesh (google hickup?) so just select those verts and run the script again to fix'm.

Also, when done, it will launch three urls. One to get a satellite image of the area covered, one to Open Street Map with a bounding box and one to directly get a png of the exact area. You can use the latter as a texture or backdrop (set it to fit.) Problem is that I still need to figure out a scale setting routine for a png export that sets the scale to something valid. Meaning that you might find yourself in a situation where OSM says it is too big.

So this is all clearly under heavy construction, but it works and maybe somebody out there will find it useful. I'd like to create a form for this script so you can type in geo locations and check if you'd like to have map images or not. Originally I wanted to use the physical sun geo feature but it turns out it misses a few digits to be accurate enough for this sort of thing. Bummer.

If you're a coder, do have a look and tell me where I can clean up things or make it more efficient. I started this project so I can learn a trick or two.

Finally, a big thank you goes out to my buddy Colas Fiszman for providing me with the getElevation function and the time loop trick. And for answering my questions. :)

Let's see where this will bring us...

Message edited by xtof on 12/3/2012 - 7:02 AM

25-Oct-2012 9:36 AM

Eterea

0

Zaragoza, Spain
958 posts

Thanks for sharing, xtof! :-)

Cristóbal Vila
etereaestudios.com

25-Oct-2012 10:35 AM

xtof

0

Belgium
304 posts

My pleasure. Use it with moderation :) Btw, the google server can be busy sometimes and the script throws an error. Currently don't know how to fix this. Error says "name 'elevation' is not defined."

Am dumping messages to the console window during the loop, expecting it to be printed each time a vert is processed but I only get something when the loop is finished completely and then all vert related info is printed. Why is that? And how I can I fix this?

Message edited by xtof on 10/25/2012 - 12:19 PM

25-Oct-2012 11:54 AM

Takumi

0

Tokyo, Japan
109 posts

Quote from xtof :
Btw, the google server can be busy sometimes and the script throws an error. Currently don't know how to fix this. Error says "name 'elevation' is not defined."

Did you check an Elevation status code before getting a value?

- OK indicating the API request was successful.
- INVALID_REQUEST indicating the API request was malformed.
- OVER_QUERY_LIMIT indicating the requestor has exceeded quota.
- REQUEST_DENIED indicating the API did not complete the request, likely - - because the requestor failed to include a valid sensor parameter.
- UNKNOWN_ERROR indicating an unknown error.

25-Oct-2012 11:32 PM

xtof

0

Belgium
304 posts

Hello Takumi,

No, currently not. It only keeps on trying if it is getting an error returned, with a time delay that gets bigger each try. Definitely need to look into error code handling. But it is a weird beast, google maps. While I'm not getting a value returned using the API from within Modo, I do get values if I use a website that uses the same API. Strange. And I hadn't exceeded the max amount of queries yet.

So far didn't manage to get a KeyboardInterrupt exception working. Ctrl+C doesn't do anything. Isn't this supported perhaps? The code should work as I added a try / except (KeyboardInterrupt, SystemExit) before entering the loop.

Thanks for your feedback, Takumi!

26-Oct-2012 1:02 AM

xtof

0

Belgium
304 posts

Changing the code, so I'll put an update online later today. Sensor part is wrong, I think, but I'm testing it some more. And I'm also waiting for a reply just to be sure.

Re the dynamically printing to the Modo console while being in a loop, would flushing resolve this? "need to flush sys.stdout or it won't print anything until after the loop has finished"

Quote is coming from here.

Have been playing around with that, but so far it doesn't work, probably due to something I'm doing wrong.

28-Oct-2012 3:40 AM

xtof

0

Belgium
304 posts

So I have a progress bar working, and a cancel button--thanks Lux'--but I still need to figure out how to intercept a particular status error code... :/

Message edited by xtof on 10/28/2012 - 4:30 PM

28-Oct-2012 4:27 PM

klanderud

0

Japan and USA
754 posts

Very Cool! seem to work out fine at first, then I couldn't get it to work after line 67 name 'elevation' is not define

Message edited by klanderud on 10/29/2012 - 4:19 AM

29-Oct-2012 2:56 AM

xtof

+1 signal

Belgium
304 posts

Hey klanderud. Try the new one, should be fixed. Ironed out most of the glitches except one: if you exceed the daily limit the script will not know it is being banned. In that case you will have to wait for 24 hours and try again. I believe that the daily limit for a regular user is 2500 queries (verts in our case.)

Code to intercept is there but I can't get it to work for some reason.

So you'll have a progress bar and a cancel option. Map scale is also taken care of and if you iterate your elevation model and you're getting an error returned (behind the scenes) your mesh is no longer being "destroyed."

So if somebody could help me out with the "OVER_QUERY_LIMIT" status code, then that would be cool. This works in a python shell--without the lx commands of course--but not in Modo:

    if resultDict["status"] == "OVER_QUERY_LIMIT":
        lx.out("You exceeded the limit, retry in 24 hours")
        sys.exit()
    else:
        return float(resultDict["results"][0]["elevation"])

So apart from that, it is time to create some user input shizzle and a form.

Message edited by xtof on 10/29/2012 - 6:30 AM

29-Oct-2012 6:26 AM

Takumi

+3 signal

Tokyo, Japan
109 posts

:)

http://popover.blogzine.jp/weblog/2012/10/get-elevation-b.html

Message edited by Takumi on 10/29/2012 - 7:59 AM

29-Oct-2012 7:57 AM

xtof

0

Belgium
304 posts

Waaaaa, those images look cool, Takumi san :) I'm reading the google translated page on your blog now. Thanks for the shout out and for getting back with some nice visuals :) This if fun!!!

Message edited by xtof on 10/29/2012 - 8:24 AM

29-Oct-2012 8:19 AM

Robert Lechl

0

Munich, Germany (München)
4018 posts

awesome script. only the speed is bringing it down.. I hope you find I way to make it faster.

Beside this I get no elevation at all. the map is ok, but zero elevation in Modo.

Message edited by Robert Lechl on 10/29/2012 - 2:49 PM

29-Oct-2012 2:33 PM

xtof

0

Belgium
304 posts

You can now enter coordinates. Go to the Mesh item properties and you'll see a "Get Elevation" entry at the bottom. Keep in mind that you need to use a "." and not a "," for the decimal coordinates--might add some code to cover that part. Click on the button to launch the query.

@Robert: can't do much about the speed. In fact, the script is too fast and the google server disconnects. That's why the timer part has been added, to delay and try again, 5 times max, waiting longer each try.

Currently I'm getting no elevation data either. Definitely need to get the status code implementation running and give the user some feedback in Modo. It works pretty ok in the morning for us users in Europe. In the late afternoon and evening it gets slow.

Check the event log for more hints.

I'm quering vert by vert because, as per the google docs, it should be more accurate than querying multiple locations at once. From what I've been able to understand, Takumi switched to submitting multiple locations. Multiple locations is maybe a faster approach and you can sample more data on a daily basis. I also think that Takumi san managed to get the google maps texture to match the area covered. How cool is that? :)

In short, not sure if I can get the status code working (my last puzzle to solve.) But I think Takumi has come up with a better tool altogether. So lets wait for him to respond.

Feel like piping the resolution result to a weightmap or something. This will give visual feedback of the accuracy of your verts.

One thing, I had to switch to using strings as the input because a float lacks digits and isn't accurate enough for this sort of thing (just like geo locations for physical sun.) The script will convert it to a float number and then do the math.

Message edited by xtof on 10/29/2012 - 3:17 PM

29-Oct-2012 3:13 PM

klanderud

0

Japan and USA
754 posts

Quote from xtof :
Hey klanderud. Try the new one, should be fixed. Ironed out most of the glitches except one: if you exceed the daily limit the script will not know it is being banned. In that case you will have to wait for 24 hours and try again. I believe that the daily limit for a regular user is 2500 queries (verts in our case.)

Great :), thank you for posting new scripts. I think that must have been the case. I probably got to excited and started to do large queries trying to see how much detail I can get.

I tried to do a query recently, it went through and I also got no elevation, but I did get a image map of the area.

I hope Luxology can devote a team on this. There is a lot of data they can capture not only from terrain elevation from google, but also roads, streets, sidewalk information from Open Street Map.

29-Oct-2012 8:22 PM

klanderud

0

Japan and USA
754 posts

Quote from Takumi :
:)

http://popover.blogzine.jp/weblog/2012/10/get-elevation-b.html

Great work Takumi! Thanks for posting the Images.

29-Oct-2012 8:27 PM

xtof

0

Belgium
304 posts

Ah, strange. The ampersand in the url construction wasn't working for some reason. V1.34, tried and tested and am getting elevation data now.

Have fun, klanderud & Robert :)

And at Takumi, that Niagara Falls still on your blog looks nice!

Message edited by xtof on 10/30/2012 - 3:04 AM

30-Oct-2012 2:51 AM

xtof

0

Belgium
304 posts

Added a Scene Scale because dealing with huge distances (kilometers) in a 3D app is probably asking for trouble. Inaccuracies and render artifacts due to rounding for sure. That's for a future upload.

Might have a look at parsing the resolution info to a weightmap

This a reference scale I've managed to come up with, related to DEM detail (the lower, the more detailed/accurate):

1/9 arc-seconds ~ 3m
1/3 arc-seconds ~ 10m
1 arc-seconds ~ 30m
3 arc-seconds ~ 90m
5 arc-seconds ~ ?
30 arc-seconds ~ 1000m

So with that as a reference, you could create a weightmap that will pinpoint inaccurate elevation.

Not sure if I can pull that one off.

30-Oct-2012 5:10 AM

Robert Lechl

0

Munich, Germany (München)
4018 posts

Don't work for me at all. Get an error on starting the script ' wrong user value'

30-Oct-2012 12:24 PM

xtof

0

Belgium
304 posts

Just deleted my config file and moved my config and scripts folders. Downloaded and extracted the rar and copied those folders in the same place. Launched Modo, made a basic grid and ran the script using the Get elevation button. It works here.

What did you do exactly? It is no longer just the .py file, you need to copy the config files too. The rar contains:

Configs
   Form_getElevation.CFG
   UserValues_getElevation.CFG
Scripts
   transform_getElevation.py

Message edited by xtof on 10/30/2012 - 1:08 PM

30-Oct-2012 1:05 PM

Robert Lechl

0

Munich, Germany (München)
4018 posts

you really should do it as a kit ;-)

30-Oct-2012 4:29 PM

Robert Lechl

0

Munich, Germany (München)
4018 posts

the really strange thing with that script is, that it starts good and after a while just leave you with zero elevations on your map.
Someone should start catching large areas and posting it here.. how funny would it be to have map in 3d of a large area?

30-Oct-2012 5:17 PM

klanderud

0

Japan and USA
754 posts

I am trying to piece together the maps as a test and I am getting some overlay.
With an experiment on a 3 km grid with 21X21 I am getting a 3 polygon point overlay when tested with another grid. Which is an easy fix, just delete the 3 rows of polys.

Per degree it should be about .008985
Or for 3km it should be .026954
I thought there was about 111.3 km per Longitude or is that a round down number or maybe my math is off?
Maybe I'll try to find the exact google number if it is different, I hope it is not different on other parts of the world.
I am trying it see if I can piece Yakushima Island, Mt Miyanoura.
If One can find the exact number then one can successfully piece things together then we can start sending it to the Share site.

30-Oct-2012 10:55 PM

klanderud

0

Japan and USA
754 posts

Ok I think I got it

.030804 degree Longtitude difference for a 3km Map
Just add or minus it to your scene Longtitude for the next tile.
I think it should work for the Latitude but not sure yet.

I still get a slight difference in the edge which can be merged together if needed.

Also I find myself starting to name the mesh like
"lat30.3359 long130.5047"
"lat30.3359 long130.535504"
So it has the location in its name, so there is no confusion.

I do sometimes get spikes so a Deform>Smooth should work as well as a little hand adjustment with the move tool.

Message edited by klanderud on 10/30/2012 - 11:35 PM

30-Oct-2012 11:06 PM

klanderud

0

Japan and USA
754 posts

Here is the Longitude tile test,
titled with the .030804 degree difference.
You can notice a very slight line between the two, which can be fixed by merging, or by finding a tighter solution.

Message edited by klanderud on 10/31/2012 - 1:25 AM

31-Oct-2012 12:01 AM

xtof

0

Belgium
304 posts

Quote from Robert Lechl :
the really strange thing with that script is, that it starts good and after a while just leave you with zero elevations on your map.
Someone should start catching large areas and posting it here.. how funny would it be to have map in 3d of a large area?

I hope it is ok that I'm hotlinking Takumi's log image in here:

As you can see he added a status code report, among other things. This will tell you what is going on during the query. The script will not change the elevation if it is getting an error back. In my script this can be anything. Before yesterday I screwed up the ampersand in the url and the sensor part wasn't accepted by Google. Now, the server can either be busy--the script does a max of 6 attempts in total per vert--or you might have exceeded your daily limit of 2500 verts.

If you run the script the first time and you have a (fresh) mesh were everything on Y is 0, and there was no elevation value retrieved within those 6 attempts, the elevation will not change and in our example it will stay put. Y stays 0. Remember that you can rerun the script and finetune your mesh by selecting the verts you want to process again. No verts selected = process all. Or process selected verts only if you have any selected.

About the kit part, on my end it is part of my Sandbox kit. A playground of personal tweaks, experiments and settings. Read about too many kits possibly screwing up Modo, and chose to drop it as a regular script with a form and user values. Feel free to adjust things as you fit.

Piecing together geo should be straightforward. If you add polys to what you already have and select those new verts and run the script, only that part will be elevated. As long as you keep the same geo location throughout the process you can keep on expanding your mesh. The math applied isn't perfect, depending on the location you are sampling because the model used for the earth is a perfect sphere, while in reality the earth is an ellipsoid. The closer you get to the poles, it starts to become less accurate. At least to my knowledge. Still good enough for most purposes however. It also depends on what model is being use by Google, I think. Right? And if you have a look at the DEM reference scale listed above, it should be ok, as I think that the most detail we're currently getting (and depending on the location,) is 30 meters at best. MAYBE 10 meters in some rare cases? This is all fairly new to me to some degree, so I had to take a crash course.

Expressing latitude and longitude as linear units

You can improve the math, but it is more complicated.Somebody could create a more precise function based on the Vincenty formula, because I didn't feel up to it :)

The grid for locations, compare it to the Modo grid to understand what is going on (and why I had to inverse the Z values coming from Modo in the script.)

Matching the satellite images to our mesh in Modo would be cool to have. How do you handle it now, klanderud? Nice to see a screenshot. Keep'm coming :)

Will try to add a checkbox for OSM map export, as it can be annoying sometimes. A link to Google maps will always be launched, because using elevation data without displaying a map for which elevation data was requested is prohibited according to the Google usage limits. Maybe the still doesn't qualify even. Does somebody know?

Tip, OSM also allows to export svg images, which is a vector format. So you can use those paths to snap to the elevated mesh and create roads. Now how about some support for SVGs Lux? I'm tired of mucking around with those finicky EPS files. SVGs can be edited within Inkscape, which is free unlike Adobe Illustrator. And it also doesn't require an install and can be deleted without leaving behind traces in the OS.

Message edited by xtof on 10/31/2012 - 3:35 AM

31-Oct-2012 3:21 AM

Topic - Geo Kit (Updated on 2012-1203)