regionator
(The Crusader)
08/10/06 05:05 PM
A new tool for creating Region-based KML

We have released some code which we hope will be of
interest to those working with Region KML.


For more information please visit:

http://code.google.com/p/regionator/


Here is a direct link to the project's README:

http://regionator.googlecode.com/svn/trunk/README

bplewe
(Tourist)
09/07/06 10:08 AM
Re: A new tool for creating Region-based KML

A non-programmer having some trouble getting regionator to work: We're running under windows (not CygWin), which may be part of the trouble, and python 2.4, GDAL, and numpy seem to be installed and working. The tests in the test folder all worked without a hitch. [ImageMagick was only needed for the image tests, not for regionator in general, right?]
However, whenever we try to run a script that opens an image (like superoverlay.py or testextractor.py), we get the following:

self.__in_ds = gdal.Open(imgfile)
AtrributeError: 'module' object has no attribute 'Open'

My guess is that this is an installation snafu, not a bug in the program. Any ideas?

regionator
(The Crusader)
09/07/06 09:32 PM
Re: A new tool for creating Region-based KML

Sounds like a problem with the installation of gdal or the gdal python bindings.

A quick first test of just gdal's python installation might be to start python
interactively and enter the following lines manually. This presumes 'screeno01.jpg'
exists (yes, ImageMagick is only to create these files for other tests and is not
needed for creating a superoverlay). Obviously if you use a different image file
the x,y values may be different.

>>> import gdal
>>> ds = gdal.Open('screeno01.jpg')
>>> print ds.RasterXSize
32
>>> print print ds.RasterYSize
32

bplewe
(Tourist)
09/07/06 11:36 PM
Re: A new tool for creating Region-based KML

Yes, it was the installation of GDAL. "import gdal" was interpreted as the gdal folder, not gdal.py, which confused everything. We ended up having to change the folder to "gdal1", then add that folder explicitly to the pythonpath. Not a clean workaround, and not sure what is different from your system, but it works. Works great, in fact. Thanks for the tool!

vestigialars
(Tourist)
10/06/06 10:45 AM
Re: A new tool for creating Region-based KML

Hi All,
I am having similar problems on Windows, I'm keen to try regionator. Anyone willing to help me? I am a newbie on Python and GDAL, all my imagery is on a Novell network and my sysman only allows logins from our Windows machines, so Windows is my only useful option for this work. I've got FWTools installed, which I think means GDAL is installed somehow as well, and have also put GDAL 1.3.2 in a couple other places, all of which are indicated on PYTHONPATH.

However, when I try and run any of the image based test scripts or the superoverlay.py script I am told:

C:\temp\subversion\build\scripts-2.5>superoverlay.py
Traceback (most recent call last):
File "C:\temp\subversion\build\scripts-2.5\superoverlay.py", line 28, in <module>
import kml.superoverlay
File "C:\Python25\Lib\site-packages\kml\superoverlay.py", line 27, in <module>

import kml.image
File "C:\Python25\Lib\site-packages\kml\image.py", line 25, in <module>
import gdal
File "C:\gdal-1.3.2\pymod\gdal.py", line 191, in <module>
import _gdal
ImportError: No module named _gdal

Anyone know what to do? Got any help? Anything at all?

Thanks,
Lars

regionator
(The Crusader)
10/08/06 10:47 PM
Re: A new tool for creating Region-based KML


Python isn't finding gdal. To more directly debug this start python
interactively and enter "import gdal". Once that works the regionator code
which uses gdal will also work.

Also fwtools includes its own fairly old python so watch you might want
to be mindful of mixing different python installations.

vestigialars
(Tourist)
10/12/06 06:40 PM
Re: A new tool for creating Region-based KML

Okay, gave up on the windows version. My gdal says it is fine but the _gdal is throwing everything off. I am probably missing something simple but can't figure it out right now.

Happily, I have it running on a linux box and it seems to be working. So far so good, thanks!

Lars

vestigialars
(Tourist)
10/17/06 06:43 PM
Re: A new tool for creating Region-based KML

Hi All, me again.

Any advice on how to improve the output image quality from regionator? I am using it on large high-res imagery showing small villages in Darfur. The output JPEGs are downsampled to a certain extent and I lose some quality overall. Not a lot, but enough that I notice. Any quick fix?

Also, at some point I ran regionator and it produced PNGs. Why did that happen, and do I have control over it?

Thanks for any tips...

pawill79
(Tourist)
10/20/06 08:46 PM
Re: A new tool for creating Region-based KML

I have been trying to get the "regionator" to output my files as PNG's only. I have found the image.py and extractor.py as possible culprits for automatically choosing my output files but have not successfully edited those files to do what I want.

Has anybody changed these files so that only PNG's are produced, it seems I am only able to create GTiff output files.

pawill79
(Tourist)
10/20/06 09:00 PM
Re: A new tool for creating Region-based KML

Ok, I figured this one out. Forgot to recompile the regionator after making changes to the extractor.py and image.py files.

PatrickB
(Tourist)
10/25/06 10:35 AM
Re: A new tool for creating Region-based KML

when running superoverlay.py or testextractor.py I am getting the following error:

ERROR 1: TIFFOpen:/tmp/file.gtiff: No such file or directory
Traceback (most recent call last):
File "C:\regionator\tests\testextractor.py", line 36, in ?
ex.Extract(0,0,256,256,'0')
File "C:\regionator\kml\extractor.py", line 100, in Extract
o_ds.WriteRaster(0,0,twid,tht,i_data)
AttributeError: 'NoneType' object has no attribute 'WriteRaster'

can anyone help?

PatrickB
(Tourist)
10/25/06 11:30 AM
Re: A new tool for creating Region-based KML

oh nevermind. I simply had to create a C:\tmp directory on my system.

regionator
(The Crusader)
10/25/06 01:43 PM
Re: A new tool for creating Region-based KML

Tnx for reporting this.

I just updated the code to use a safer and platform independent temp file method.

happy regionating...

PatrickB
(Tourist)
10/26/06 05:34 AM
Re: A new tool for creating Region-based KML

Awesome! Thanks Regionator... I just checked out the new code and it works great.

I am new to python (started using it 2 days ago when I checked out regionator).

when running the regionator code I am getting the following errors:

version 81
original dimensions (880, 880)
maxdepth 4
count 24
root region (28.828125, 28.125, -80.15625, -80.859375)
llb-3\4.kml
llb-3\5.kml
llb-3\6.kml
llb-3\7.kml
llb-3\3.kml
llb-3\9.kml
llb-3\10.kml
llb-3\11.kml
llb-3\12.kml
llb-3\8.kml
llb-3\2.kml
llb-3\15.kml
llb-3\16.kml
llb-3\14.kml
llb-3\13.kml
llb-3\18.kml
llb-3\20.kml
llb-3\21.kml
llb-3\22.kml
llb-3\23.kml
llb-3\19.kml
llb-3\17.kml
llb-3\24.kml
llb-3\1.kml
Extractor NASA_KSC.jpg 3 bands
1/24 llb-3\1.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpx-lf88GTiff failed.

2/24 llb-3\2.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmp__uo3bGTiff failed.

3/24 llb-3\3.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpudegm5GTiff failed.

4/24 llb-3\4.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmp78vghwGTiff failed.

5/24 llb-3\5.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpctoubgGTiff failed.

6/24 llb-3\6.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpl8odn2GTiff failed.

7/24 llb-3\7.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpungpbyGTiff failed.

8/24 llb-3\8.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpzaxovtGTiff failed.

9/24 llb-3\9.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpisjsccGTiff failed.

10/24 llb-3\10.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpan4qtcGTiff failed.

11/24 llb-3\11.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmp4qcwkyGTiff failed.

12/24 llb-3\12.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpo-ppxvGTiff failed.

13/24 llb-3\13.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpef8lb1GTiff failed.

14/24 llb-3\14.JPEG
15/24 llb-3\15.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpp7swwtGTiff failed.

ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpcdunluGTiff failed.

16/24 llb-3\16.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpxvz7h5GTiff failed.

17/24 llb-3\17.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpsf4jhcGTiff failed.

18/24 llb-3\18.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpffdsakGTiff failed.

19/24 llb-3\19.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpq5c5s3GTiff failed.

20/24 llb-3\20.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpcchkdmGTiff failed.

21/24 llb-3\21.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmppeoxfeGTiff failed.

22/24 llb-3\22.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpf3wcv4GTiff failed.

23/24 llb-3\23.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmpiqfgotGTiff failed.

24/24 llb-3\24.JPEG
ERROR 1: GTiff: Attempt to unlink c:\docume~1\admini~1\locals~1\temp\tmptfzq5sGTiff failed.

What does that mean and where is it comming from? The error message is not very descriptive.

Thanks!

PatrickB
(Tourist)
10/26/06 05:35 AM
Re: A new tool for creating Region-based KML

I had to do the same thing on my Windows system. Has anyone figured out a better way to do this?

PatrickB
(Tourist)
10/26/06 05:45 AM
Re: A new tool for creating Region-based KML

Here are the steps that I followed to get it working on my windows machine.

1. Go to Google Earth Regionator Project Home and checkout the latest version of the code using svn (I used TortuseSVN )
2. Go to Python 2.4 and download Python 2.4
3. Go to UrbanSim's GDAL installation page and download/install GDAL (I followed the instructions for Quick Installation )
4. Go to NumPy and download/install NumPy from it’s Sourceforge download site (you should download numpy-1.0.win32-py2.4.exe)

Put all those pieces together in the same order as above and everything should work great (on windows).

For fun I also downloaded PyDev which is a super awesome extension to Eclipse 3.2 for python development!!

PatrickB
(Tourist)
10/26/06 09:38 AM
Re: A new tool for creating Region-based KML

So,
I have a ridiculously big jpg image ~ 20,000px x 20,000px. I tried running regionator on it and it quickly creates ~ 5,500.kml files. Then it starts making the .jpeg files. The first few take almos 1 minute each to create but it gets faster after that. I have not delved into the code yet and I am not saying that it runs too slow, but I am wondering if there is any easy way to make it faster? Or could someone at least point me in the right direction or have any ideas about this?

Thanks!

Forkboy2
(Cartographer)
10/26/06 02:21 PM
Re: A new tool for creating Region-based KML

I have a feeling you're going to start running into problems with a file that large even if you do get it to work. The more levels of nested KMLs you have, the more GE seems to choke trying to process all of them as you zoom in. Anything more than a 32x32 grid of tiles and things really start to slow down. At least that's been my experience. Might not be so bad if you're not planning on putting the files online.

Matt

regionator
(The Crusader)
10/26/06 06:30 PM
Re: A new tool for creating Region-based KML

20k x 20k is entirely reasonable (and pretty much the whole point :-)
and we've done significantly larger with this exact software.

Extracting 5000+ tiles takes time indeed. Some will be faster than
others as the code is extracting and resampling the original at
each level of hierarchy. I would guess 10's of minutes, but it
obviously depends on your hardware and likely very much on
how much RAM you have.

We've created SuperOverlays upwards of 400,000 tiles and pretty
deep hierarchies.

PatrickB
(Tourist)
10/27/06 05:23 AM
Re: A new tool for creating Region-based KML

Thats good news.

Say I am running on a multiprocessor system does anyone have any suggestions on how I could better thread the program to make it run faster? Would adding threading even make a difference?

Or is the only solution to this problem (kinda slow with huge files) to throw more hardware at it?

denisew
(Tourist)
10/28/06 05:14 PM
Re: A new tool for creating Region-based KML

Colégio Inovação II

ink_polaroidAdministrator
(Sysop)
10/31/06 01:05 PM
Re: A new tool for creating Region-based KML

Internally, GDAL (the heart of the regionator) will work with TIFF files, no matter what you throw at it.

In the case of a very large JPEG, it will decompress that file and convert to TIFF many, many times. This is the source of the slow operations.

You should convert any large source file to TIFF before regionating it...

The brain-off method is to use gdal_translate:

gda_translate img.jpg img.tif

regionator
(The Crusader)
11/01/06 08:22 AM
Re: A new tool for creating Region-based KML

A 2006 vintage computer with a couple of gig of ram and a ghz or 2 of cpu
completes a 20k x 20k SuperOverlay in less than 15 minutes. Some tiles will
take far longer than others as the process involves downsampling all to most
of the entire image for the coarsest levels in the hierarchy. I most commonly
use an Apple PowerBook G4 (yes, not even an Intel MacBook) with 2gb of ram
and a mere 1.67 ghz PowerPC cpu. This generally presumes TIFF input for
some reasons Ink mentioned previously.

regionator
(The Crusader)
11/01/06 08:51 AM
Re: A new tool for creating Region-based KML

Anyone creating SuperOverlay tools might be interested in this:

http://code.google.com/p/regionator/issues/detail?id=1&can=2&q=

PatrickB
(Tourist)
11/01/06 10:01 AM
Re: A new tool for creating Region-based KML

Thanks Ink, Regionator. Converting to a .tiff and using a Windows Server with a 3.80GHz processor and 3.5 GB of RAM solved all my problems!

Converted a 20,000px x 20,000px in 11mins

PatrickB
(Tourist)
11/02/06 08:41 AM
Re: A new tool for creating Region-based KML

I modified the code in extractor.py starting at line 99.. so that only 1 temporary
file is created. Since on Windows there is a bug in deleting a temporary file using
Python, this change will create only 1 temporary file that will always be used each
time the program is run.

There may be a problem with this change if you run more than one instance of
regionator simultaneously or if the code is modified to be multithreaded.

Any suggestions or corrections would be greatly appreciated

The code in extractor.py after line 99 was changed to the following:

# Have to Create out to GTiff first (?)
tmpfilePath = tempfile.gettempdir()+"\\tmpfile"
tmpfile = open(tmpfilePath, "w")
o_ds = self.__gtiff_driver.Create(tmpfilePath,twid,tht,bands=self.__bands)
o_ds.WriteRaster(0,0,twid,tht,i_data)

# Save off using the specified driver
filename = '%s.%s' % (basename,self.__fmt)
self.__o_driver.CreateCopy(filename,o_ds)

if self.__verbose:
print filename

# Delete the tmp file
tmpfile.close()

# Does not delete the file if running on Windows.
# Will produce the following:
# [Errno 13] Permission denied:
# 'c:\\docume~1\\admini~1\\locals~1\\temp\\tmpfile'
self.__gtiff_driver.Delete(tmpfilePath)

regionator
(The Crusader)
11/02/06 09:35 AM
Re: A new tool for creating Region-based KML

What exact version of Python and Windows are you have this problem?
I happen to have a colleague who knows alot about Python :-)
(The original extractor code actually did what you did to your copy,
but as you point out this is unsafe).

Other Python on Windows hackers are free to chime in with their
tmp file creation experiences.

Also, just to make your life difficult kml/extractor.py was just modified
to explicitely set a higher quality level on the JPEG output driver.
The default is "75" which is just plain too low most of the time.
extractor.py now sets it to "90".

PatrickB
(Tourist)
11/02/06 10:00 AM
Re: A new tool for creating Region-based KML

I am using Python 2.4 on Windows XP and Windows 2003 Server (both give the same error). I will try using Python 2.5 instead!

regionator
(The Crusader)
11/02/06 08:55 PM
Re: A new tool for creating Region-based KML

The latest release of GE4 (241X) supports GroundOverlays at altitude and
the Regionator code has been updated to support this as well.

superoverlay.py now takes a groundoverlay.kml as input and
parses out LatLonBox, altitude, drawOrder as well as time and
reflects such into the superoverlay.

Here is an example:

http://regionator.googlecode.com/svn/trunk/tests/ksc-llb-a.kml

An update to the KML 2.1 Reference will appear soon...

pawill79
(Tourist)
11/16/06 01:55 AM
Re: A new tool for creating Region-based KML

To make the regionator more dynamic for your webserver... You can edit /usr/local/lib/python2.4/site-packages/kml/regionator.py (or your path to python site packages) and change...

link.href = '%s/1.kml' % dir
to...
link.href = 'http://www.yourwebsite.com/%s/1.kml' % dir
or even
link.href = 'http://www.yourwebsite.com:8080/%s/1.kml' % dir
if you have to run your webserver on a different port.

This prevents you from having to edit the root.kml after running the superoverlay.py

regionator
(The Crusader)
11/16/06 09:06 AM
Re: A new tool for creating Region-based KML


I'm not following why it's easier to change every KML file
within the hierarchy than it is to change just the root.kml file?
(This is the whole point of root.kml).

But, if you for some reason wish to bake absolute <href>'s
into your KML that will work most certainly. You might
also wish to do that with the <href>'s of the <GroundOverlay>
<Icon>'s as well?

It's certainly possible to bake in <href>'s to multiple
servers as well. Put kml on one server, and imagery on another.
Or, put different parts or levels of the overlay on different
servers. But, even then I suspect that load balancing
technologies at the webserver might be a better idea.

It's a very powerful concept that all <href>'s are relative
within the KML hierarchy. It means the exact same hierarchy
can be simultanously served over http or accessed as
local files (as I do pretty much all the time: I create the
overlay and verify it on my local file system and then
simply move the whole thing to a directory my favorite
http server can see and now everyone else is one
"Add Networklink" away from seeing it too).

Anyways... tnx for your interest in KML and the Regionator
and let me know if there's something I'm not understanding
about what you are trying to accomplish.

pawill79
(Tourist)
11/16/06 02:39 PM
Re: A new tool for creating Region-based KML

Making the change I stated previously only changes the "href" for the root.kml. The other associated kml's still have relative urls for its hierarchy. The main reason that the root.kml has to be changed for the href is because if it does not have a absolute http address then it tries to find the associated kml's on the local file system.

Here is an excerpt from the superoverlay.py in the svn trunk...

"The root file is a small NetworkLink KML file which points
to the top of the SuperOverlay. The href to 1.kml should be
changed to be the URL of the directory. This file is
distributed to users of the SuperOverlay. Note that the
NetworkLink within is a Region-based load and that opening
this file in Google Earth neither loads any imagery nor
flies to the SuperOverlay. When the user flies to the
vicinity of the SuperOverlay this first NetworkLink is loaded
which triggers further NetworkLink loads within the SuperOverlay
as appropriate for the users viewpoint."

Since this seems that the root.kml has to be manually changed, I wanted just to show that you could make the change more dynamic by changing some of the python code.

jsundqui
(Tourist)
01/29/07 08:43 PM
Re: A new tool for creating Region-based KML

Quote:

Sounds like a problem with the installation of gdal or the gdal python bindings.

A quick first test of just gdal's python installation might be to start python
interactively and enter the following lines manually.

--SNIP--

>>> import gdal

--SNIP--



I have the problem of python not seeing gdal.

>>> import gdal

just gives me

ImportError: No module named gdal.

I am running Kubuntu Edgy, and I've installed python-gdal, which puts the modules in /usr/share/pycentral/python-gdal/site-packages

This is the same place that Debian testing puts this, although I noticed that Debian stable put the gdal python bindings in /usr/lib/python2.3/site-packages/

I suppose I could do a brute force kludge with sym-links if python is looking at /usr/lib/python2.*, but is there a better way?

Or is the /usr/share/pycentral/* approach correct, but I have other problems? Seems like there are plenty of other modules in /usr/share/pycentral/* For example, I built numeric before installing regionator, and the numeric python bindings also showed up there.

Any help is appreciated.

jsundqui
(Tourist)
01/29/07 08:53 PM
Re: A new tool for creating Region-based KML

Well, I looked all over before posting, but then I found that this is a recently patched bug:

http://www.mail-archive.com/pkg-grass-devel@lists.alioth.debian.org/msg01172.html

I'll see if I can get the fixed packages.

vestigialars
(Tourist)
02/01/07 03:40 PM
Re: A new tool for creating Region-based KML

Greetings All,
My organization has a grant funded project exploring applications of geospatial technologies, including high-resolution satellite imagery, to human rights work. Working in collaboration with groups such as Amnesty International and others, we attempt to develop and or acquire geospatial data and information to support legal, advocacy, and monitoring work undertaken by such organizations. Over the last year we've undertaken limited activities in Darfur/Chad, Zimbabwe, and Lebanon, which we've reprocessed for GoogleEarth using regionator code and some other tools.

I know there is a lot on GE already for some of these areas, and our GE products are probably crude compared to some, but regardless here they are:

http://shr.aaas.org/geotech/ge.shtml

We welcome any feedback and suggestions from the broader community out there. We will continue to post aspects of our work in similar fashion as we make progress. Above all, thanks to the regionator crew!

Thanks for your time,

Lars Bromley
American Association for
the Advancement of Science
lbromley@aaas.org

urbanlayer
(Tourist)
03/07/07 05:58 PM
Re: A new tool for creating Region-based KML

Has anybody tried the new modelator.py? I'm trying to regionate my kml models into one network link but I am getting stuck on the rootkml file I think... Can anybody help? In command prompt, I ran " C:\Python25\Scripts>modelator.py C:\model_kml test.kml C:\temp7 " and get the following error many times in a row: " File "C:\Python25\Lib\site-packages\kml\region.py", line 194, _Search return child._Search(r) "

and then it tells me "RuntimeError: maximum recursion depth exceeded in cmp"

Any ideas?

Thanks in advance.

regionator
(The Crusader)
03/08/07 12:05 PM
Re: A new tool for creating Region-based KML

What is the result of running testall.sh in the tests directory?
And/or what is the result of running testregion.py and testregionator.py?

I don't run this terribly often on Windows I'm afraid (I develop mostly on Mac OSX
and Linux) so if a Python-on-Windows guru wants to chime on some
obvious issue there we're all ears...


earth.google.com    bbs.keyhole.com

*
UBB.threads™ 6.5.1.1