Where did THAT come from?

All too often Python programmers ask themselves, “Who took the last cup of coffee and didn’t start another pot brewing!?”  Probably more often they’ll wonder “Where the hell is my proggy loading that function (or library or module) from?”   Whereas one can usually track down the SOB who left the empty pot and go all BobP* on him, it is often harder, depending on the platform, to answer the latter question.

Recently I found myself with several virtualenv paths as well as two or three actual installations (talking Python, here) along with several environment variables (PATH, LD_LIBRARY_PATH, PYTHONPATH, DYLD_LIBRARY_PATH, DYLD_FRAMEWORK_PATH, etc.) that were, to be perfectly blunt, FUBAR.   The code I was working on needed to load a modified version of the SQLITE3 module, which itself loaded a shared object (.so) file somewhere within its bowels.    No matter how I changed the several *PATH variables, I kept getting results that indicated the modified shared library simply wasn’t being referenced.   I could have spent a great deal of time trying various permutations of the paths in each of the PATH variables.   Having no desire to make a career (let alone a life-long persuit) of this, I decided to take a somewhat more sane approach.   Surely, there must be a way to tell which module python actually loads when it encounters a line like:

import sqlite3

Enter the modulefinder module (section 30.6 of the Python Library Reference).  This contains a class by the same name that gathers information from import statements encountered in the course of executing a script.  Its report method can be invoked after the script has finished and will list the modules imported  and the actual files where those modules were found.    So, to find out which sqlite3 module was being imported, I first tried:

from modulefinder import ModuleFinder
mf = ModuleFinder()
mf.run_script('import sqlite3')

which rewards me with:

Traceback (most recent call last):
  File "", line 1, in 
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/modulefinder.py", line 112, in run_script
    fp = open(pathname, READ_MODE)
IOError: [Errno 2] No such file or directory: 'import sqlite3'

What modulefinder really wants is a script that contains the import statements you’re looking to trace.   So, I put

import sqlite3

into a file named import_this.py and tried again:


and … Success!

Name                      File
----                      ----
m __main__                  import_this.py
m _sqlite3                  /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_sqlite3.so
m datetime                  /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
P sqlite3                   /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py
m sqlite3.dbapi2            /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py
m time                      /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so

This canned report is nice, but you might want to use ModuleFinder‘s results
programatically, in which case you’ll want to refer to its modules attribute, which is
a Python dict object mapping module names to the files from whcih they were actually imported.

Beyond Python

Although this doesn’t tell me where my modified SQLITE3 library is being loaded, it does bring me a step closer.
I can now tell where python is picking up its sqlite3 module along with several other modules, some of which I can readily dismiss (datetime.so and time.so.) It’s obvious that sqlite3 is a Python package from the final part of its path (…/sqlite3/__init__.py) as well as from the capital “P” (for “package”) to the left of the module’s name in the report. Looking inside this file we find just one (executable) line of Python:

from dbapi2 import *

From the ModuleFinder report we already knew this module was being imported, so we continue digging there:

from _sqlite3 import *

which, again, confirms what we already knew from the report. But, that’s all we find in this file, aside from a few class declarations that are nothing more than wrappers and dressing for the contents of _sqlite3. If we go fossicking about inside the source code for _sqlite3.so (which can be found under the Modules subdirectory of the Python source tree) we’ll find a very simple and typical Python extension module that wraps the functions found in the SQLITE3 library itself. This seems to be a bit of a dead end. We still don’t know where the actual location of the SQLITE3 library (i.e. libsqlite3.dylib) that the extension module is referencing. For this, we’ll need a different tool: the ctypes module.

Ctypes let’s us load and easily access any dynamically linked library (DLL, .so, .dyld, etc) object and, in the case of function objects, call them from within python code. It also contains a very useful function called find_library that will … well … find shared libraries. It uses the environment’s *PATH variables the same way the dynamic linker would use them (i.e., searching these in the same order) for the specified library. So, we look for sqlite3:

from ctypes.util import find_library


and get:


Looking at my DYLD_LIBRARY_PATH environment variable, I have only /opt/local/lib in the path. I can either replace the libsqlite3.dyld file in this directory with my own, modified version, or I can prepend this environment variable with the path where the modified library can be found:

export DYLD_LIBRARY_PATH="/home/nick/tryit/sqlite-3.6.23/.libs:$DYLD_LIBRARY_PATH"

and then use find_library as before

from ctypes.util import find_library


which yields


just as I’d expect.


There is always the temptation to automate processes such as these. I could have tied this all together into a nice neat little function that would take the module name as its argument and return the path, if any, of the associated dynamic library. If I were trying to develop a large scale code scanning tool, say, for doing a security audit of the code, I might do just that. Here, I just wanted to find out which of several possible dynamic libraries was being loaded by importing a single, particular module. A few lines of code are all that’s needed in this case. I’ll leave the fancier tool development to the intrepid reader. Right now, I have to go make some more coffee … before I get yelled at. (Again.)

Acquainted with the Night

A poem by Robert Frost, with Dave and Angela in mind:

Acquainted with the Night

I have been one acquainted with the night.
I have walked out in rain — and back in rain.
I have outwalked the furthest city light.

I have looked down the saddest city lane.
I have passed by the watchman on his beat
And dropped my eyes, unwilling to explain.

I have stood still and stopped the sound of feet
When far away an interrupted cry
Came over houses from another street,

But not to call me back or say good-bye;
And further still at an unearthly height,
A luminary clock against the sky

Proclaimed the time was neither wrong nor right.
I have been one acquainted with the night.

“No Maccas” Campaign, now with Flash-mob!

My mates at BurgerOff.Org have begun organizing flashmob protests via facebook and twitter.   Within hours, the movement is able to organize dozens of protesters to convene at specified McDonalds locations all for the purpose of delivering the message to McDonalds:  We do not want McDonalds in Tecoma or anywhere in the Dandenong Ranges.


Over-doing It

Last night’s dinner was to include a medley of broccoli and cauliflower to go with a slow-cooked pork roast.   Distracted by a phone call (or a political article I just had to comment on), the five-or-so minute steaming of the veggies went on for more like 15 minutes.   The broccoli was almost impossible to pick up with anything but a fork, and the cauliflower was almost as soft.   I just hate to through anything out, most of all food, so rather than consign it to the compost (or the chooks), I thought I’d just go for broke and turn these into a paleo version of mashed potatoes.    Normally I’d add roasted garlic and fresh rosemary from the garden, but I took a shortcut on the garlic and just used a bit of garlic powder.   Herbs added, I got out the kitchen wand and literally whipped up a rather delicious, creamy  concoction.   I’d include a recipe, here, but it’s just too damned simple.   There are so many ways to vary this, but the basic idea is to salvage overcooked veggies (or deliberately overcook them.)

White veggies such as cauliflower and parsnip will produce something that has roughly the same look and texture of “real” mash.  You could probably even hold back a bit of the parsnip during the initial whipping to add in for a few seconds at the end to give a more chunky texture.   Other, far more colorful variations are easily created using yams, beetroot, capsicum (red, yellow, green), carrots, etc.  Add a bit of ghee or olive oil or avocado oil to make it a bit more buttery.

For More Years

President Barack Obama has won another term in office.   The Congress that awaits him is much the same as it was before the election in the sense that Republicans still hold the House whereas the Democrats control the Senate.  The differences in those two chambers are small, but significant.   There are more Democratic Senators now making passage of the Democratic agenda there somewhat easier.  There are also more women Senators now, all the new ones being Democrats.  This should help mitigate the systemic misogyny exemplified by Tod Akins and Richard Mourdocks, neither of whom, I’m glad to say, will darken the halls of the US Capitol now.

What can we expect from President Obama now that he’s a “lame duck”?   In addition to the continued push for reforms in the financial and insurance industries, as well as a greater push to make America a leader in the field of Green Energy and sustainable living, there are a few lingering matters that will not get proper attention — or inattention, as the case may be.    All of those things the Republicans hold so dear, particularly all those tax cuts for the wealthy, are still very much in play.  You can be all but certain that the Bush tax cuts will now be allowed to expire unless the recalcitrant Republicans are provided with some other impetus to move off their collective … agendas.

Kentucky is home to one of the “supermax” prisons, the sort of facility you’d need to house the detainees now incarcerated at Guantanamo Bay.   Kentucky’s senator Mitch McConnell will probably be a bit annoyed to learn that Obama will now make good on his promise to close the base and move all those detainees to Mitch’s home state.   I’m sure his constituency will be very impressed with his inability to keep that from happening.   Bye bye, Mitch.

The insurance companies will start to see that they’re no longer able to make the sorts of obscene profits on health insurance premiums once Obamacare becomes fully implemented, so they’ll start to bow out, making a little-mentioned public option built into the law an increasingly viable alternative.   With a little luck, we’ll not only have universal health care in the US, but it’ll evolve into single-payor health care.  What this means for you and me is that we’ll no longer be paying through the nose for insurance that can, at their whim, deny or cancel our coverage.

One thing you probably won’t see is another war.  No, not even with Iran.   They’ve already offered an olive branch and Obama now has the latitude — and the Senate — to negotiate with Iran to eliminate their nuclear fuel program and make its way back into the family of nations.

We’ll stop “eating our seed corn”, and begin to invest in education and research the way we once did.   Years after Obama finally does leave office, we’ll start to see the fruits of this as we become smarter as a whole.  We’ll start solving the real problems that plague us, that have gone neglected in favor of beating up on other countries, taking their wealth.  Once solved, we’ll once again be looking far beyond where we are, and start the journey in earnest to get there.   It might be Mars, or it might be a world that no longer fights over oil fields or water or imaginary friends or lines on a map.

What we have, now, is a breather.  But only that.     We need to set things in motion now that will have momentum, four years from now, to withstand the ignorance that plagues us as a nation, and the feeding frenzy of fear that the corporatocracy would have us all gorge on.   We must stop thinking of ourselves as teams, one that must beat the other.   We’re all in this together, and we’ll continue to be for more than just the next four years.


The Day Before Tomorrow

It’s already Tuesday here, in Oz.  My fellow Americans are all bedding down for the night, happy in the knowledge that their long, national nightmare is nearly over.  The political ads will soon cease, and what we’ll be left with is a yawning abyss of economic suicide otherwise known as the Romney Administration, or the slow but steady rebuilding of America that wil be Obama Redux.    I’m fairly certain the outcome will be the latter, for two reasons.   First, I’ve been following Nathan Silver’s blog and unless there’s something very subtle and yet very hugely wrong with his methodology, his analysis is pretty sound. And, second, even though I’ve seen GWB ascend to and maintain the office of POTUS for a miserable eight years, I have this irrational yet undying belief that there are still a sufficiently large number of my compatriots who have enough brain cells (i.e. greater than, say ten) to understand just how important it is to re-elect Obama.

What I’m a bit more concerned about are the congressional races.   The good news is that Akins’ and Mourdock’s self-destructive strategies will work brilliantly: both will — and SHOULD — lose (or fail to gain) the seats they seek.   The Democrats’ Senate majority, however, is still a bit shaky and I’ll feel better not only if they hold on to it, but if they’ll also give Harry Reid a nice watch, a great party, and a nice, comfy place by the window, but keep him as far away from the legislative process as possible, other than to vote as he’s told to.     The House, of course, is the real worry.  Democrats are in peril, shockingly enough, of losing seats, so I’ll be glad if they simply hang on to what they have.    Any gain will be icing on the cake.

I got into a bit of a … discussion … the other day with someone on FB and pointed out that his kids and mine will be left to clean up a Romney/Republican-made mess, or benefit greatly from a second Obama term.   His response was to set his own hair on fire and scream at me for “bringing our kids into this”, and summarily bowed out of the discussion.   In fact, for all our arguing and hand-wringing and all the energies over which candidate or which policy, it is PRECISELY about our kids and the country and the future we will leave to them.   It seems incredible to me that anyone with children would enter a voting booth without thinking to themselves Which of these candidates will do better not just for me, but for my children, too?

When I look at my kids, all of them grown now, two with families of their own, I see the future, I see tomorrow.   I don’t really think much about the future beyond their lifetimes.  Maybe my thinking is too restricted, too temporally parochial.   I will; certainly never see all their tomorrows with my own eyes, but, I can help to see that tomorrow is better than today, and far better than yesterday.    We all can.  And should.  Never in my lifetime has the political landscape been so stark, and the choice so clear, the consequences so far-reaching.  The tomorrow I wish for, and for which I cast my vote (via absentee ballot) over a month ago  is one in which intelligence is valued and praised, not mocked, in which reason, not religion, forms the foundation of public policy, in which my daughters and grand-daughters will not be subject to the superstitious, misogynist whim of men, and in which all of my children live in a country that is truly of, for, and by the people.    This tomorrow will never be possible if Romney is elected.    May that tomorrow never come.


So, originally, this blog was to be about my dabbling and forays into sustainable gardening and energy generation and storage.   However, I’ve decided to turn it into a more general compendium, largely due to my impending secession from a certain extremely popular social media site that shall remain nameless.  Once this election cycle is over, I’m done with that, however, I won’t be done commenting on politics or opera or sustainable living or whatever.